build_inputs_with_special_tokens

fxb392 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: prefix_tokens = self.get_prefix_tokens() token_ids_0 = prefix_tokens + token_ids_0 if token_ids_1 is not None: token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("")] return token_ids_0

这个方法，为什么返回[gMASK],\<sop> sentence1 sentence2 \<eos> 不应该是：sentence1,[gMASK],\<sop> sentence2\<eop> 吗？

Expected Behavior

No response

Steps To Reproduce

无

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response

supdizh commented 1 year ago

对！同样有这个问题，难道开始不应该是"<bos>"吗，然后这个"<eop>"应该是模型的输出吧