neukg / GRTE

51 stars 15 forks source link

The return value is not match #7

Open bigbrother001 opened 1 year ago

bigbrother001 commented 1 year ago

The code of data_generator.iter() in main.py is tokenids, ,mask = self.tokenizer.encode(d['text'], max_length=self.max_len) while the tokenizer in bert4keras only return 2 value, first_token_ids and first_segment_ids, how do you make it?

bigbrother001 commented 1 year ago

I think it's better just to get rid of it. Please remind me if I'm wrong

ElysiumCreed commented 7 months ago

I got this problem too. Just change tokenids, ,mask to token_ids, mask ???

zlh-source commented 4 months ago

抱歉现在才回复。甚至可能已经不需要了回复了😂 这里就是文本转化为token id,使用huggingface transformers的tokenizer.encode函数也是一样的