Open WaitingOak opened 4 months ago
I've only done a simple test but if you remove the whole line 65 from file tokenizer.py then the 's' at the end of was and his don't get stripped as it were. This line below: text = text.strip("<s>").strip("</s>")
tokenizer.py
text = text.strip("<s>").strip("</s>")
I've only done a simple test but if you remove the whole line 65 from file
tokenizer.pythen the 's' at the end of was and his don't get stripped as it were. This line below:text = text.strip("<s>").strip("</s>")