eole-nlp / eole

Open language modeling toolkit based on PyTorch
https://eole-nlp.github.io/eole
MIT License
24 stars 6 forks source link

fix filtertoolong transform when there is an empty token #27

Closed vince62s closed 1 month ago

vince62s commented 1 month ago

see here: https://github.com/OpenNMT/CTranslate2/pull/1687#issuecomment-2163523905

but maybe we also need to avoid empty tokens resulting from a.split(" ") when there is two consecutive spaces in a