Closed sshleifer closed 4 years ago
Yes, the last line is correct. You should preprocess.sh without the language flag. That will be added after tokenisation - so yes, it should come after spm-encode. preprocess.sh should work if you have plain text as input without the language token. Does that work?
Yes, thanks!
For this multilingual model (en-ROMANCE), test set I am having trouble running inputs with language codes through marian. Specifically, if I add the codes after running
spm_encode
, things are fine. If I try to use the preprocessing logic inpreprocess.sh
, things break.test set:
>>pt<< Don't spend so much time watching TV.
Using preprocess.sh, I get either
or
but both of those cause marian_decoder to error.
The correct input seems to be
With the code added after
spm_encode
is run. Is that correct?