Should the source/reference/hypothesis text be tokenized or detokenized?

mahfuzibnalam / terminology_evaluation

MIT License

21 stars 7 forks source link

Should the source/reference/hypothesis text be tokenized or detokenized? #4

Open nangying1112 opened 2 years ago

nangying1112 commented 2 years ago

Dear authors, thanks a lot for the great work. I have a question about the input of "evaluate_term_wmt.py". Should the source/reference/hypothesis text be tokenized or detokenized?