search

THUNLP-MT / THUMT

An open-source neural machine translation toolkit developed by Tsinghua Natural Language Processing Group
BSD 3-Clause "New" or "Revised" License
703 stars 197 forks source link

A question of the replacement of "@@" #69

Closed minorfox closed 5 years ago

minorfox commented 5 years ago

Lines 172 and 173, hooks:

for i, l in enumerate(decoded_symbols): decoded_symbols[i] = " ".join(l).replace("@@ ", "").split()

Whether it is a substitution method of the below command?

sed -r 's/(@@ )|(@@ ?$)//g')

(I test it on my BPEed corpus, but it doesn't work...)

minorfox commented 5 years ago

After "deBPE", the model's output file also remains some "@@", did those lines to replace those "@@"?

GrittyChen commented 5 years ago

@minorfox Which version of Python do you use? 2.x or 3.x?

minorfox commented 5 years ago

@minorfox Which version of Python do you use? 2.x or 3.x?

python 3.x and tf 1.11.0

thx.

GrittyChen commented 5 years ago

@minorfox I suspect that the problem is caused by the code incompatibility with python 3.x. Could you try to run with python2.x and then give the feedback? Thank you very much!

minorfox commented 5 years ago

@minorfox I suspect that the problem is caused by the code incompatibility with python 3.x. Could you try to run with python2.x and then give the feedback? Thank you very much!

OK! my pleasure.

minorfox commented 5 years ago

@GrittyChen I have been tested it on another computer and it is ok. Thank you for your help.