Contributing MosesTokenizer from NLTK to Moses

moses-smt / mosesdecoder

Moses, the machine translation system

http://www.statmt.org/moses

GNU Lesser General Public License v2.1

1.58k stars 777 forks source link

Contributing MosesTokenizer from NLTK to Moses #195

Closed alvations closed 6 years ago

alvations commented 6 years ago

As per the discussion on

https://www.mail-archive.com/moses-support@mit.edu/msg15864.html
https://github.com/nltk/nltk/issues/2000

It's best to keep the license of the code LGPL within Moses' repo since it can't stay in NLTK (removal in https://github.com/nltk/nltk/pull/2001)

hieuhoang commented 6 years ago

cheers

stefan-it commented 6 years ago

@alvations I hope it's ok to ask the question here, but do you plan to support a commandline interface? E.g. tokenizer.py could mimic the commandline arguments of tokenizer.perl, so I can be used as a kind of replacement.