Closed PonteIneptique closed 5 years ago
Yes, I agree. It is though a bit tricky because you should ensure compatible tokenization with the training data, which is not always possible, since it's not clear what tokenizer was used for most training sets. But yes, feel free to add something like that.
I think we can close this if you are happy with the resulting PR :)
Ye, it looks good, thanks!
Right now, the webapp is run from webapp.py. It contains directly the tokenizer. While keeping the ability to use the webapp in a simply python webapp.py, it should be possible to create the application passing it another tokenizer if deemed necessary.