search

PetrochukM / PyTorch-NLP

Basic Utilities for PyTorch Natural Language Processing (NLP)
https://pytorchnlp.readthedocs.io
BSD 3-Clause "New" or "Revised" License
2.21k stars 258 forks source link

Add BPE Text Encoder #7

Open PetrochukM opened 6 years ago

PetrochukM commented 6 years ago

Tokenizer Reference: https://github.com/eladhoffer/seq2seq.pytorch/blob/master/seq2seq/tools/tokenizer.py

PetrochukM commented 5 years ago

No, it's an open issue. Would you like to add it?

Columbine21 commented 4 years ago

hi, my co-worker and I finish this part of work using subword-nmt lib recently. We are looking forwards to your review & advice.

abhishinde24 commented 2 years ago

this is issue is still open?