feature request: separate initial loading and per-line operations

glample / fastBPE

Fast BPE

MIT License

656 stars 96 forks source link

feature request: separate initial loading and per-line operations #10

Closed bittlingmayer closed 5 years ago

bittlingmayer commented 5 years ago

In order to use this lib (or downstream libs) interactively or at run-time, fastBPE should support sending a single line without loading the initial model each time.

See https://github.com/facebookresearch/LASER/issues/46 and https://github.com/facebookresearch/UnsupervisedMT/issues/27

glample commented 5 years ago

Should work with https://github.com/glample/fastBPE/commit/fea9a450ec3761da8593d005e443972667ee2cad