search

JuliaText / WordTokenizers.jl

High performance tokenizers for natural language processing and other related tasks
Other
96 stars 25 forks source link

Simple reversible tokenizer #12

Closed MikeInnes closed 5 years ago

MikeInnes commented 5 years ago

https://sjmielke.com/papers/tokenize/

oxinabox commented 5 years ago

Isn't this how RevTok.jl works?