search

anowell / are-we-learning-yet

How ready is Rust for Machine Learning?
http://arewelearningyet.com
Creative Commons Attribution 4.0 International
454 stars 62 forks source link

Add crate tokenizers #64

Closed huitseeker closed 4 years ago

huitseeker commented 4 years ago

See https://github.com/huggingface/tokenizers

huitseeker commented 4 years ago

@anowell CI failure seems unrelated to the PR

anowell commented 4 years ago

Sorry about the delays. Finally fixed CI (which was blocking changes from being published). Also, there are now more eyes on PRs, so that I don't continue to be the bottleneck.

Your addition is now live: http://www.arewelearningyet.com/data-preprocessing/