Added integration tests

daulet / tokenizers

Go bindings for HuggingFace Tokenizer

MIT License

92 stars 23 forks source link

Closed michele-sama-dialpad closed 1 year ago

daulet commented 1 year ago

quite a large LOC, if still interested in merging i'd ask the following:

find a smaller tokenizer config
explain makefile & vendor changes
not sold on distributing prebuilt binaries this way - wouldn't we have to install all combinations even if we only use one?