search

daulet / tokenizers

Go bindings for HuggingFace Tokenizer
MIT License
85 stars 23 forks source link

Update to allow for platform dependent libs in CGO #18

Closed jmoney closed 3 months ago

jmoney commented 4 months ago

This simplifies other go programs usage of this library. Currently, you have to copy the libtokenizers.a file to $(go env GOMODCACHE)/daulet/tokenizers@${version}. The change here forces the user to specify where to load this library from instead of it being required to be at a certain place.

daulet commented 3 months ago

hm interesting, let me test it in a larger repo, see how it works before merging

jmoney commented 3 months ago

hey @daulet , so what do you think of this PR? Do you believe it allows more ease of use of these bindings that we can merge and release?

daulet commented 3 months ago

hey @daulet , so what do you think of this PR? Do you believe it allows more ease of use of these bindings that we can merge and release?

Yes, I think making this build time parameter satisfies all use cases, however i find it annoying to specify LD flags every time i go test, is there a way to simplify that?

daulet commented 3 months ago

i suppose we could just use CGO_LDFLAGS