search

Snowflake-Labs / snowflake-arctic

Apache License 2.0
511 stars 41 forks source link

Add sentencepiece to requirements #25

Closed jeffra closed 3 months ago

jeffra commented 3 months ago

It seems LlamaTokenizer requires sentencepiece as a dependency. This resolves an issue where fetching the Arctic tokenizer triggers the following issue:

AutoTokenizer.from_pretrained("Snowflake/snowflake-arctic-instruct" , trust_remote_code=True)

-> 

ImportError: cannot import name 'LlamaTokenizer' from 'transformers.models.llama'