cannot access gs://unified-io-2-us-east/tokenizer/llama_tokenizer.model

allenai / unified-io-2

Apache License 2.0

572 stars 27 forks source link

cannot access gs://unified-io-2-us-east/tokenizer/llama_tokenizer.model #3

Closed WorldHellooo closed 10 months ago

WorldHellooo commented 10 months ago

can you please share this file ? I cant access it for unknown reason..

xk-huang commented 10 months ago

Just edit the t5x/examples/unified_io/config.py :)

The LLaMa tokenizer also needs to be installed, download the .model file from https://github.com/facebookresearch/llama/tree/main?tab=readme-ov-file and then update t5x/examples/unified_io/config.py so LLAMA_TOKENIZER_PATH points to the download location.

chrisc36 commented 10 months ago

Yes, we just noticed this issue and updated the repo to fix it. Look at the README to see how to make it work.