Warning with TikToken: "Warning: model not found. Using cl100k_base encoding."

[mjwarren3]

I just started trying to replicate routerbench locally on a Windows machine.

Context on things that occurred before this warning:

• Ran into a few early issues while attempting to use local embeddings, so I commented out anything using the MongoDB connection string.
• I did not run convert_data.py - instead I downloaded the processed data from the Hugging Face repository for zero shot (and switched data_path to data_path: ./data/routerbench_0shot.pkl)

When trying to run evaluate_routers I started getting hundreds (maybe thousands?) of warnings with the same message: Warning: model not found. Using cl100k_base encoding.

I assume we should be pulling the encoding for the various models we are testing in the router? Any guidance on how to fix this?

[SenmiaoORZ]

same problem here

[SenmiaoORZ]

Got the word from author. It is safe to run with that warning.