google / maxtext

A simple, performant and scalable Jax LLM!
Apache License 2.0
1.39k stars 247 forks source link

Try a token_path relative to the base config path if it can't be found #715

Closed obrienadam closed 4 weeks ago

obrienadam commented 1 month ago

For model configs, we check a filepath relative to the base config. Recently, a MaxText change made finding a tokenizer file path mandatory even when using synthetic data, which caused many tests to start failing since these paths can be changed on Borg. This update fixes these failures.