fix internal_eval lengths

stanford-crfm / levanter

Legible, Scalable, Reproducible Foundation Models with Named Tensors and Jax

https://levanter.readthedocs.io/en/latest/

Apache License 2.0

519 stars 82 forks source link

Closed dlwh closed 2 weeks ago

dlwh commented 2 weeks ago

previously we were padding to max tokenizer length, which is real bad with llama3