search

stanford-crfm / levanter

Legible, Scalable, Reproducible Foundation Models with Named Tensors and Jax
https://levanter.readthedocs.io/en/latest/
Apache License 2.0
519 stars 82 forks source link

backport tokenize-independelty-then-copy from marin #783

Closed dlwh closed 2 weeks ago

dlwh commented 2 weeks ago

this one should subsume the one in marin