Closed wenxindongwork closed 3 months ago
Just updated the examples to load the models in bf16 instead, hope that works!
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The code style workflow is failing, can you run it locally (make style
) and push again so we can merge this please?
just installed ruff and ran make sytle. Thanks!
Use transformers' AutoModelForCausalLM instead of optimum-tpu's AutoModelForCausalLM for finetuning.
The
from optimum.tpu
version imports models that are specifically optimized for inference. While the colab example works for smaller models, it fails with a HBM OOM error for llama3-70b (on a v4-256). Changing the following import statement solved the problem.Before submitting