FEAT: Support torchao - Githubissues

huggingface / peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

https://huggingface.co/docs/peft

Apache License 2.0

15.98k stars 1.56k forks source link

FEAT: Support torchao #2062

Open BenjaminBossan opened 2 weeks ago

BenjaminBossan commented 2 weeks ago

Add support for torchao.

The current status is:

only LoRA explicitly supported
only linear layer
int8_weight_only works fully
int8_dynamic_activation_int8_weight only works partly (as dequantize is not supported, merging and DoRA won't work)
int4_weight_only not supported as some ops for forward call are missing
nf4 not supported on transformers side

HuggingFaceDocBuilderDev commented 2 weeks ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan commented 2 weeks ago

With https://github.com/huggingface/transformers/pull/33361 being merged (which marks torchao as traininable), once the next transformers version is released (>4.44.2), the GPU tests on this PR should pass (I tested locally). This PR should not be merged before that.

review-notebook-app[bot] commented 1 week ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.