peft-givens

This repo is the source code of (quasi-)Givens Orthogonal Fine Tuning integrated to Huggingface peft library (ver.0.6.2)

The implementation of (quasi-)Givens OFT is in ./tuners/givens/

Publication Details 🔗:

Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation, presented at ICML'24.

arXiv: https://arxiv.org/abs/2404.04316

Thanks for your interest in our work! If our work helps, please don't forget to cite us!🌟

@InProceedings{pmlr-v235-ma24a,
  title =    {Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation},
  author =       {Ma, Xinyu and Chu, Xu and Yang, Zhibang and Lin, Yang and Gao, Xin and Zhao, Junfeng},
  booktitle =    {Proceedings of the 41st International Conference on Machine Learning},
  pages =    {33686--33729},
  year =     {2024},
  editor =   {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},
  volume =   {235},
  series =   {Proceedings of Machine Learning Research},
  month =    {21--27 Jul},
  publisher =    {PMLR},
  pdf =      {https://raw.githubusercontent.com/mlresearch/v235/main/assets/ma24a/ma24a.pdf},
  url =      {https://proceedings.mlr.press/v235/ma24a.html},
}

or

@article{ma2024parameter,
  title={Parameter efficient quasi-orthogonal fine-tuning via givens rotation},
  author={Ma, Xinyu and Chu, Xu and Yang, Zhibang and Lin, Yang and Gao, Xin and Zhao, Junfeng},
  journal={arXiv preprint arXiv:2404.04316},
  year={2024}
}

qGOFT integrated to newer version of peft lib is still in progress. Stay tuned! 🚀