search

awslabs / fast-differential-privacy

Fast, memory-efficient, scalable optimization of deep learning with differential privacy
Apache License 2.0
83 stars 11 forks source link

How to use in PEFT? #28

Closed addingIce closed 6 months ago

addingIce commented 6 months ago

Could you give an example of using it in LoRA?

woodyx218 commented 6 months ago

It is pretty the same as in the standard training: modify your model with LoRA, then attach the DP engine. You can check BiTFiT examples for CIFAR10 in this repo.

addingIce commented 6 months ago

So I cannot just use "get_peft_model" from peft? And could you point out the specific position of the BiTFiT examples for CIFAR10? Thanks a lot!

woodyx218 commented 6 months ago

You can. Our design is compatible to any PEFT and optimizers. For example, you can change your model to LoRA or BiTFiT in examples/image_classification/CIFAR_TIMM.py between Line 49-52, then attach the DP engine.