search

awslabs / fast-differential-privacy

Fast, memory-efficient, scalable optimization of deep learning with differential privacy
Apache License 2.0
83 stars 11 forks source link

multi-GPU distributed learning test abbout table2text #31

Closed dong-ai-7 closed 4 months ago

dong-ai-7 commented 4 months ago

Hi, all! Thanks for providing such great tool ! After running the command： bash table2text/run_ZERO1.sh table2text/prefix-tuning table2text/output "e2e" "gpt2" The contents of the output file (table2text/output/generations_model/eval/global_step_00000410.txt ) are just periods and empty lines. Is this an expected result? How to fix it?

woodyx218 commented 4 months ago

I just tested the exact script and the output is normal, no empty lines or periods. Did you observe warnings or errors during the run? Also double check the python package versions.

dong-ai-7 commented 4 months ago

OK, thanks! I will do that!

dong-ai-7 commented 4 months ago

I noticed that the implementation of DP-Zero in the test cases (including image classification and table-to-text) defaults to using the book-keeping (BK) algorithm https://arxiv.org/pdf/2210.00038.pdf for DP full fine-tuning. For PEFT, the BiTFiT examples for CIFAR10 in this repository use DP-BiTFiT https://arxiv.org/pdf/2210.00036.pdf, Perhaps it could be called BK-BiTFiT? As you mentioned, your design can be adapted to any type of PEFT and optimizer. Moreover, the appendix of the BK paper also describes the algorithms for BK-LoRA and BK-Adapter. Are there any plans to integrate these algorithms with ZeRO to enrich the current open-source library in the future?

woodyx218 commented 4 months ago

I think DP-BiTFiT is the proper name because BK is just an implementation of DP-BiTFiT. Just like Opacus and TF-privacy both implement DP-SGD but we don't call it Opacus-SGD. As for PEFT, they are already supported by DP-ZeRO.