search

microsoft / robustlearn

Robust machine learning for responsible AI
http://aka.ms/roblearn
MIT License
458 stars 57 forks source link

about RiFT #59

Closed yaque78 closed 8 months ago

yaque78 commented 9 months ago

I would like to ask why I used the tiny-imagenet200 dataset you gave for AT, and the accuracy of the test set is not as accurate as in your paper? Can you help me with it?

Immortalise commented 9 months ago

Hi, could you please provide more details about the training, inference, and the detailed accuracy discrepancy?

yaque78 commented 9 months ago

When I use the data loading method you gave for tiny imagenet200, the test set accuracy of the initial AT model is about 48%, and the robust accuracy is only about 0.23%. I learned that the val data set and train data set of tiny imagenet The file storage method is inconsistent. When I changed the tiny imagenet class, the test set accuracy increased to 48.81% and the robust accuracy increased to 23.12%. However, it still cannot reach the initial indicators given in your paper. I would like to ask whether this is a problem with data loading. Or is my data damaged during transmission? Thank you very much！

At 2024-02-03 20:17:22, "Kaijie Zhu" @.***> wrote:

Hi, could you please provide more details about the training, inference, and the detailed accuracy discrepancy?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

Immortalise commented 9 months ago

Thanks for brining this issue to us!

I also encountered this error. It seems that I may uploaded the wrong checkpoint (or the dataload method is wrong). I have tested CIFAR10 & CIFAR100, fortunately they are correct. So you may use these checkpoints first. Hopefully I can fix this after two weeks since the Chinese Lunar Year is coming.

yaque78 commented 9 months ago

Thank u very much！

---- Replied Message ---- | From | Kaijie @.> | | Date | 02/04/2024 23:42 | | To | microsoft/robustlearn @.> | | Cc | yaque78 @.>, Author @.> | | Subject | Re: [microsoft/robustlearn] about RiFT (Issue #59) |

Thanks for brining this issue to us!

I also encountered this error. It seems that I may uploaded the wrong checkpoint (or the dataload method is wrong). I have tested CIFAR10 & CIFAR100, fortunately they are correct. So you may use these checkpoints first. Hopefully I can fix this after two weeks since the Chinese Lunar Year is coming.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

yaque78 commented 8 months ago

I have another question, do you just use the most basic settings when generating adversarial weights, do you include strategies such as early stops?

---- Replied Message ---- | From | Kaijie @.> | | Date | 02/04/2024 23:42 | | To | microsoft/robustlearn @.> | | Cc | yaque78 @.>, Author @.> | | Subject | Re: [microsoft/robustlearn] about RiFT (Issue #59) |

Thanks for brining this issue to us!

I also encountered this error. It seems that I may uploaded the wrong checkpoint (or the dataload method is wrong). I have tested CIFAR10 & CIFAR100, fortunately they are correct. So you may use these checkpoints first. Hopefully I can fix this after two weeks since the Chinese Lunar Year is coming.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

Immortalise commented 8 months ago

Hi,

Regarding your original question, I discovered that the issue with low accuracy on the TinyImageNet val set is due to the necessity for preprocessing when we downloaded the TinyImageNet from Stanford website. You can find the required preprocessing steps in this blog post (Chinese Version). After applying the appropriate preprocessing to the validation set, you should observe the expected accuracy.

As for your second question, yes, our training process stops at 110 epochs. The adversarial training approach we follow is outlined in this repository.