search

ain-soph / nas_backdoor

5 stars 0 forks source link

Failed to get expected ASR #1

Open 1171000712 opened 1 year ago

1171000712 commented 1 year ago

Hello, I successfully installed the environment according to the README, but when I run search_genetic.py using the default command given, I cannot get the expected ASR. I'm confused that the models all only get around 10% ASR. In addition, I noticed that model_params is passed in as a parameter in line 139 of search_genetic.py, but according to the paper, generator_params should be passed in as a parameter. Please tell me if there is a problem with my understanding.

ain-soph commented 1 year ago

Hi, thanks for using the code. Sorry that I’m currently busy with a deadline. Will get back to you next week.

lovian14 commented 1 year ago

Hi @ain-soph, I also have a similar issue. Could you please give a quick guidance on how to obtain the ASR as reported in the paper? I used the default command to run the code, but the ASR is always around 10%.

1171000712 commented 1 year ago

Hi, thanks for using the code. Sorry that I’m currently busy with a deadline. Will get back to you next week.

Hello, I also encountered a problem that search_genetic.py will increase the memory without limit when running. Can you answer it together?

ain-soph commented 1 year ago

Not sure if it's the trojanzoo version issue because I update that library a lot after this repo get published.
Will test it.

lovian14 commented 1 year ago

Not sure if it's the trojanzoo version issue because I update that library a lot after this repo get published. Will test it.

Could you please let me know the exact version of trojanzoo you used for attacking the model?

1171000712 commented 1 year ago

Not sure if it's the trojanzoo version issue because I update that library a lot after this repo get published. Will test it.

Could you please let me know the exact version of trojanzoo you used for attacking the model? I tried https://github.com/ain-soph/trojanzoo and pip install trojanzoo . Both of them didn't work correctly.

lovian14 commented 1 year ago

Not sure if it's the trojanzoo version issue because I update that library a lot after this repo get published. Will test it.

Could you please let me know the exact version of trojanzoo you used for attacking the model?

@ain-soph Could you please let me know if there are any updates on this?

1171000712 commented 1 year ago

Not sure if it's the trojanzoo version issue because I update that library a lot after this repo get published. Will test it.

Could you please let me know the exact version of trojanzoo you used for attacking the model?

@ain-soph Could you please let me know if there are any updates on this?

I have no idea how to solve the problem. Can you please provide the specific version of trojanzoo corresponding to this repository?

ain-soph commented 1 year ago

Sorry for the latency. Had a fever recently. I'll see if I can figure out this weekend.

ain-soph commented 1 year ago

_stateless.functional_call has been deprecated in new pytorch versions. I'm currently replacing it with torch.func.functional_call.

As for the 10% ASR, that seems to be related to the attack itself, because the attack shall work in some extent for any networks. I'm now testing it on a ResNet to see if it's the case.

lovian14 commented 1 year ago

_stateless.functional_call has been deprecated in new pytorch versions. I'm currently replacing it with torch.func.functional_call.

As for the 10% ASR, that seems to be related to the attack itself, because the attack shall work in some extent for any networks. I'm now testing it on a ResNet to see if it's the case.

Thanks for checking this. I installed an older version of trojanzoo, the ASR can be improved to be around 40%, but it is still far from the ASR in the paper

ain-soph commented 1 year ago

Attack on ResNet is good on my side. Everything is up-to-date. Running script: https://github.com/ain-soph/trojanzoo/blob/d5085fd01d6923108861601aaf483f1c29050843/trojanvision/attacks/backdoor/dynamic/input_aware_dynamic.py#L4 I'll check some archs on NATS-Bench tomorrow.

image
lovian14 commented 1 year ago

Attack on ResNet is good on my side. Everything is up-to-date. Running script: https://github.com/ain-soph/trojanzoo/blob/d5085fd01d6923108861601aaf483f1c29050843/trojanvision/attacks/backdoor/dynamic/input_aware_dynamic.py#L4 I'll check some archs on NATS-Bench tomorrow.

image

Thanks for checking. In table 1 of the paper, the ASR for ResNet18 is 59.73%. However, in the snapshot, the validation ASR is 84.41%. Can you clarify why there is such a big difference or do I missing something? Thanks!!

Also, when you attack the model, did you fix the model weights and only train the generators or you train both the model and the generators?

ain-soph commented 1 year ago

I just take a check. I forgot to add --natural in the previous snapshot to freeze the model parameters as what I did in nas search. So it's certainly higher ASR.
Will update a new run here. Sorry for making such a mistake since it's over years. I need to remind what I did in the past...

1171000712 commented 1 year ago

I just take a check. I forgot to add --natural in the previous snapshot to freeze the model parameters as what I did in nas search. So it's certainly higher ASR. Will update a new run here. Sorry for making such a mistake since it's over years. I need to remind what I did in the past...

Hi, thanks for checking. Could you please provide the data in figure 2 in the paper (the ASR value of each arch) as a replacement?