Closed persistz closed 3 years ago
Yes, you are right. The complexity of CAA is 800 calculated by: 50 9 + 25 9 + 125. This complexity is the sum of backward times when attack the target model.
In our implementation, we do attack twice for getting the both policy output and sub-policy output. However, this process can be combined to a single run: Suppose that we have A and B attacker, we can feed x
to A and (A(x)
, x
) to B. And the complexity is A+B. Same for when we have A,B,C, we can feed x
to A and (A(x)
, x
) to B and (B(A(x))
, B(x)
, x
) to C, and the complexity is A+B+C.
You can modify the code to implement the way of attack mentioned above. It will lead more memory consumption. And the sum of backward times is still: 50 9 + 25 9 + 125 = 800.
Thank you for your reply. Do you mean complexity is the sum of backward times, not the total number of images for gradient calculation? For example, if the user feeds (A(x), x) to B, this operation actually performs 2 image gradient calculations, but under the definition of complexity, it is regarded as one time.
If the answer to the previous comment is yes, then the evaluation of other methods will seem a little strange.
Using `100-step PGD with 10 restarts' as an example, the starting point of each restart in PGD paper is the original image. Therefore, the user can perform these 10 restart operations in parallel and pass in (x1, x2,..., x10) to the attack method, so that the complexity is only 100. The comparison complexity of PGD is not the optimal one, it seems a bit unfair.
If the answer to the previous comment is yes, then the evaluation of other methods will seem a little strange.
Using `100-step PGD with 10 restarts' as an example, the starting point of each restart in PGD paper is the original image. Therefore, the user can perform these 10 restart operations in parallel and pass in (x1, x2,..., x10) to the attack method, so that the complexity is only 100. The comparison complexity of PGD is not the optimal one, it seems a bit unfair.
Yes, you are right. Considering `100-step PGD with 10 restarts' as complexity of 100 is fair. So we think we should modify it.
Got it. However, CAA's performance advantage over other SOTA methods is still obvious. Closed.
In the paper, authors mentioned the complexity of CAA is 800, when using [('MT-LinfAttack', =8/255, t=50), ('MT-LinfAttack', =8/255, t=25), ( 'CWLinfAttack', =8/255, t=125)].
I think the calculation formula is: 50 9 + 25 9 + 125 = 800
But in the code, when the attack method idx is not zero, you use:
which means that in subsequent attacks, the same kind of attack is executed twice, so why the correct complexity is not: 50 9 + 25 9 2 + 125 2 = 1150?