Implementation of "query"

[sullendhy]

Thank you for your wonderful job and code. when I try to reproduce your result, i can not find the result of "query" metric. So what does the "query" metric mean in the paper. And i will appreciate it if you can tell me the specific implementation of calculating "query". Thanks again for your nice job !

[cg563]

Thank you for your interest in our paper!

Query refers to the number of black-box queries to the target model before successfully attacking it. This is only relevant for the transfer attack experiment. You can extract the number of queries from the result of the evaluate_adv_samples.py as follows.

• The script stores a result file that contains the tensor all_corr in https://github.com/facebookresearch/text-adversarial-attack/blob/main/evaluate_adv_samples.py#L216. The (i,j)th entry indicates whether the model correctly classified the j-th adversarial example for the i-th clean sample or not.
• For each sample i, if the first occurrence of 0 (i.e. incorrect classification) in the i-th row of all_corr happens at (i,j) then (j+1) queries were needed to successfully attack the sample. You can compute a mean over a batch of samples to get an average query count.

Hope this helps.