Results differ from paper on AG news dataset

[RishabhMaheshwary]

I got the following result on AG news:

For target model bert: original accuracy: 94.200%, adv accuracy: 24.400%, avg changed rate: 24.965%, num of queries: 446.3

There are slight modifications in the attack_classification.py file from the orignal one, but I have not changed the logic at all. You can see the revisions in the gist, the first one is your code and the second revision shows my changes.

Is the accuracy drop because of the difference between computation of cosine_similarity_matrix between the two files ? I used the implementation of compute_cos_sim_mat.py file in the attack_classification file because it fits in the memory.

Thanks!

[RishabhMaheshwary]

is it because of the semantic similarity? In paper it is 0.57 and in the code it is 0.7.

[jind11]

hi, thanks for pointing this out! The default value for "--sim_score_threshold" in this code is set to 0.7, after running my code using this default value, I got the same numbers you mentioned in this issue. In my original experiments, I changed its value from 0.7 to 0.5, and after running my code with this setting, I got the numbers of "For target model bert: original accuracy: 94.200%, adv accuracy: 15.300%, avg changed rate: 23.292%, num of queries: 384.1". So this hyperparameter does control the final adversary accuracy.

[RishabhMaheshwary]

Thanks !