[Feature] Add a command to save all transformed query attacks

[threemmm]

A command to save all transformed query attacks with their details such as the model output confidence is needed for people who work on attack detection systems especially for the black-box attacks in which they need a sequence of all queries that an attacker makes to fool the model. I'm working on algorithms to detect the attacks and I need sequences of attack queries, so a command for it could be useful.

[jxmorris12]

Hey @threemmm -- thanks for the suggestion! To be specific: you're suggesting a feature in which we save intermediate transformed texts and their scores to a file, not just the last transformation, right?

[threemmm]

Hey @jxmorris12 , thank you for your awesome package. Yes, exactly. I did it by adding "--save-sequences-csv" command, and I tried to follow your structure to save them (adding a new class to logger, and save into a file by flush() at the end of the process). However, it doesn't work well with all search methods, as I return the result of "get_goal_results()" in addition to the final perturbed result in search methods to quickly meet my need. I tested with textbugger recipe and albert-base-v2-sst2 , and the output was as the photo below --recipe textbugger --model albert-base-v2-sst2 --save-sequences-csv --num-examples 3

I'd like to hear your thoughts.

[github-actions[bot]]

Stale issue message

[qiyanjun]

Good enhancement to add.. Will update.