Use of CheckList framework to generate counterfactuals

[MartaMarchiori]

If I wanted to use CheckList to automate the generation of counterfactuals from a sample text, how could I do it? My reflection:

• templates and masking token for suggestions and replacement are both user-made operation, hand-coded;
• the perturbation functions are "automatic", but I can't be sure that the entity to be perturbed is present in the text, so I would have to run multiple perturbation functions, thus risking overlapping perturbations and distorting too much the input...

I am also studying your new model, Polyjuice, to see if it can be more suitable but I am still exploring and trying to figure out how to use/integrate it within my work The aspect that makes me lean towards the use of CheckList is the conceptual framework behind it and the possibility of generating examples that test specific language skills

Really thank you for your work and suggestions!

[MartaMarchiori]

PS: I'm also considering the CheckList recipe implemented in https://github.com/QData/TextAttack, but I would like to fully embrace CheckList framework and perturb/create samples exploring/starting from linguistic capabilites, not limiting the generation to Robustness or NER, as performed in TextAttack

[marcotcr]

If I wanted to use CheckList to automate the generation of counterfactuals from a sample text, how could I do it?

I guess CheckList isn't really made to do generic 'generation of counterfactuals'. Each perturbation should be tied to a test, with an expectation function (e.g. 'add typos, prediction shouldn't change').

If you want to generate a bunch of counterfactuals to explore the data, Polyjuice would be more suitable. However, if you want to codify a certain perturbation into a test, I think CheckList is more suitable. If you want to test for adversarial robustness, then TextAttack is what you should be using. It really depends on what you're doing :)

the perturbation functions are "automatic", but I can't be sure that the entity to be perturbed is present in the text, so I would have to run multiple perturbation functions, thus risking overlapping perturbations and distorting too much the input...

You should probably run one perturbation function at a time, for each specific test.