Evasion attacks (DeepFool & possibly others) do not support multiple inputs

[zacps]

Is your feature request related to a problem? Please describe. I have a tensorflow model which utilizes multiple inputs which are of varying shape. Specifically I have a video input of shape (instances, timesteps, width, height, channels)=(None,45,32,32,3) and a secondary input of (instances, timesteps, features)=(None, 45, 9). As these inputs do not have consistent dimensions it is not possible to reduce them to a single input.

Attempting to run DeepFool (and I suspect any other attack) on this model produces the error:

AttributeError                            Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_14284/979395968.py in <module>
      1 clf = TensorFlowV2Classifier(model, model.layers[-1].output.shape[-1], model.layers[0].input.shape)
      2 attack = DeepFool(clf)
----> 3 attack.generate([X_val_padded, X_val_mvm])

~\AppData\Local\pypoetry\Cache\virtualenvs\venv\lib\site-packages\art\attacks\evasion\deepfool.py in generate(self, x, y, **kwargs)
     99         :return: An array holding the adversarial examples.
    100         """
--> 101         x_adv = x.astype(ART_NUMPY_DTYPE)
    102         preds = self.estimator.predict(x, batch_size=self.batch_size)
    103 

AttributeError: 'list' object has no attribute 'astype'

Describe the solution you'd like I would like support for multiple input models to be added to evasion attacks where possible.

Describe alternatives you've considered I am not aware of any possible workarounds.

[beat-buesser]

Hi @zacps Great to hear from you again! That's correct, the tools of ART are, at the moment, assuming a single input tensor. As you describe this allows multi-input models as long as their inputs can be stacked into a single input tensor. I agree with you about the usefulness of supporting multiple inputs in multi-modal scenarios with different shapes.

Do you have a proposal for a general solution? Would you be interested to work on a solution?

[zacps]

Hi @beat-buesser :)

A general solution might be hard. One approach which could work for attacks which don't assume 2+ dimensional input shapes is:

1. Flatten all input tensors into shape (None, product of all other dimensions)
2. Concatenate all input tensors into (None, product of flattened tensor dimensions)
3. Proceed through the attack as normal
4. When interacting with the classifier:
   1. Reshape the flattened tensor into the original shapes
   2. Pass the tensors into the model predict/etc
5. At the end of the attack reshape our result (which should be a flattened tensor) into the original shapes

I think this should work as long as the attack doesn't depend on the shape information, like some of the image/video attacks do(?).

I would be interested in working on a solution, but my time is quite limited at the moment so I'm not sure if I'll get around to it.