attacking convolutional layer's gradient - shape mismatch

[cakirogluozan]

Hello everyone,

Firstly, thanks for the great work. I am trying to attack with ML-Privacy-Meter to a target model as in TF example model. You may see the model code in below.

However, when I try to exploit Convolutional Gradients, specifically the third Convolutional layer, I have a shape mismatch error as in below.

Traceback (most recent call last): File "tutorials/attack_alexnet.py", line 104, in attackobj.train_attack() File "/home/ml-privacy-meter/ml_privacy_meter/attack/meminf.py", line 518, in train_attack moutputs = self.forward_pass(model, mfeatures, mlabels) File "/home/ml-privacy-meter/ml_privacy_meter/attack/meminf.py", line 454, in forward_pass attack_outputs = self.attackmodel(self.inputArray) File "/home/ml-privacy-meter/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 998, in call input_spec.assert_input_compatibility(self.input_spec, inputs, self.name) File "/home/ml-privacy-meter/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py", line 271, in assert_input_compatibility raise ValueError('Input ' + str(input_index) + ValueError: Input 3 is incompatible with layer model: expected shape=(None, 64, 3, 64), found shape=(1, 3, 3, 64, 64)

I'd like to get help from you in this regard.

Thanks,

[2016312357]

I have the same problem. Did you figure this out?

[cakirogluozan]

I have the same problem. Did you figure this out?

@2016312357 Unfortunately, still waiting for a response.

[amad-person]

Hi @cakirogluozan @2016312357 could you let me know the environment you are running ML Privacy Meter in i.e. python version, tensorflow version, etc.

[cakirogluozan]

python 3.8.10 pip freeze output is here:

absl-py==0.12.0
astunparse==1.6.3
cachetools==4.2.2
certifi==2020.12.5
chardet==4.0.0
cycler==0.10.0
flatbuffers==1.12
gast==0.3.3
google-auth==1.30.0
google-auth-oauthlib==0.4.4
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
idna==2.10
joblib==1.0.1
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
Markdown==3.3.4
matplotlib==3.4.1
-e git+git@github.com:########@14d8f7cc1db89260ed019e316330ff9e40bd7592#egg=ml_privacy_meter
numpy==1.19.5
oauthlib==3.1.0
opencv-python==4.5.1.48
opt-einsum==3.3.0
Pillow==8.2.0
pkg-resources==0.0.0
protobuf==3.15.8
psutil==5.8.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7.2
scikit-learn==0.24.2
scipy==1.6.3
six==1.15.0
tensorboard==2.5.0
tensorboard-data-server==0.6.0
tensorboard-plugin-wit==1.8.0
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.1
termcolor==1.1.0
threadpoolctl==2.1.0
tqdm==4.60.0
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.25.8
Werkzeug==0.16.0
wrapt==1.12.1

After your response, I've also run the code in order to exploit 1. gradient of alexnet_pretrained (the only change in code is in tutorials/attack_alexnet.py: gradients_to_exploit=[1] line)

  attackobj = ml_privacy_meter.attack.meminf.initialize(
  target_train_model=cmodelA,
  target_attack_model=cmodelA,
  train_datahandler=datahandlerA,
  attack_datahandler=datahandlerA,
  layers_to_exploit=[26],
  gradients_to_exploit=*[1]*,
  device=None, epochs=10, model_name='blackbox1')

with the following environment:

Python 3.6.13

absl-py==0.13.0
astor==0.8.1
cached-property==1.5.2
cachetools==4.2.2
certifi==2021.5.30
charset-normalizer==2.0.4
cycler==0.10.0
dataclasses==0.8
gast==0.2.2
google-auth==1.35.0
google-auth-oauthlib==0.4.5
google-pasta==0.2.0
grpcio==1.39.0
h5py==3.1.0
idna==3.2
importlib-metadata==4.6.4
joblib==1.0.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
Markdown==3.3.4
matplotlib==3.3.4
-e git+https://github.com/privacytrustlab/ml_privacy_meter/@975ec702b2eee80b4ff94b22b3b6806e3904d496#egg=ml_privacy_meter
  numpy==1.19.5
  oauthlib==3.1.1
  opt-einsum==3.3.0
  Pillow==8.3.1
  protobuf==3.17.3
  pyasn1==0.4.8
  pyasn1-modules==0.2.8
  pyparsing==2.4.7
  python-dateutil==2.8.2
  requests==2.26.0
  requests-oauthlib==1.3.0
  rsa==4.7.2
  scikit-learn==0.24.2
  scipy==1.4.1
  six==1.16.0
  tensorboard==2.1.1
  tensorflow-estimator==2.1.0
  tensorflow-gpu==2.1.0
  termcolor==1.1.0
  threadpoolctl==2.2.0
  typing-extensions==3.10.0.0
  urllib3==1.26.6
  Werkzeug==2.0.1
  wrapt==1.12.1
  zipp==3.5.0

I have an another error:

Target model test accuracy 0.4412
/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/ml_privacy_meter/utils/attack_utils.py:147: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  hashval = hash(bytes(np.array(example)))
/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/ml_privacy_meter/utils/attack_utils.py:150: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  hashval = hash(bytes(np.array(example)))
WARNING:tensorflow:5 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fb7340de598> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
Traceback (most recent call last):
  File "tutorials/attack_alexnet.py", line 52, in <module>
    attackobj.train_attack()
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/ml_privacy_meter/attack/meminf.py", line 464, in train_attack
    moutputs = self.forward_pass(model, mfeatures, mlabels)
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/ml_privacy_meter/attack/meminf.py", line 403, in forward_pass
    self.get_gradients(model, features, labels)
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/ml_privacy_meter/attack/meminf.py", line 354, in get_gradients
    toappend = tf.reshape(grads[g], reshaped)
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 193, in reshape
    result = gen_array_ops.reshape(tensor, shape, name)
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 7436, in reshape
    tensor, shape, name=name, ctx=_ctx)
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 7463, in reshape_eager_fallback
    ctx=ctx, name=name)
  File "/home/cakiroglu@scch.at/test_ml/ml_privacy_meter/venv/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 23232 values, but the requested shape has 121 [Op:Reshape]

[asarmadi]

Hi, I have the same issue. My question is that the gradients of a CNN are 5D tensors, I'm wondering how to reshape them to feed into a CNN layer in the attack model? I checked your code, and it seems that in "cnn_for_cnn_gradients" the input shape of the CNN is (ch2,k1,ch1) when the input shape (i.e., the gradient of loss w.r.t. target model CNN parameters) is (k1,k2,ch1,ch2). However, in get_gradient function, the gradients are reshaped to (k1,k2,1).