mnist_dpsgd_tutorial_keras.py does not learn/achieve reported accuracy

[jpgard]

When running the mnist_dpsgd_tutorial_keras.py with all default parameters, the train/test accuracies do not improve over the course of training. The final accuracy is around 10%, which is of course equivalent to random guessing on MNIST.

This seems related to issues about training in eager mode, e.g. #87 .

Also, I see that the main README currently suggests using TensorFlow 2.4, but I don't even see a public release of TensorFlow 2.4 yet at https://github.com/tensorflow/tensorflow/releases .

As a side note, the training steps are extremely fast when running the buggy version of the script vs. using the workaround below. This suggests to me that the gradients aren't being fully computed or applied.

Expected behavior: based on the results table in the tutorials README, I would expect accuracy of at least 95%.

Steps to reproduce: Install tensorflow and tensorflow privacy; specifically I am using tensorflow==2.3.1 and tensorflow_privacy==0.5.1. Then run python3 mnist_dpsgd_tutorial_keras.py.

Workaround: Insert tf.compat.v1.disable_eager_execution() in the line immediately following import tensorflow as tf. Of course, this means we lose all of the benefits of eager execution, but training works as expected.

Sample of output from mnist_dpsgd_tutorial_keras.py:

Epoch 1/60
240/240 [==============================] - 9s 39ms/step - loss: 1027.9260 - accuracy: 0.1020 - val_loss: 7.2073 - val_accuracy: 0.0982
Epoch 2/60
240/240 [==============================] - 9s 36ms/step - loss: 7.1395 - accuracy: 0.1008 - val_loss: 6.1844 - val_accuracy: 0.1028

...

Epoch 59/60
240/240 [==============================] - 9s 36ms/step - loss: 6.8704 - accuracy: 0.1007 - val_loss: 6.2492 - val_accuracy: 0.0958
Epoch 60/60
240/240 [==============================] - 9s 37ms/step - loss: 6.3550 - accuracy: 0.1010 - val_loss: 6.7348 - val_accuracy: 0.0958
For delta=1e-5, the current epsilon is: 66419.90

[fraboeni]

Same issue for me when running the script (or any other training) with tensorflow 2.4.1.

[guoJohnny]

Same issue for me .,but I changed the layers of convolutional network like this:

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

Trained and Tested accuracy will be nearly 96%. But I don't know why code in ' mnist_dpsgd_tutorial_keras.py ' got low accuracy.

[guoJohnny]

Same issue for me .,but I changed the layers of convolutional network like this:

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

Trained and Tested accuracy will be nearly 96%. But I don't know why code in ' mnist_dpsgd_tutorial_keras.py ' got low accuracy.

the effect of this code like this:

[guoJohnny]

Same issue for me .,but I changed the layers of convolutional network like this:

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

Trained and Tested accuracy will be nearly 96%. But I don't know why code in ' mnist_dpsgd_tutorial_keras.py ' got low accuracy.

the effect of this code like this:

Finally I changed the version of tensorflow and tensorflow_privacy, and used the convolutional network of mnist_dpsgd_tutorial_keras.py , got the learn/achieve accuracy.

tensorflow-gpu 2.2.0 -> 2.4.1, tensorflow_privacy 0.4.0 -> 0.5.1

Maybe you can check your version of tensorflow_privacy.