Low watermark success on extracted model

[ClaudiaRaffaelli]

Hello, I tried to reproduce the results reported in the article using tensorflow==1.14.0 as recommended. In particular I ran your original code, training on MNIST and default hyperparameters (python train.py --dataset mnist --default 1) and got the following results:

Victim Model || validation accuracy: 0.9899259870871902, watermark success: 1.0
Extracted Model || validation accuracy: 0.9869449017569423, watermark success: 0.004720052083333333

Running a few more times the results continue to oscillate around these values.

I have done the same for Fashion MNIST (train.py --dataset fashion --default 1) obtaining analogous results:

Victim Model || validation accuracy: 0.9912860579788685, watermark success: 1.0 
Extracted Model || validation accuracy: 0.9904847759753466, watermark success: 0.0015285326086956522

Do you have any idea what the problem might be? Thanks.

[ClaudiaRaffaelli]

Turns out that since the files that make up the two datasets MNIST and Fashion MNIST share the same filenames, I had accidentally generated both .pkl files with the script prepare_mnist using the same four files both times.