what tensorflow_privacy version are you using when you are using the tensorflow==1.4.0 and python==3.5?

[meiyor]

Hi @fjxmlzn I'm still trying to reproduce your generation of data for WWT and the best case scenario I obtained is this for eps=1.8E8

Can you specify me what version of tensorflow_privacy did you use when you test your evaluation for tensorflow==1.4.0 and python==3.5? For the example I'm attaching here the tensorflow_privacy is 0.4.0, for the the tensorflow_privacy>0.6.0 a version of tf>=2.0 is needed, again can you let me know what version are you using for obtaining your results exactly?

[fjxmlzn]

Sorry that the environment I used was cleared as I graduated, and I don't remember which version I used. But it's possible that even with such a big EPS the fidelity is still bad, as GAN training is already not stable and the gradient clipping in DP-SGD training (if not well-tuned) could make it even worse. The takeaway message in the paper about this experiment is that "DP-GAN mechanisms require significant improvements for privacy-preserving time series generation."

[meiyor]

Hi @fjxmlzn, the thing is that I have seen that implementing with different versions of tf, python and tf_privacy gives completely different results, for any combination of them. Have you tried to replicate the dp_training and dp_generating results with tf==1.14 or tf==1.15? For that case it is possible to install tf_privacy versions between 0.4.0 and 0.6.0, did you obtain the same plots? If you can tell me exactly what combination of versions did you obtain your plots, I will be really glad to try to replicate that. Again as I try to install a newer version of tensorflow_privacy from pip or conda, it force me to install a newer version of tensorflow, so it is a thing to know what is the best combination of versions. Please if you can inform me that I will be really glad.

[meiyor]

@fjxmlzn, I have other question, to reproduce the Figure 1 did you run the training till epoch 15, or you run the training for the whole 400 epochs? How did you do exactly to generate the Figure 1? Can you explain me in detail how the users in this issue replicate Figure 1? https://github.com/fjxmlzn/DoppelGANger/issues/22 I really want to know all the details.

There says that you are using TF Privacy 0.5.1 and tensorflow 1.14.0, can you reconfirm me that?

[fjxmlzn]

TF version: When I developed the code, I think I used TF 1.4.0 as I listed in the readme. But I think I also checked TF 1.4.0 - 1.15 after the code release and they all worked (which is also stated in the README)

TF-privacy version: As I said previously, I don't have access to the code environment I used previously so I am not able to check the exact version I used. But since I said TF Privacy 0.5.1 in #22, I guessed that's the version I used.

Figure 1: Please note that Figure 1 is non-DP. Not that by saying "non-DP", I meant that you should use either https://github.com/fjxmlzn/DoppelGANger/tree/master/example_training or https://github.com/fjxmlzn/DoppelGANger/tree/master/example_training(without_GPUTaskScheduler), INSTEAD OF https://github.com/fjxmlzn/DoppelGANger/tree/master/example_dp_training. The non-DP code does NOT need TF Privacy at all. In this reply I talked about the hyper-parameters I used to generate the results in Figure 1: https://github.com/fjxmlzn/DoppelGANger/issues/22#issuecomment-859697824

Figure 16: The only DP-related plot in the paper is Figure 16. As you can see in Figure 16, even with epsilon=1e8, the fidelity is not good. Although the shape of the autocorrelation curves in Figure 16 and what you got are different, they are both bad. I remembered that in the DP experiments, the training is very unstable and the variance across different runs is high, so the different curves might not be due to different versions of TF or TF-privacy, but might be due to the inherent instability in DP GAN training. (You can run your code twice with the same version of TF and TF-privacy and see how different the results are.) About why DP GANs give bad results, I explained it in this reply: https://github.com/fjxmlzn/DoppelGANger/issues/49#issuecomment-2226794056

[meiyor]

Hi @fjxmlzn, I inform you here that I can replicate Figure 16 and Figure 1 from tensorflow version 1.14.0 and tensorflow_privacy version 0.5.1. This is what I obtained for eps=1.86E8. Now after we run the evaluation without the DP approach, we obtained the following gifs.

This for sample-len=5: as you can see this, it is not so steady and it is a bit noisy in the convergence process.

This is for sample-len=10: as you can se here, it is more stability in the fidelity (MSE) convergence in comparison with sample-len=5.

For me, I can say we have replicated your results on the web dataset. If you want you can close this issue now :).

As I mentioned previously, I have tested Doppelganger for tf version >=2.0. For this particular case, I can't replicate your results because of obvious reasons: (1) tensorflow_privacy version matching and (2) some tf commands that are not exactly equivalent with previous versions - in terms of some input parameters and given outputs. We will need to do more extensive tests on that. If you consider I can push the version I have for tf>=2.0 and we can analyze it together, so we can see if we can obtain similar results with time. Ofc, I won't take all my time on that, but I will try to test the code with keras or compat.v1 benchmarks. Let me know what you think.

Thank you!

[fjxmlzn]

That's great! Thank you for all the efforts and it is great to hear that you were able to reproduce the results!

I think it would be helpful to the community to put your tf>=2.0 version somewhere, and other people can refer to that too :) Previously @ yzion helped make a TF2 version of DoppelGANger in this branch: https://github.com/fjxmlzn/DoppelGANger/tree/TF2 But I am not sure if it also has the issues you described above.

[meiyor]

Thank you very much @fjxmlzn I sent you an email (to the address you have in Github) if you agree, we can keep contacting there.