search

privacytrustlab / ml_privacy_meter

Privacy Meter: An open-source library to audit data privacy in statistical and machine learning algorithms.
MIT License
604 stars 100 forks source link

Bug in notebook examples that use PyTorch models #95

Closed dxoigmn closed 3 weeks ago

dxoigmn commented 1 year ago

The PyTorch model in shadow_metric.ipynb uses nn.CrossEntropyLoss, which expects unnormalized logits. However, the model outs probabilities due to the use of nn.Softmax. This causes the model to not achieve 100% accuracy on the training set.

Additionally, criterions in PyTorch typically take arguments in the order of logits, targets. However, the code provides targets, logits. This, however, is not a functional concern because targets contains class probabilities (rather than class indices). It will probably become an issue once the bug above is fixed.

Both of these issues also exist in avg_loss_training_algo.ipynb

changhongyan123 commented 1 year ago

Hi @dxoigmn, thank you for bringing this issue to our attention. I'm pleased to inform you that we have fixed the bug and released an updated version of the tutorials. You can check the shadow_metric.ipynb and avg_loss_training_alg.ipynb files for the latest version.

If you have any questions or feedback, please let us know. Thank you for your support!