search

apple / pfl-research

Simulation framework for accelerating research in Private Federated Learning
http://apple.github.io/pfl-research/
Apache License 2.0
285 stars 27 forks source link

Speed up privacy tests and re-enable them #26

Closed grananqvist closed 7 months ago

grananqvist commented 7 months ago

don't sample many random numbers by calling the mechanism 1000s of times, just make the dimensions larger and have the same values.

before improvement:

158.79s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-2.0-5000]
157.43s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-5000]
156.22s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-5000]
155.62s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-5000]
155.05s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-5000]
153.19s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-5000]
151.11s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-5000]
150.11s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10000-1000-numpy_ops]
149.95s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-5000]
148.77s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-5000]
148.58s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-5000]
148.00s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10000-1000-numpy_ops]
138.51s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[1000-10000]
130.80s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10000-1000-numpy_ops]
127.47s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-False-tensorflow_ops]
119.91s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-tensorflow_ops]
116.51s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-True-tensorflow_ops]
108.36s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-True-pytorch_ops]
107.74s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-pytorch_ops]
106.75s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-False-pytorch_ops]
105.58s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-False-numpy_ops]
79.60s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-False-numpy_ops]
73.31s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-False-tensorflow_ops]
69.09s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-2.0-5000]
67.39s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-True-numpy_ops]
48.97s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-5.0-5000]
31.37s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-numpy_ops]
24.57s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-tensorflow_ops]
24.12s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-tensorflow_ops]
20.53s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-False-pytorch_ops]
20.18s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-tensorflow_ops]
20.17s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-tensorflow_ops]
19.79s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-tensorflow_ops]
18.93s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-tensorflow_ops]
18.32s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10000-100-numpy_ops]
17.85s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[100-10000]
16.33s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10000-100-numpy_ops]
14.12s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-tensorflow_ops]
13.95s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10000-100-numpy_ops]
13.10s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-tensorflow_ops]
12.66s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-tensorflow_ops]
12.08s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-pytorch_ops]
11.12s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-pytorch_ops]
10.83s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-numpy_ops]
10.25s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-pytorch_ops]
9.46s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-pytorch_ops]
8.63s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-tensorflow_ops]
7.66s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-numpy_ops]
4.97s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-numpy_ops]
4.97s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-100]
4.94s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-100]
4.31s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-100]
4.26s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-numpy_ops]
4.22s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-1000-1000-numpy_ops]
4.15s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-20]
4.03s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-numpy_ops]
3.92s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-100]
3.86s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-100]
3.72s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-20]
3.71s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_constrain_sensitivity
3.69s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-pytorch_ops]
3.68s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-20]
3.67s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-100]
3.51s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-numpy_ops]
3.50s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-20]
3.45s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-100]
3.42s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-numpy_ops]
3.37s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-2.0-100]
3.37s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-20]
3.35s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-1000-1000-numpy_ops]
3.26s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-1000-1000-numpy_ops]
3.23s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-pytorch_ops]
3.18s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-pytorch_ops]
3.13s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-20]
2.90s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_optimal_gamma
2.84s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-2.0-20]
2.77s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-pytorch_ops]
2.45s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-100]
2.36s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-2.0-100]
2.35s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-20]
2.27s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-20]
1.86s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-100-1000-numpy_ops]
1.85s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-20]
1.70s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-2.0-20]
1.68s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-100-1000-numpy_ops]
1.68s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-numpy_ops]
1.67s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-100-1000-numpy_ops]
1.56s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-numpy_ops]
1.55s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[10-10000]
1.48s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10-1000-numpy_ops]
1.47s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-pytorch_ops]
1.45s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10-1000-numpy_ops]
1.40s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-pytorch_ops]
1.38s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10000-10-numpy_ops]
1.34s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-100]
1.32s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10000-10-numpy_ops]
1.30s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-numpy_ops]
1.16s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10000-10-numpy_ops]
1.07s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10-1000-numpy_ops]
0.86s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-tensorflow_ops]
0.82s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-1000-10-numpy_ops]
0.82s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[1000-1000]
0.73s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-500-1000]
0.73s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-500-1000-numpy_ops]
0.72s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-500-1000-numpy_ops]
0.69s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-tensorflow_ops]
0.69s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[4-0.5-500-1000]
0.65s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-20-1000-numpy_ops]
0.61s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-5.0-20]
0.55s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-5.0-100]
0.55s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-20-1000]
0.54s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[4-0.5-20-1000]
0.54s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-tensorflow_ops]
0.52s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-20-1000-numpy_ops]
0.45s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-1000-100-numpy_ops]
0.44s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-tensorflow_ops]
0.41s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[1000-10]
0.28s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-1000-100-numpy_ops]
0.27s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-1000-100-numpy_ops]
0.26s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-tensorflow_ops]
0.25s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[0.1-tensorflow_ops]
0.24s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1000.0-tensorflow_ops]
0.22s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1.0-tensorflow_ops]
0.22s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10-100-numpy_ops]
0.22s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10-100-numpy_ops]
0.21s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-tensorflow_ops]
0.20s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-tensorflow_ops]
0.16s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-100-100-numpy_ops]
0.16s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[4-0.5-500-100]
0.16s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[100-1000]
0.16s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-100-100-numpy_ops]
0.16s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10-100-numpy_ops]
0.14s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[1-10000]
0.13s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1.0-tensorflow_ops]
0.13s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[1000-100]
0.12s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10-10-numpy_ops]
0.11s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_get_gamma
0.11s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1000.0-tensorflow_ops]
0.11s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[0.1-tensorflow_ops]
0.09s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_theoretical_error
0.07s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-100-100-numpy_ops]
0.07s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-20-100-numpy_ops]
0.06s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-20-100-numpy_ops]
0.06s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-1000-10-numpy_ops]
0.05s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-numpy_ops]
0.04s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-500-100-numpy_ops]
0.03s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-20]
0.03s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_variance[0.1-100]
0.03s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-100]
0.02s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-1000-10-numpy_ops]
0.02s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-500-100]
0.02s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-100]
0.01s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-100]
0.01s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[10-1000]
0.01s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-tensorflow_ops]
0.01s setup    tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10000-10-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-500-100-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10-10-numpy_ops]
0.01s teardown tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-5000]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-100-10-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-100-10-numpy_ops]
0.01s teardown tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-20-100-numpy_ops]
0.01s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[100-100]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-20-100]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[4-0.5-20-100]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10-10-numpy_ops]

(416 durations < 0.005s hidden.  Use -vv to show these durations.)
============================================================================================================================================================================= 194 passed, 66 warnings in 463.15s (0:07:43) ==============================================================================================================================================================================

real    7m43.804s
user    24m53.229s
sys     49m37.814s

After improvement:

3.96s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-numpy_ops]
3.69s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-True-pytorch_ops]
3.43s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-True-numpy_ops]
2.66s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-True-tensorflow_ops]
2.49s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-tensorflow_ops]
1.74s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-numpy_ops]
1.67s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-5000]
1.61s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-5000]
1.51s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-5000]
1.49s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-5000]
1.37s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-pytorch_ops]
1.34s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-5000]
1.32s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-2.0-5000]
1.32s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-5000]
1.31s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-tensorflow_ops]
1.27s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-5000]
1.25s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-5000]
1.22s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_optimal_gamma
1.19s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-5000]
1.18s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-2.0-5000]
1.06s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-False-numpy_ops]
1.05s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-1000-20-numpy_ops]
1.03s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-1000-20-numpy_ops]
1.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-1000-20-numpy_ops]
0.97s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-False-tensorflow_ops]
0.88s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-tensorflow_ops]
0.73s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-tensorflow_ops]
0.73s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-numpy_ops]
0.68s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-tensorflow_ops]
0.68s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-pytorch_ops]
0.66s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-False-numpy_ops]
0.62s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-tensorflow_ops]
0.60s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_constrain_sensitivity
0.59s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-1000-20-numpy_ops]
0.56s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-pytorch_ops]
0.56s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-pytorch_ops]
0.54s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-5.0-5000]
0.53s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-numpy_ops]
0.51s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-tensorflow_ops]
0.50s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-True-pytorch_ops]
0.46s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-1000-20]
0.46s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[4-0.5-1000-20]
0.45s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-tensorflow_ops]
0.44s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-tensorflow_ops]
0.44s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-tensorflow_ops]
0.42s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-True-tensorflow_ops]
0.39s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-tensorflow_ops]
0.38s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-False-pytorch_ops]
0.36s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-1000-20-numpy_ops]
0.35s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-pytorch_ops]
0.33s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-numpy_ops]
0.33s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-False-tensorflow_ops]
0.32s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-numpy_ops]
0.26s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-numpy_ops]
0.25s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-numpy_ops]
0.25s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-tensorflow_ops]
0.24s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-numpy_ops]
0.24s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-True-tensorflow_ops]
0.24s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-pytorch_ops]
0.23s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-True-tensorflow_ops]
0.23s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[0.1-tensorflow_ops]
0.23s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-tensorflow_ops]
0.22s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-tensorflow_ops]
0.22s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-pytorch_ops]
0.22s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-tensorflow_ops]
0.21s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-numpy_ops]
0.21s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1000.0-tensorflow_ops]
0.21s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[6000000.0-True-pytorch_ops]
0.21s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1.0-tensorflow_ops]
0.21s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_priv_unit_mechanism[0.02-False-pytorch_ops]
0.19s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.5-0.1-False-pytorch_ops]
0.16s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_theoretical_error
0.15s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[10-1000]
0.14s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-2.0-100]
0.14s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10-1000-numpy_ops]
0.14s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10-1000-numpy_ops]
0.14s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-False-pytorch_ops]
0.13s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[10000-1]
0.13s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1.0-tensorflow_ops]
0.12s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-2.0-100]
0.11s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-100-20-numpy_ops]
0.11s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-20]
0.11s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-2.0-20]
0.11s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[1000.0-tensorflow_ops]
0.10s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping[0.1-tensorflow_ops]
0.10s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-20]
0.10s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-100-20-numpy_ops]
0.10s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.6-100]
0.09s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-20]
0.08s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-100]
0.08s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-20]
0.08s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[1000-10]
0.08s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-100]
0.08s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-2.0-100]
0.07s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-5.0-20]
0.07s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-100]
0.07s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-20]
0.07s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-20]
0.06s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-100-20-numpy_ops]
0.06s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-20]
0.06s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[2.0-4.0-10-20-numpy_ops]
0.05s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_mean_expectation[100-100]
0.05s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[0.02-1.0-False-numpy_ops]
0.05s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-100]
0.05s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-5000]
0.05s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-100]
0.05s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-100-20-numpy_ops]
0.04s call     tests/internal/test_priv_unit_functions.py::TestPrivUnit::test_get_gamma
0.04s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-5.0-100]
0.03s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.6-100]
0.03s call     tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_norm_clipping_only[inf-3.01-expected_arrays3-0.0-3.01]
0.03s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-100]
0.03s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-5.0-100]
0.03s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[6000000.0-False-pytorch_ops]
0.02s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10-1000-numpy_ops]
0.02s setup    tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-0.0-5000]
0.02s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-10-20-numpy_ops]
0.02s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-100-20-numpy_ops]
0.02s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[oscillating-0.0-100]
0.01s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[one-hot-5.0-20]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-10-1000]
0.01s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_laplace_mechanism[0.02-False-tensorflow_ops]
0.01s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.6-100]
0.01s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-2.0-20]
0.01s call     tests/privacy/test_priv_unit_mechanism.py::TestPrivUnitMechanism::test_l2_error[random-0.0-20]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[2.0-5.0-100-20]
0.01s setup    tests/privacy/test_privacy_mechanism.py::TestMechanisms::test_gaussian_mechanism[6000000.0-1.0-True-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_laplace_mechanism_approximation[4-0.5-100-20]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[4-0.5-10-1000-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[6.0-4.0-10-20-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_priv_unit_mechanism_approximation[4.0-4.0-10-20-numpy_ops]
0.01s call     tests/privacy/test_approximate_mechanism.py::TestApproximateMechanism::test_gaussian_mechanism_approximation[2.0-5.0-10-1000-numpy_ops]

(336 durations < 0.005s hidden.  Use -vv to show these durations.)
=================================================================================================================================================================================== 156 passed, 6 warnings in 14.86s ====================================================================================================================================================================================

real    0m15.533s
user    1m20.162s
sys     1m0.860s

only 15 seconds!

rdar://122182309

grananqvist commented 7 months ago

I added back integration tests as well. changing to cohort size = 3 (sufficient for testing?) increases the test runtime by 1min. the majority of the time in test builds is now from installing horovod. We can later improve that with caching