OpenMined / PyDP

The Python Differential Privacy Library. Built on top of: https://github.com/google/differential-privacy
Apache License 2.0
500 stars 138 forks source link

Numerical Instability in Tests #365

Open madhavajay opened 3 years ago

madhavajay commented 3 years ago

Description

I have noticed one of the tests flaps and sometimes fails, I guess its due to some numerical instability? https://github.com/OpenMined/PyDP/runs/2679233108?check_suite_focus=true

Screen Shot 2021-05-27 at 12 21 47 pm
______ TestLaplaceDistribution.test_check_statistics_for_geo_unit_values _______
[gw1] linux -- Python 3.6.13 /home/runner/.cache/pypoetry/virtualenvs/pydp-KC8mieZ4-py3.6/bin/python

self = <test_distributions.TestLaplaceDistribution object at 0x7fe191d630b8>

    def test_check_statistics_for_geo_unit_values(self):

        ld = LaplaceDistribution(epsilon=1.0, sensitivity=1.0)
        samples = [ld.sample(scale=1.0) for _ in range(k_num_geometric_samples)]
        mean = dp.util.mean(samples)
        var = dp.util.variance(samples)

        assert expect_near(0.0, mean, 0.01)
        assert expect_near(2.0, var, 0.1)
        assert expect_near(0.0, skew(samples, mean, math.sqrt(var)), 0.1)
>       assert expect_near(3.0, kurtosis(samples, mean, var), 0.1)
E       assert False
E        +  where False = expect_near(3.0, 3.1097853317836956, 0.1)
E        +    where 3.1097853317836956 = kurtosis([-0.18739762318364228, 0.19336953977835947, 2.2439366761800557, -0.120824026083028, 0.3653907802163303, 2.181137632465834, ...], 0.00018447984155104587, 2.0099539175529073)

How to Reproduce

See CI

Expected Behavior

Not failing.

Screenshots

See above

System Information

See Link

Additional Context

See above link