AstarVienna / ScopeSim

A telescope observation simulator for Python.
GNU General Public License v3.0
16 stars 10 forks source link

Make tests deterministic #329

Open hugobuddel opened 10 months ago

hugobuddel commented 10 months ago

Some tests can randomly fail, e.g.

https://github.com/AstarVienna/ScopeSim/actions/runs/7443459990/job/20248391071

================================== FAILURES ===================================
________ TestApplyTo.test_noise_reduces_with_square_root_of_ndit[9-1] _________

self = <scopesim.tests.tests_effects.test_BasicReadoutNoise.TestApplyTo object at 0x0000027593CEF8B0>
ndit = 9, noise_std = 1

    @pytest.mark.parametrize("noise_std", [1, 9, 100])
    @pytest.mark.parametrize("ndit", [1, 9, 100])
    def test_noise_reduces_with_square_root_of_ndit(self, ndit, noise_std):
        dtcr = _basic_detector(width=256)
        ron = PoorMansHxRGReadoutNoise(noise_std=noise_std, n_channels=64, ndit=ndit)
        dtcr = ron.apply_to(dtcr)
        noise_real = np.std(dtcr._hdu.data)

>       assert noise_real == approx(noise_std * ndit**0.5, rel=0.1)
E       assert 3.3188548593454184 == 3.0 � 3.0e-01
E         comparison failed
E         Obtained: 3.3188548593454184
E         Expected: 3.0 � 3.0e-01

scopesim\tests\tests_effects\test_BasicReadoutNoise.py:36: AssertionError

It would be better to make these deterministic.

PoorMansHxRGReadoutNoise can be made deterministic by passing the random_seed parameter, and np.std probably has something similar.

teutoburg commented 10 months ago

Yes, I've encountered cases like this before. Usually pass if run again. But indeed, deterministic should be the goal! Perhaps we can fix this in the course of #298