search

desihub / specsim

Quick simulations of spectrograph response
2 stars 9 forks source link

Script results are not reproducible #83

Closed dkirkby closed 6 years ago

dkirkby commented 7 years ago

Running the following command twice gives similar but not identical results, which I wasn't expecting:

quickspecsim -c desi --airmass 1 --exposure-time 48hour --ab-mag 25.3 --filter sdss2010-i

For example:

Median SNR in b camera = 0.898 / 0.5 Angstrom
Median SNR in r camera = 0.712 / 0.5 Angstrom
Median SNR in z camera = 0.509 / 0.5 Angstrom

and

Median SNR in b camera = 0.910 / 0.5 Angstrom
Median SNR in r camera = 0.712 / 0.5 Angstrom
Median SNR in z camera = 0.492 / 0.5 Angstrom

@julienguy Is this perhaps a "feature" of the new fastsim method calculating fiberloss? Do you use any random numbers?

julienguy commented 7 years ago

Yes. lines 710 and 711 of instrument.py, random offsets. You would need to set a random seed to get reproducible results.

alxogm commented 6 years ago

Hi, @dkirkby @julienguy we face to this issue, or a related one, by looking at the reproducibility of quasar spectra with DLAs features. See desihub/desisim#386. Attempting to solve such issue I made a branch of specsim: test_repr_qquasar which modifies only the generate_random_noise function in simulator.py. What tests do you suggest to make to see if this affects other applications? or to corroborate such modifications makes sense? Thanks

dkirkby commented 6 years ago

The generate_random_noise function should give reproducible results if you pass in a np.random.RandomState() object.

This issue refers to random centroid offsets that are currently not reproducible. You can test if this is the source of the problem you are seeing by setting instrument.offset.sigma1d to zero in your specsim config.