search

icecube / skyreader

An API for Results Produced by SkyDriver & the Skymap Scanner
MIT License
0 stars 0 forks source link

SkyScanResult tolerances #8

Open mlincett opened 1 year ago

mlincett commented 1 year ago

From tests comparing a MillipedeWilks run on condor/grid vs CI it seems that reconstructed energy losses can differ up to 7% (so far) between the two.

LLH relative differences are still < 1E-4 so they pass the test.

I think we should allow reco losses differences up to 10%. @tianluyuan you think this is reasonable?

mlincett commented 1 year ago

I have to correct myself, LLH differences are < 2E-4 but > 1E-4 so the current limit is a bit too strict. This assuming we create the test data with Condor to compare against the CI test.

If we produce the test data from the CI worker the tolerance could be reduced, but is it worth the additional work to extract the data? (To be discussed when Ric is back).

tianluyuan commented 1 year ago

That's interesting.. I guess this is something that doesn't appear in MillipedeOriginal because those were generated on the CI worker. Though, if everything is run inside Singularity then shouldn't the results be reproducible to better than 10%? If there are differences in the llh between workers then we would also expect to see some of those fluctuations in the skymap.

In any case I think those tolerances are fine for now, maybe we can leave this open for reference.

mlincett commented 1 year ago

I don't see such a big difference for the GOLD event, I will try to re-run the BRONZE to be sure I didn't mess up something.

mlincett commented 1 year ago

From the "realistic" tests I see reco energies with differences as large as 16% and 1E-3 for LLH values.

For the moment I am adjusting the tolerances in the comparison script that runs in CI so to make all test pass.

Once that is achieved, I could regenerate all the test data with the latest docker tag of the branch, so we have a baseline.

Note that there seem to be no environment variable that changes the setup between CI and condor for millipede_wilks (only for millipede_original we reduce the number of position variations in CI).

ric-evans commented 1 year ago

not sure if we want to hard code these values, for the different configurations, or leave it open to the user. Also, see #9