search

icecube / flarestack

Unbinned likelihood analysis code for astroparticle physics datasets
https://flarestack.readthedocs.io/en/latest/?badge=latest
MIT License
8 stars 7 forks source link

Unexpected behaviour for crosscheck with skylab #15

Closed JannisNe closed 4 years ago

JannisNe commented 4 years ago

Describe the bug When stacking more than about 10 sources, flarestack's results deviate from expectations as well as from the reference skylab results.

To Reproduce Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/skylab_crosscheck/calculate_skylab_sensitivity.py'
  2. Look at the produced plots

Expected behavior The expected relation of sensitivity flux and number of stacked sources lies in between F proportional so sqrt(N) and F=const.

sens_nsources_gamma=2.0.pdf sens_nsources_gamma=2.0_persource.pdf

JannisNe commented 4 years ago

I attached the script used to run skylab. (.txt since no .py files can be uploaded :( )

skylab_crosscheck.txt

robertdstein commented 4 years ago

So investigating this further, I get exactly the expected behaviour for one season of IceCube data. sens_nsources_gamma=2.0.pdf, and things behave better if IC40 is excluded but the rest of the seasons are included. I think this might be related to #5.

JannisNe commented 4 years ago

A little comment: I realised flarestack was using different sources as skylab. The argument 'since' in the function fs_sources() in line 82 and 94 was missing. I pushed the corrected script to GitHub and attached the updated plot. The essential behaviour hasn't changed. I think it gets clearer though that the problem might be related to #5 as @robertdstein pointed out.

sens_nsources_gamma=2.0.pdf

robertdstein commented 4 years ago

I am becoming more convinced that the problem was the long-suspected "PDFs from IC40 are built with low statistics" might be to blame.

If I look at IC40-only, fitting the spectral index, I get: sens_nsources_gamma=2.0.pdf

If I look at IC40-only, and fix the spectral index to 2.0, I get a pretty-perfect: sens_nsources_gamma=2.0.pdf

I will investigate further, but hopefully will solve this and #5 together.

JannisNe commented 4 years ago

Turned out it was a bug in the skylab script. No everything looks fine.

sens_nsources_gamma=2.0_persource.pdf