cta-observatory / pyirf

Python IRF builder
https://pyirf.readthedocs.io/en/stable/
MIT License
15 stars 25 forks source link

Remove nonsensical condition from relative_sensitivity #241

Closed maxnoe closed 1 year ago

maxnoe commented 1 year ago

The function required that the relative sensitsivity (to the assumed spectrum) must be <= 1, which actually has no special meaning at all, it just means that a source brighter than the reference spectrum is needed.

This prevented computing the sensitivity of the last two highest energy bins for North Alpha layout compared to event display.

rlopezcoto commented 1 year ago

and this was affecting the two highest energy bins (because the sensitivity was >100%) but no low energy ones? PS: Tests need to be modified as well

maxnoe commented 1 year ago

and this was affecting the two highest energy bins (because the sensitivity was >100%) but no low energy ones?

Yes, exactly.

maxnoe commented 1 year ago

Here is the official prod5 50h sensitivity with Crab Nebula flux levels, only the last (here three) bins are above 1 CU

prod5_sensitivity

rlopezcoto commented 1 year ago

ok, thanks! I thought that low energy sensitivity was also overcoming the Crab Nebula flux (maybe in the Southern Array with the alpha configuration?)

maxnoe commented 1 year ago

ok, thanks! I thought that low energy sensitivity was also overcoming the Crab Nebula flux (maybe in the Southern Array with the alpha configuration?)

Without LSTs probably, yes, I didn't check.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (4e0e0ac) 94.33% compared to head (7ae02c4) 94.34%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #241 +/- ## ======================================= Coverage 94.33% 94.34% ======================================= Files 58 58 Lines 2736 2739 +3 ======================================= + Hits 2581 2584 +3 Misses 155 155 ``` | [Impacted Files](https://app.codecov.io/gh/cta-observatory/pyirf/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory) | Coverage Δ | | |---|---|---| | [pyirf/sensitivity.py](https://app.codecov.io/gh/cta-observatory/pyirf/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory#diff-cHlpcmYvc2Vuc2l0aXZpdHkucHk=) | `91.89% <ø> (-0.22%)` | :arrow_down: | | [pyirf/tests/test\_sensitivity.py](https://app.codecov.io/gh/cta-observatory/pyirf/pull/241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory#diff-cHlpcmYvdGVzdHMvdGVzdF9zZW5zaXRpdml0eS5weQ==) | `100.00% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

mdpunch commented 1 year ago

Here is the official prod5 50h sensitivity with Crab Nebula flux levels, only the last (here three) bins are above 1 CU

prod5_sensitivity

Minor comment @maxnoe @rlopezcoto : this is (presumably) the 50h sensitivity to an $E^{-2}$ spectrum, whereas the Crab is more like $E^{-2.7}$ so a direct comparison between the Crab spectrum and the sensitivity just gives an idea of detectability in the bins. (I just realized this myself a few days ago, so made a pull request on the Gammapy SensitivityEstimator doc to make this clearer).
There are some CTA papers with the sensitivity for a Crab Spectrum, but from 2015, so probably a bit outdated, e.g. https://arxiv.org/pdf/1509.01943.pdf "Real-Time Analysis sensitivity evaluation of the Cherenkov Telescope Array" which has "The MC simulation assumes a source with a power law spectrum and a Crab-like index located at the center of the field of view and observed at a zenith angle of 20 degrees."

maxnoe commented 1 year ago

Minor comment @maxnoe @rlopezcoto : this is (presumably) the 50h sensitivity to an $E^{-2}$ spectrum, whereas the Crab is more like $E^{-2.7}$ so a direct comparison between the Crab spectrum and the sensitivity just gives an idea of detectability in the bins.

No, the simulated events are weighted to the Crab Nebula flux (per the CTAC document prepared by @GernotMaier and @moralejo to the Crab spectrum as published by HEGRA), which is:

$$ 2.83\cdot 10^{-11} \mathrm{TeV}^{-1} \mathrm{cm}^{-2} \mathrm{s}^{-1} \left(\frac{E}{1 \mathrm{TeV}}\right)^{-2.62} $$

However, this is affects only the distribution of energies inside a bin, so I expect the differences to be minor between different fluxes the events are weighted to.

mdpunch commented 1 year ago

Minor comment @maxnoe @rlopezcoto : this is (presumably) the 50h sensitivity to an E−2 spectrum, whereas the Crab is more like E−2.7 so a direct comparison between the Crab spectrum and the sensitivity just gives an idea of detectability in the bins.

No, the simulated events are weighted to the Crab Nebula flux (per the CTAC document prepared by @GernotMaier and @moralejo to the Crab spectrum as published by HEGRA), which is:

2.83⋅10−11TeV−1,cm−2,s−1(E1,TeV)(−2.62)

However, this is affects only the distribution of energies inside a bin, so I expect the differences to be minor between different fluxes the events are weighted to.

Oh, interesting, thanks!

For the CTA sensitivity given on the official website https://www.cta-observatory.org/science/ctao-performance/#1472563157332-1ef9e83d-426c there is no description of the spectrum for which the sensitivity is calculated, so that information is missing there. So I suppose I should make a "pull request" on the CTAO info page!

For the zenodo links https://zenodo.org/record/7298569 for the current prod5 simulations, and https://zenodo.org/record/5163273 to previous baseline prod3 simulation, this information is indeed there: "A power-law gamma-ray spectrum with photon index 2.62 was assumed in the calculations."

But as I mentioned Gammapy assumes an index of 2 by default. So many people could fall into the trap.

Finally, I and others had thought, like you, that the change in index for the sensitivity would only move events around within bins. So much so that we found it hard to believe the effect when changing the spectral model by default used in Gammapy's SensitivityEstimator! But I checked this a bit brutally with my pre-Gammapy sensitivity code (by pruning the gammas as a function of energy to change the index), and I confirm that the effect is striking, especially for softer sources with an index >3 (not to mention MAGIC's GRB with an index of 5.43).

Conclusion: Nothing, just for info, but I may push the webmaster of CTAO to include full information on the webpage.

GernotMaier commented 1 year ago

Main reason why no spectrum is mentioned is that we are showing differential sensitivities. 5 bins per log decade are probably enough that the effect of the assumed spectral index can be neglected (is well below other uncertainties).

mdpunch commented 1 year ago

Main reason why no spectrum is mentioned is that we are showing differential sensitivities. 5 bins per log decade are probably enough that the effect of the assumed spectral index can be neglected (is well below other uncertainties).

Hi Gernot @GernotMaier ... except that I have seen that it can't with the ALTO WCD array. I will see if I can do the same demonstration with the CTA IRFs (which maybe indeed isn't so affected, since the energy resolution is better than WCDs)... but I am sometimes surprised...

mdpunch commented 1 year ago

So, indeed this is not a violent for CTA: image vs. image

Still, it would be nice to get people into the habit of giving the info, since it will have a bigger effect for HAWC, LHASSO, SWGO, ALTO...

mdpunch commented 1 year ago

Superimposed: image ... so a 50% difference at the lowest energies (where the energy resolution is worse).