search

cta-observatory / protopipe

Prototype data analysis pipeline for the Cherenkov Telescope Array Observatory
https://protopipe.readthedocs.io/en/latest/
Other
5 stars 13 forks source link

Comparison with CTA-MARS: Cuts optimization #94

Open HealthyPear opened 3 years ago

HealthyPear commented 3 years ago

Context

Some of these steps are different between CTA-MARS and EventDisplay. We should save as much information as possible in terms of what operations are missing in our API.

What CTA-MARS does

Paraphrasing from the IRF document:

In each energy bin, cuts in:

  • multiplicity (scanned from >= 2 to >= 9 telescopes of any type),
  • Hadronness (from 0.5 to 0.95 gamma efficiency)
  • event direction (from 0.5 to 0.95 gamma efficiency)

are optimized in order to achieve the best possible (pointlike- source) gamma-ray flux sensitivity (against the background of cosmic-ray electrons and protons), according to the standard definition within CTA ( 5 sigma, >= 10 events, >= 5% of residual background).

Current situation (to be updated as the discussion in this issue grows)

Questions for @moralejo :

moralejo commented 3 years ago

Questions for @moralejo :

Apologies, I had overlooked this - while I had independently opened https://github.com/cta-observatory/pyirf/issues/103 on the same topic - leading to https://github.com/cta-observatory/pyirf/pull/104

  • for each energy bin you choose the multiplicity value in the range [2,9] which maximizes sensitivity - correct?

Correct

  • in which order/combination are the 3 cuts applied?

Note that the efficiencies of the hadronness cut and the theta2 cuts arbitrarily forced to be the same. Together with the relatively coarse efficiency steps, this means that the optimization on the test MC (the same used for performance calculations) will in general not pick on lucky fluctuations. Of course cut optimization on an independent sample should be preferred - one could even divide the test sample in, say, odd/even event id's, and use one half to optimize the cuts on the other and vice-versa, so that the whole test sample is used. There is no order of the cuts, all combinations of multiplicity and (hadronness+theta2) efficiency (same for both) are considered. Hence it is just a 2D scan.

  • what type of cuts are (namely are all 3 only best-sensitivity cuts or there also e.g. a theta 68% cut applied in combination with the others like EventDisplay does)

Best sensitivity, with the constraint of same efficiency for theta2 & hadronness cuts.

HealthyPear commented 3 years ago

As soon as cta-observatory/pyirf#123 is merged and a new release is out we can use it here and reproduce this approach.