Validating `point-like` DL3 files

[TarekHC]

Together with @mireianievas , we are trying to gather the tests we need to perform to validate the point-like DL3 files generated with ED v4 + V2DL3. Note 1D analysis is only possible with gammapy, so all comparisons will come from ED v4 vs gammapy 0.16 (latest stable release):

• [ ] Compare skymaps (e.g. significance distributions).

• [ ] Compare significance vs time.

• [ ] Compare high-statistics spectrum from the Crab (and perhaps a couple of bright AGNs). Note gammapy performs a built-in unfolding while ed doesn't by default (maybe compare with @steob92 likelihood methods?).

• [ ] Check status of lightcurve generation within gammapy (we are not sure if it's properly implemented within gammapy).

As suggested by gernot:

• [ ] a steep blazar (e.g. PKS 1424)
• [ ] an empty field (e.g. Segue I or other dSphs)
• [ ] a week source (e.g. Tycho)

Anyone, any other test?

[mireianievas]

Just a teaser. I will create a more detailed comparison, one against the other, in the next days.

Crab with V6 - moderate cuts - (/lustre/fs23/group/cta/users/mnievas/veritas/Projects/DL3_test_20200220 )

ED

VSpectralFitter::fit Applying Fit
curved power law fit
 FCN=24.1291 FROM MINOS     STATUS=SUCCESSFUL     66 CALLS         547 TOTAL
                     EDM=7.76254e-07    STRATEGY= 1      ERROR MATRIX ACCURATE 
  EXT PARAMETER                                   STEP         FIRST   
  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  p0           2.90711e-11   5.79827e-13   2.43502e-15   3.55993e+07
   2  p1          -2.51528e+00   2.25540e-02   1.24203e-04  -1.13761e-04
   3  p2          -3.42431e-02   1.04435e-02   1.04435e-02   1.06509e-02
VSpectralFitter::fit Getting Confidence Interval (cl = 0.68)

Gammapy

name    value   error   unit    min max frozen
str9    float64     float64     str14   float64 float64 bool
amp 2.878e-11   6.100e-13   cm-2 s-1 TeV-1  nan nan False
ref 1.000e+00   0.000e+00   TeV nan nan True
alpha   2.662e+00   2.267e-02       nan nan False
beta    1.084e-01   1.743e-02       nan nan False

While there are some differences in alpha and beta (there's probably some degeneracy between them and there might be differences due to the differences in terms of unfolding) the gammapy results does not look totally absurd.

[GernotMaier]

Very nice!

Can you uploaded also the run list used?

Are all changes to the code submitted to github? Also the parameter files with the corrections for each season: where can I find this one?

[TarekHC]

Just for reference, the fit used within gammapy is in true energy, so it already contains an unfolding. To keep in mind when doing a one-to-one comparison.

[GernotMaier]

But that is the same for all analysis; using A_eff(E_rec) for the reconstruction is similar to unfolding in the approximation of a migration matrix with not to many off-diagonal entries (and yes, I know that this approximation is not correct at all energies)

[mireianievas]

Very nice! Can you uploaded also the run list used? Are all changes to the code submitted to github? The changes are not yet made, because I still need to think how to best make the pull request so that it is easy for you to implement the changes in the v500 code and before that I want to test a bit that there are no more important bugs.

The anasum runlist is in the directory I mentioned before, but here is the simple list of runs:

runlist_test_dl3.txt

Also the parameter files with the corrections for each season: where can I find this one?

I was going to upload it here: https://github.com/VERITAS-Observatory/Eventdisplay_AnalysisFiles but then realized that is for v5. Either I upload it to the svn or maybe you can create a repository for the Eventdisplay files for v480 in github and then I make the changes there (I prefer this latter option to be honest).

You can anyway use my cluster directory: /lustre/fs23/group/cta/users/mnievas/veritas/EVNDISP_AnalysisFiles_Mireia/v480e/VTS

It contains the period definition in the MSCW runparameter and the changes also applied to the EPOCH runparameter (needed if one wants the RUNLIST.whichRunsAreArrayEpoch.sh script to discriminate the different V6 subperiods).

[GernotMaier]

Differences for the runparameters files between v480 and v500 are not very big, so I don't think it merits a new repository (please convince me otherwise).

Different versions are handled with branches and tags.

I've created a new branch for you to submit the new files: https://github.com/VERITAS-Observatory/Eventdisplay_AnalysisFiles/tree/v481

This is a branch starting from the v480 tag.

[mireianievas]

In some cases they are, like if I remember correctly in the EVNDISP runparameter (maximum and minimum instead of just a single value). I would still prefer to create a different repository.

[mireianievas]

A different branch is an alternative. But I don't feel it is totally safe to submit changes that may broke v500 compatibility.

[GernotMaier]

But it doesn't affect the master or any other branch if you submit this to the v481 branch (as long as we don't merge it)

[mireianievas]

Ok, if there is a branch (I didn't know) then it is fine.

[GernotMaier]

I've created one earlier today (all earlier versions are in tags, not branches)

[mireianievas]

Done here: https://github.com/VERITAS-Observatory/Eventdisplay_AnalysisFiles/pull/3

[mireianievas]

First direct comparison between ED (no cuts to generate ULs though) and ED+V2DL3+Gammapy. V6 data from 2013, CARE_June1702, Moderate cuts. Assuming a Powerlaw for the IRFs to be reduce analysis differences with ED.

[GernotMaier]

Looks really nice.

Difficult to see how power laws agree - do you have a ratio plot?

What exactly does the color code mean? 'fit statistics difference'

[GernotMaier]

Close this - needs to be done on the V2DL3 / gammapy discussion level.