search

VERITAS-Observatory / EventDisplay_v4

A reconstruction and analysis pipeline for VERITAS.
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

Effective area file sizes #148

Closed GernotMaier closed 2 years ago

GernotMaier commented 2 years ago

Effective area files are very big, dominated by histograms to be used by gammapy analysis.

GernotMaier commented 2 years ago

Suggest to have a anasum only version of the effective areas with the following tree entries. Please have a look and confirm that this is fine:

******************************************************************************
*Tree    :fEffArea  : effective area values                                  *
*Entries :   302940 : Total =       819185992 bytes  File  Size =  152026730 *
*        :          : Tree compression factor =   5.39                       *
******************************************************************************
*Br    0 :ze        : ze/D                                                   *
*Entries :   302940 : Total  Size=    2426708 bytes  File Size  =      16281 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 148.99     *
*............................................................................*
*Br    1 :az        : az/I                                                   *
*Entries :   302940 : Total  Size=    1214103 bytes  File Size  =      11990 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression= 101.20     *
*............................................................................*
*Br    2 :azMin     : azMin/D                                                *
*Entries :   302940 : Total  Size=    2426813 bytes  File Size  =      39492 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  61.43     *
*............................................................................*
*Br    3 :azMax     : azMax/D                                                *
*Entries :   302940 : Total  Size=    2426813 bytes  File Size  =      34532 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  70.25     *
*............................................................................*
*Br    4 :Xoff      : Xoff/D                                                 *
*Entries :   302940 : Total  Size=    2426778 bytes  File Size  =      21158 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 114.65     *
*............................................................................*
*Br    5 :Yoff      : Yoff/D                                                 *
*Entries :   302940 : Total  Size=    2426778 bytes  File Size  =      22712 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 106.81     *
*............................................................................*
*Br    6 :Woff      : Woff/D                                                 *
*Entries :   302940 : Total  Size=    2426778 bytes  File Size  =      22236 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 109.10     *
*............................................................................*
*Br    7 :noise     : noise/I                                                *
*Entries :   302940 : Total  Size=    1214181 bytes  File Size  =      12625 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression=  96.11     *
*............................................................................*
*Br    8 :pedvar    : pedvar/D                                               *
*Entries :   302940 : Total  Size=    2426848 bytes  File Size  =      27828 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  87.18     *
*............................................................................*
*Br    9 :index     : index/D                                                *
*Entries :   302940 : Total  Size=    2426813 bytes  File Size  =      58855 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  41.22     *
*............................................................................*
*Br   10 :nbins     : nbins/I                                                *
*Entries :   302940 : Total  Size=    1214181 bytes  File Size  =      34504 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression=  35.17     *
*............................................................................*
*Br   11 :e0        : e0[nbins]/D                                            *
*Entries :   302940 : Total  Size=   85784285 bytes  File Size  =    2153436 *
*Baskets :      749 : Basket Size=    6243840 bytes  Compression=  39.83     *
*............................................................................*
*Br   12 :eff       : eff[nbins]/D                                           *
*Entries :   302940 : Total  Size=   85785038 bytes  File Size  =   72214376 *
*Baskets :      749 : Basket Size=    6244352 bytes  Compression=   1.19     *
*............................................................................*
*Br   13 :Rec_nbins : Rec_nbins/I                                            *
*Entries :   302940 : Total  Size=    1214285 bytes  File Size  =      52558 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression=  23.09     *
*............................................................................*
*Br   14 :Rec_e0    : Rec_e0[Rec_nbins]/D                                    *
*Entries :   302940 : Total  Size=   71873603 bytes  File Size  =    1770038 *
*Baskets :      655 : Basket Size=    5445632 bytes  Compression=  40.60     *
*............................................................................*
*Br   15 :Rec_eff   : Rec_eff[Rec_nbins]/D                                   *
*Entries :   302940 : Total  Size=   71874262 bytes  File Size  =   60079301 *
*Baskets :      655 : Basket Size=    5445632 bytes  Compression=   1.20     *
*............................................................................*
*Br   16 :esys_rel  : esys_rel[nbins]/F                                      *
*Entries :   302940 : Total  Size=   43502142 bytes  File Size  =    2660131 *
*Baskets :      390 : Basket Size=    3163648 bytes  Compression=  16.35     *
*............................................................................*
*Br   17 :hEsysMCRelative : TProfile                                         *
*Entries :   302940 : Total  Size=  436095470 bytes  File Size  =   12753304 *
*Baskets :     1738 : Basket Size=   25600000 bytes  Compression=  34.19     *
*............................................................................*

Removed are:

*Br   13 :effNoTh2  : effNoTh2[nbins]/D                                      *
*Entries :   302940 : Total  Size=   85746962 bytes  File Size  =   71742586 *
*Baskets :      358 : Basket Size=     312320 bytes  Compression=   1.20     *
*............................................................................*
*Br   17 :Rec_effNoTh2 : Rec_effNoTh2[Rec_nbins]/D                           *
*Entries :   302940 : Total  Size=   71842588 bytes  File Size  =   59881588 *
*Baskets :      340 : Basket Size=     271360 bytes  Compression=   1.20     *
*Br   19 :Rec_angRes_p68 : Rec_angRes_p68[Rec_nbins]/F                       *
*Entries :   302940 : Total  Size=   36545705 bytes  File Size  =    2842198 *
*Baskets :      329 : Basket Size=     137728 bytes  Compression=  12.86     *
*............................................................................*
*Br   20 :Rec_angRes_p80 : Rec_angRes_p80[Rec_nbins]/F                       *
*Entries :   302940 : Total  Size=   36545705 bytes  File Size  =    2854665 *
*Baskets :      329 : Basket Size=     137728 bytes  Compression=  12.80     *
*............................................................................*
*Br   21 :hEsysMCRelative : TProfile                                         *
*Entries :   302940 : Total  Size=  435931990 bytes  File Size  =   12155646 *
*Baskets :      304 : Basket Size=    2840576 bytes  Compression=  35.86     *
*............................................................................*
*Br   22 :hEsysMCRelative2D : TH2F                                           *
*Entries :302940 : Total  Size= 9153905056 bytes  File Size  = 101062485 *
*Baskets :     2615 : Basket Size=   25600000 bytes  Compression=  90.58     *
*............................................................................*
*Br   23 :hEsysMCRelative2DNoDirectionCut : TH2F                             *
*Entries :302940 : Total  Size= 9163366508 bytes  File Size  = 613688375 *
*Baskets :     2615 : Basket Size=   25600000 bytes  Compression=  14.93     *
*............................................................................*
*Br   24 :hAngularLogDiffEmc_2D : TH2F                                       *
*Entries :302940 : Total  Size= 5083231848 bytes  File Size  = 105916139 *
*Baskets :     1677 : Basket Size=   25600000 bytes  Compression=  47.99     *
*............................................................................*
steob92 commented 2 years ago

The ED likelihood spectral analysis also won't work with the slimmed down effective area files. Like the gammapy analysis they use hEsysMCRelative2D.

GernotMaier commented 2 years ago

Yes, I had this in mind but forgot to mention. Using the likelihood spectral analysis would require to download the full IRFs.

GernotMaier commented 2 years ago

Removing the branches above reduces the effective area file size by a factor (!) 6-8.

GernotMaier commented 2 years ago

fixed and release with v486b