search

cta-observatory / lstosa

On-site data processing pipeline for the LST-1
https://lstosa.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
3 stars 4 forks source link

DL1ab stage, NSB-adapted settings #274

Open moralejo opened 7 months ago

moralejo commented 7 months ago

Hi, I am thinking on how to implement the adaptation of the "base" cleaning levels to the NSB conditions. As of now, the "pedestal cleaning" increases the picture threshold on a subrun-wise and pixel-wise basis. This is ok for moderating the effect of stars, but for high NSB conditions it will not just change the cleaning for a few pixels around the stars, but for a large fraction of the camera. This makes the rest of the analysis very difficult. In contrast, having the same cleaning (in # of p.e. for picture and boundary threshold) for the bulk of the camera in real data analysis means that just by applying the same to the MC (with similar noise level) you will get a pretty good match between them.

So we need a way to decide on higher "base" cleaning levels as a function of the NSB. And we do not want that to be decided on a subrun-wise basis, again on the grounds that it would make further analysis complicated (as of now, a whole "run" is supposed to be processed with the same settings - it is the closest we have to GTIs)

Hence, for launching DL1ab we need first to determine the base cleaning levels for the whole run. Is the DL1ab stage launched by lstosa subrun by subrun, or with a single job per run? The latter would make the implementation of the above easier, I think.

marialainez commented 7 months ago

Hi, lstosa launches one job array per run, which launches one job for each subrun in that sequence. So the DL1ab stage currently is done subrun by subrun.

moralejo commented 7 months ago

Hi, lstosa launches one job array per run, which launches one job for each subrun in that sequence. So the DL1ab stage currently is done subrun by subrun.

Ok, then before the subrun-wise job array is launched, one could launch a small job over all subruns of the run (or just a subset of them, 1/10 of them) just to determine the NSB level, derive from there the common cleaning thresholds, and then launch the array of jobs. Would that work?

marialainez commented 7 months ago

Yes, we could implement that, but how are the new cleaning levels used in the next step (r0_to_dl1)? Is this already implemented in lstchain?

moralejo commented 7 months ago

Yes, we could implement that, but how are the new cleaning levels used in the next step (r0_to_dl1)? Is this already implemented in lstchain?

The levels are set in the .json configuration file. What is still missing (I will take care) is a program to determine what would be good "base" cleaning levels (the equivalent of our default 8, 4) for a given run, based on interleaved pedestals. The levels should be such that the (anti-stars) "pedestal cleaning" only sets in for a small fraction of the camera pixels (those around the brightest stars in the FoV).