search

cta-observatory / ctapipe

Low-level data processing pipeline software for CTAO or similar arrays of Imaging Atmospheric Cherenkov Telescopes
https://ctapipe.readthedocs.org
BSD 3-Clause "New" or "Revised" License
64 stars 269 forks source link

Support reading sim_telarray DST0 files #1777

Open ParsonsRD opened 3 years ago

ParsonsRD commented 3 years ago

Hi all, The reading of the DST0 files (essentially dl1 data level) produced by Konrad's sim_telarray is currently broken. When reading these files the pedestal subtraction is currently not performed correctly. This seems to be due to the following code in simteleventsource.py:

pedestal = cam_mon["pedestal"] / cam_mon["n_ped_slices"] In this case cam_mon["n_ped_slices"] will have a value of whatever the number of slices was in the original simulation (typically something large like 40), but in the DST0 the pedestal stored is that of the integrated signal, i.e. a number of slices = 1. So will result in a subtraction of a far too low value. As far as I know there is no flag in the data to confirm that you are reading a DST0 file so a fix is not obvious. One option could be to use the n_samples length of the waveform from

n_gains, n_pixels, n_samples = adc_samples.shape

As this should always be the same as the pedestal samples for a sim_telarray output file, but one for a DST0 file. But there may be safer alternatives out there.

kosack commented 3 years ago

Do you have an example DST0 file (ideally small) we could add to the test suite?

Right now SimTelEventSource only supports raw simtel files, but it should be easy to implement handling the DST format ones, either in SimTelEventSource itself, or a derived version. At the very least it should fail with an error if a DST0 file is used, so our current is_compatible() function is not sufficient.

I guess DST0 files just have the traces already integrated? Or are there other differences?

ParsonsRD commented 3 years ago

Sure attached is my 10 event example file that I use for the template generation test functions. I found this issue when updating the template generation to v0.11.0.

gamma_HESS_example.simhess.gz

maxnoe commented 1 year ago

Sorry for only coming back to this now. The file you provided only contains ADCSums, not ADCSamples:

In the case where no adc samples are found in the SimTelEventSource, adc_samples is filled from adc_sums, upshaed to a 3d array.

From todays point of view, this is certainly wrong and should not be done, instead we should directly fill the DL1 images using the ADCSums, right?

https://github.com/cta-observatory/ctapipe/blob/6217b8517e02d5f811f0a1f65ca25cf1e3b079e6/ctapipe/io/simteleventsource.py#L762-L765