Closed morcuended closed 1 month ago
There were no calibration laser events at the beginning of run 15982, then shifters recovered the laser during the run. I think that is the reason.
Thanks @SeiyaNozaki, that's it.
What should we do with the monitoring tables in these cases to be able to merge these files?
Why is this a problem at all? Why do we require that subruns have the same number of flatfield measurements?
Why is this a problem at all? Why do we require that subruns have the same number of flatfield measurements?
It was decided like this in #1174. For keys to be copied, among which "/dl1/event/telescope/monitoring/flatfield"
is, auto_merge_h5files
checks that they have all the same length as the one in the first file.
Found another run with the same merging problem (16129)
File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 117, in main
auto_merge_h5files(
File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/io/io.py", line 362, in auto_merge_h5files
raise ValueError("Length of key {} from file {} different than in file {}".format(k, filename, file_list[0]))
ValueError: Length of key /dl1/event/telescope/monitoring/calibration from file /fefs/aswg/data/real/DL1/20231215/v0.10/tailcut84/dl1_LST-1.Run16129.0117.h5 different than in file /fefs/aswg/data/real/DL1/20231215/v0.10/tailcut84/dl1_LST-1.Run16129.0000.h5
Another failing run 16224
File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 117, in main
auto_merge_h5files(
File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/io/io.py", line 362, in auto_merge_h5files
raise ValueError("Length of key {} from file {} different than in file {}".format(k, filename, file_list[0]))
ValueError: Length of key /dl1/event/telescope/monitoring/pedestal from file /fefs/aswg/data/real/DL1/20231217/v0.10/tailcut84/dl1_LST-1.Run16224.0124.h5 different than in file /fefs/aswg/data/real/DL1/20231217/v0.10/tailcut84/dl1_LST-1.Run16224.0000.h5
What should we do with the monitoring tables in these cases to be able to merge these files?
I do not understand the problem. If flatfield are missing, what happens if we just remove the check that currently fails?
I do not understand the problem. If flatfield are missing, what happens if we just remove the check that currently fails?
We could do that. @FrancaCassol, if skip the check that all flatfield tables have the same length, could that be a problem for any other part of the analysis?
Hi, sorry for the delayed reply. This problem comes from the Cat-A calibration table which contains also the output of the interleaved processed by lstchain_data_r0_to_dl1 , this processing should be removed soon (since the interleaved must be processed by the Cat-B calibration script). For what concerns the cat-B calibration table, it is read from the Cat-B calibration file at the beginning of lstchain_dl1ab , so per definition always the same per each subrun. Therefore, I think we can safely remove this check for the moment
While merging DL1 files, OSA failed for one of them with the following error:
The merged DL1 file was halfway produced, though.
For approximately half of the files of run
dl1_LST-1.Run15982.*.h5
the length of the flatfield table is 1, while for the other half is 2. For other runs is always 2.Is this expected @FrancaCassol?