search

cta-observatory / cta-lstchain

LST prototype testbench chain
https://cta-observatory.github.io/cta-lstchain/
BSD 3-Clause "New" or "Revised" License
22 stars 77 forks source link

Merging of DL1 takes up to ~1 h #1230

Closed morcuended closed 2 weeks ago

morcuended commented 4 months ago

With the current lstchain version, the merging of DL1b files run-wise can take quite long (~1 h). I do not know if it's something related to FEFS I/O saturation or something genuine of the merging script. With previous versions, the merging took a shorter time (about 10 min).

moralejo commented 4 months ago

Did you try (in parallel) with an earlier version? That should tell between the two possibilities.

maxnoe commented 4 months ago

Related to https://github.com/cta-observatory/cta-lstchain/pull/1174 ?

morcuended commented 4 months ago

Related to https://github.com/cta-observatory/cta-lstchain/pull/1174 ?

That was my impression

Did you try (in parallel) with an earlier version? That should tell between the two possibilities.

I will do it

morcuended commented 3 months ago

Tried to test the merging of DL1 v0.10 files using lstchain v0.10 and v0.9, but it's not possible to merge v0.10 data with v0.9 

Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/bin/lstchain_merge_hdf5_files", line 8, in <module>
    sys.exit(main())
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 88, in main
    auto_merge_h5files(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/lstchain/io/io.py", line 317, in auto_merge_h5files
    file_list = merging_check(file_list)
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/lstchain/io/io.py", line 395, in merging_check
    subarray_info0 = SubarrayDescription.from_hdf(first_file)
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/ctapipe/instrument/subarray.py", line 512, in from_hdf
    geometry = CameraGeometry.from_table(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/ctapipe/instrument/camera/geometry.py", line 589, in from_table
    return cls(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.14/lib/python3.8/site-packages/ctapipe/instrument/camera/geometry.py", line 133, in __init__
    raise TypeError(
TypeError: pix_type most be a PixelShape or the name of a PixelShape, got b'hexagon'
moralejo commented 3 months ago

Any idea of where is the time being spent? Can you make a simple test to find out? (e.g. disabling in the code some of the changes from #1174)

morcuended commented 2 weeks ago

Closing this after #1256. Now it is not as bad as before. We'll keep an eye on this anyway in the future.