Merging hdf file breaks subarray description

[gabemery]

I am trying to use merged dl1 files from LST data and the following method fails :

subarray.camera_types[0].geometry.neighbor_matrix_sparse where subarray = SubarrayDescription.from_hdf(input_file)

The same applied to a subrun works correctly. It happens with both the available (v0.9) merged run (13396) and a merging I produced with 10 subruns and lstchain 0.9.6.

Here is the error message :

print(subarray_Lmerged.camera_types[0].geometry.neighbor_matrix_sparse)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[33], line 1
----> 1 print(subarray_Lmerged.camera_types[0].geometry.neighbor_matrix_sparse)

File /fefs/aswg/workspace/gabriel.emery/mambaforge-pypy3-2/envs/magic-lst1-031/lib/python3.8/site-packages/astropy/utils/decorators.py:767, in lazyproperty.__get__(self, obj, owner)
    765         val = obj_dict.get(self._key, _NotFound)
    766         if val is _NotFound:
--> 767             val = self.fget(obj)
    768             obj_dict[self._key] = val
    769 return val

File /fefs/aswg/workspace/gabriel.emery/mambaforge-pypy3-2/envs/magic-lst1-031/lib/python3.8/site-packages/ctapipe/instrument/camera/geometry.py:629, in CameraGeometry.neighbor_matrix_sparse(self)
    627     return self._neighbors
    628 else:
--> 629     return self.calc_pixel_neighbors(diagonal=False)

File /fefs/aswg/workspace/gabriel.emery/mambaforge-pypy3-2/envs/magic-lst1-031/lib/python3.8/site-packages/ctapipe/instrument/camera/geometry.py:680, in CameraGeometry.calc_pixel_neighbors(self, diagonal)
    677 neighbors = neighbor_candidates[pixels, neigbor_index]
    678 data = np.ones(len(pixels), dtype=bool)
--> 680 neighbor_matrix = csr_matrix((data, (pixels, neighbors)))
    682 # filter annoying deprecation warning from within scipy
    683 # scipy still uses np.matrix in scipy.sparse, but we do not
    684 # explicitly use any feature of np.matrix, so we can ignore this here
    685 with warnings.catch_warnings():

File /fefs/aswg/workspace/gabriel.emery/mambaforge-pypy3-2/envs/magic-lst1-031/lib/python3.8/site-packages/scipy/sparse/_compressed.py:53, in _cs_matrix.__init__(self, arg1, shape, dtype, copy)
     49 else:
     50     if len(arg1) == 2:
     51         # (data, ij) format
     52         other = self.__class__(
---> 53             self._coo_container(arg1, shape=shape, dtype=dtype)
     54         )
     55         self._set_self(other)
     56     elif len(arg1) == 3:
     57         # (data, indices, indptr) format

File /fefs/aswg/workspace/gabriel.emery/mambaforge-pypy3-2/envs/magic-lst1-031/lib/python3.8/site-packages/scipy/sparse/_coo.py:148, in coo_matrix.__init__(self, arg1, shape, dtype, copy)
    146 if shape is None:
    147     if len(row) == 0 or len(col) == 0:
--> 148         raise ValueError('cannot infer dimensions from zero '
    149                          'sized index arrays')
    150     M = operator.index(np.max(row)) + 1
    151     N = operator.index(np.max(col)) + 1

ValueError: cannot infer dimensions from zero sized index arrays

I also see in the hdf5 file the LST geometry (in configuration/instrument/subarray/telescope/camera/geometry_0) contains 18550 elements instead of 1855. So it looks like the pix_id, pix_x, pix_y and pix_area are stacked for each subrun.

[maxnoe]

The same applied to a subrun works correctly. It happens with both the available (v0.9) merged run (13396) and a merging I produced with 10 subruns and lstchain 0.9.6.

Just to be sure: did you merge 0.9 files using lstchain 0.9? or did you try merging files written with 0.9 using lstchain 0.10?

Looking at the code, it just iterates over all tables in the file and doesn't make any difference between things that should stay the same (e.g. instrument tables) and things that need to be concatenated.

I wonder why this ever worked....

[maxnoe]

Ok, short answer is: "this never worked"

The OSA merged files have this issue. It seems this only worked because the camera geometry wasn't used after dl1.

[gabemery]

Just to be sure: did you merge 0.9 files using lstchain 0.9? or did you try merging files written with 0.9 using lstchain 0.10?

I merged v0.9 (OSA) files using lstchain 0.9.6 and also got the same error on already merged files with v0.9 (OSA) .

It seems this only worked because the camera geometry wasn't used after dl1.

Probably, i only got this issue because I was using the merged file in magic-cta-pipe. When initiating HillasReconstructor(subarray) before adding stereo parameters.