nens / threedigrid

grid administration for 3Di models
Other
9 stars 0 forks source link

Customized result files: filtered read gives OSError inflate() failed #244

Closed leendertvanwolfswinkel closed 1 month ago

leendertvanwolfswinkel commented 1 month ago

Same data as #243

If I do

water_levels_23 = customized_results_admin.nodes.filter(id__eq=23).timeseries(start_time=0, end_time=last_timestamp).s1

I get this error:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[19], line 1
----> 1 water_levels_23 = customized_results_admin.nodes.filter(id__eq=23).timeseries(start_time=0, end_time=last_timestamp).s1

File C:\ProgramData\anaconda3\envs\threedi_dev\Lib\site-packages\threedigrid\orm\base\models.py:227, in Model.__getattribute__(self, attr_name)
    223         return attr.get_value(self._datasource, attr_name)
    225     # Use the get function to retrieve the computed/filtered
    226     # value for the ArrayField with name: 'name'
--> 227     return super().__getattribute__("get_filtered_field_value")(attr_name)
    229 # Default behaviour, return the attribute from superclass
    230 return attr

File C:\ProgramData\anaconda3\envs\threedi_dev\Lib\site-packages\threedigrid\orm\base\models.py:204, in Model.get_filtered_field_value(self, field_name, ts_filter, lookup_index, subset_index)
    200 def get_filtered_field_value(
    201     self, field_name, ts_filter=None, lookup_index=None, subset_index=None
    202 ):
    203     # Redirect via datasource
--> 204     return self._datasource.get_filtered_field_value(
    205         self, field_name, ts_filter, lookup_index, subset_index
    206     )

File C:\ProgramData\anaconda3\envs\threedi_dev\Lib\site-packages\threedigrid\admin\h5py_datasource.py:62, in H5pyGroup.get_filtered_field_value(self, model, field_name, ts_filter, lookup_index, subset_index)
     53     if field_name in model.Meta.composite_field_insert_values:
     54         kwargs.update(
     55             {
     56                 "insert_value": model.Meta.composite_field_insert_values[
   (...)
     59             }
     60         )
---> 62 value = model.get_field_value(field_name, **kwargs)
     64 # Transform the base_filter by prepending slice(None) to
     65 # match the dimensionality of the nparray_dict[key].shape
     66 #
   (...)
     76 # Return a numpy array with None as only element when
     77 # the value is None.
     78 if value is None or value.size == 0:

File C:\ProgramData\anaconda3\envs\threedi_dev\Lib\site-packages\threedigrid\orm\base\models.py:169, in Model.get_field_value(self, field_name, **kwargs)
    163         update_dict["insert_value"] = self.Meta.composite_field_insert_values[
    164             field_name
    165         ]
    167 kwargs.update(update_dict)
--> 169 return self._meta.get_field(field_name).get_value(
    170     self._datasource, field_name, **kwargs
    171 )

File C:\ProgramData\anaconda3\envs\threedi_dev\Lib\site-packages\threedigrid\orm\base\fields.py:174, in TimeSeriesCompositeArrayField.get_value(self, datasource, name, **kwargs)
    172 if isinstance(timeseries_filter_to_use, np.ndarray):
    173     if len(source.shape) > 1:
--> 174         values.append(source[timeseries_filter_to_use, :])
    175     elif source.size == timeseries_filter_to_use.size:
    176         values.append(source[timeseries_filter_to_use])

File h5py\_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py\_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File C:\ProgramData\anaconda3\envs\threedi_dev\Lib\site-packages\h5py\_hl\dataset.py:758, in Dataset.__getitem__(self, args, new_dtype)
    756 if self._fast_read_ok and (new_dtype is None):
    757     try:
--> 758         return self._fast_reader.read(args)
    759     except TypeError:
    760         pass  # Fall back to Python read pathway below

File h5py\_selector.pyx:376, in h5py._selector.Reader.read()

OSError: Can't synchronously read data (inflate() failed)
leendertvanwolfswinkel commented 1 month ago

I did the same simulation again after the staging release and with the new files I do not have this issue