search

OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
96 stars 73 forks source link

CI Failures for `test_convert_ek80.py::test_convert` and `test_convert_source_target_locs.py::test_convert_time_encodings` #1348

Closed ctuguinay closed 3 months ago

ctuguinay commented 3 months ago

Caused via the merging of #1337. Still investigating

ctuguinay commented 3 months ago

For some reason np.float16 causes the errors but np.float32 doesn't.

Works:

EXPECTED_VAR_DTYPE = {
    "channel": np.str_,
    "cal_channel_id": np.str_,
    "beam": np.str_,
    "channel_mode": np.float32,
    "beam_stabilisation": np.byte,
    "non_quantitative_processing": np.int16,
}  # channel name  # beam name

Doesn't Work:

EXPECTED_VAR_DTYPE = {
    "channel": np.str_,
    "cal_channel_id": np.str_,
    "beam": np.str_,
    "channel_mode": np.float16,
    "beam_stabilisation": np.byte,
    "non_quantitative_processing": np.int16,
}  # channel name  # beam name
ctuguinay commented 3 months ago

Error:

_____________________________ test_convert[D20211004-T233354.raw] _____________________________

ek80_new_file = PosixPath('/home/exouser/echopype/echopype/test_data/ek80_new/D20211004-T233354.raw')
dump_output_dir = PosixPath('/home/exouser/echopype/echopype/test_data/dump')

    @pytest.mark.test
    def test_convert(ek80_new_file, dump_output_dir):
        print("converting", ek80_new_file)
        echodata = open_raw(raw_file=str(ek80_new_file), sonar_model="EK80")
>       echodata.to_netcdf(save_path=dump_output_dir, overwrite=True)

echopype/tests/convert/test_convert_ek80.py:86: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
echopype/echodata/echodata.py:556: in to_netcdf
    return to_file(
echopype/convert/api.py:88: in to_file
    _save_groups_to_file(
echopype/convert/api.py:186: in _save_groups_to_file
    io.save_file(
echopype/utils/io.py:72: in save_file
    ds.to_netcdf(path=path, mode=mode, group=group, encoding=encoding, **kwargs)
../miniforge3/envs/echopype/lib/python3.9/site-packages/xarray/core/dataset.py:2326: in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
../miniforge3/envs/echopype/lib/python3.9/site-packages/xarray/backends/api.py:1337: in to_netcdf
    dump_to_store(
../miniforge3/envs/echopype/lib/python3.9/site-packages/xarray/backends/api.py:1384: in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
../miniforge3/envs/echopype/lib/python3.9/site-packages/xarray/backends/common.py:397: in store
    self.set_variables(
../miniforge3/envs/echopype/lib/python3.9/site-packages/xarray/backends/common.py:435: in set_variables
    target, source = self.prepare_variable(
../miniforge3/envs/echopype/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:539: in prepare_variable
    nc4_var = self.ds.createVariable(**default_args)
src/netCDF4/_netCDF4.pyx:2967: in netCDF4._netCDF4.Dataset.createVariable
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: Illegal primitive data type, must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got float16 (variable 'channel_mode', group 'Beam_group1')

src/netCDF4/_netCDF4.pyx:4149: TypeError
ctuguinay commented 3 months ago

The errors are caused because netcdf doesn't support float16 dtype:

https://forum.hdfgroup.org/t/hdf5-rfc-adding-support-for-16-bit-floating-point-and-complex-number-datatypes-to-hdf5/11975 https://docs.unidata.ucar.edu/nug/current/md_types.html https://github.com/pydata/xarray/issues/1877