search

OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
95 stars 73 forks source link

Latest xarray version caused echopype tests/functions to fail #776

Closed lsetiawan closed 1 year ago

lsetiawan commented 2 years ago

Overview

As of August 4th from commit https://github.com/OSOceanAcoustics/echopype/commit/72909babcfbc567943072c2d7293fcf7665810c4. CI test failure is observed. These tests failures are observed when using xarray version 2022.6.0.

Some failures are:

  1. ep.calibrate.compute_Sv function: https://github.com/OSOceanAcoustics/echopype/runs/7700692440?check_suite_focus=true#step:13:1149 ERROR: ValueError: dimensions () must have the same length as the number of data dimensions, ndim=1

    Stacktrace ```python _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ echopype/calibrate/api.py:200: in compute_Sv return _compute_cal(cal_type="Sv", echodata=echodata, **kwargs) echopype/calibrate/api.py:88: in _compute_cal cal_ds = cal_obj.compute_Sv(waveform_mode=waveform_mode, encode_mode=encode_mode) echopype/calibrate/calibrate_ek.py:962: in compute_Sv return self._compute_cal( echopype/calibrate/calibrate_ek.py:928: in _compute_cal ds_cal = self._cal_complex(cal_type=cal_type, waveform_mode=waveform_mode) echopype/calibrate/calibrate_ek.py:751: in _cal_complex gain = self._get_gain_for_complex(waveform_mode=waveform_mode, chan_sel=chan_sel) echopype/calibrate/calibrate_ek.py:631: in _get_gain_for_complex if ch_id in self.echodata.vendor.cal_channel_id: /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/dataarray.py:810: in __contains__ return key in self.data /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/_typed_ops.py:248: in __eq__ return self._binary_op(other, nputils.array_eq) /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/dataarray.py:3530: in _binary_op f(self.variable, other_variable) /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/nputils.py:77: in array_eq return _ensure_bool_is_ndarray(self == other, self, other) /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/_typed_ops.py:438: in __eq__ return self._binary_op(other, nputils.array_eq) /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/variable.py:2521: in _binary_op result = Variable(dims, new_data, attrs=attrs) /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/variable.py:320: in __init__ self._dims = self._parse_dimensions(dims) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ```

  2. env_params._apply function: https://github.com/OSOceanAcoustics/echopype/runs/7700692440?check_suite_focus=true#step:13:1493 ERROR: ValueError: conflicting sizes for dimension 'time1': length 1932 on 'time1' and length 2933 on {'time1': 'pressure'}

    Stacktrace ```python _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ echopype/calibrate/env_params.py:125: in _apply env_params = xr.Dataset( /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/dataset.py:599: in __init__ variables, coord_names, dims, indexes, _ = merge_data_and_coords( /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/merge.py:575: in merge_data_and_coords return merge_core( /usr/share/miniconda3/envs/echopype/lib/python3.9/site-packages/xarray/core/merge.py:761: in merge_core dims = calculate_dimensions(variables) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ```

  3. test_nan_range_entries test failure: https://github.com/OSOceanAcoustics/echopype/runs/7700692440?check_suite_focus=true#step:13:1687 ERROR: AssertionError: assert False

    Failure detail ```python > assert xr.Dataset.equals(nan_locs_backscatter_r, nan_locs_Sv_range) E AssertionError: assert False E + where False = (\narray([[[ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n ...,\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True]],\n\n [[ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n ...,\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True]],\n\n [[ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n ...,\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True]]])\nCoordinates:\n * channel (channel) \narray([[[ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n ...,\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True]],\n\n [[ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n ...,\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True]],\n\n [[ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n [ True, True, True, ..., False, False, False],\n ...,\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True],\n [ True, True, True, ..., True, True, True]]])\nCoordinates:\n * channel (channel) object 'GPT 18 kHz 009072058c8d 1-1 ES18-11'...\n * ping_time (ping_time) datetime64[ns] 2017-01-15T15:01:22.90599987...\n * range_sample (range_sample) int64 0 1 2 3 4 ... 7911 7912 7913 7914\n pulse_length_bin (channel, ping_time) int64 1 1 1 1 1 1 1 ... 4 4 4 4 4 4 4) E + where = .equals E + where = xr.Dataset ```

Current solution

As of August 9th, 2022, a patch by pinning xarray to 2022.3.0 has been implemented.

To do

In the future, we'll need to fix these functions to work with the latest xarray version so we can unpin xarray.

lsetiawan commented 1 year ago

As seen in https://github.com/OSOceanAcoustics/echopype/pull/981 the issue of unpinning xarray still persists in some of the test.

lsetiawan commented 1 year ago

Done in #981