OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
98 stars 73 forks source link

Remove timestamp correction from `combine_echodata` #833

Closed leewujung closed 2 years ago

leewujung commented 2 years ago

@b-reyes had a great insight that the reason why we currently (up until v0.6.2) have detection and correction of reversed or duplicated timestamps (ping_time, time1, etc) in combine_echodata was because we have been using xarray's merge function to combine multiple EchoData objects.

With @b-reyes new implementation to efficiently distribute data combination directly leveraging dask/zarr functionalities, we don't actually have to have this check and can keep the original data in the combined EchoData object/associated zarr file. This is actually better because we want the EchoData object to be as lossless as possible from the original instrument file.

So, we can:

See #798 for bug fixes/improvements needed for the timestamp cleaning function.

b-reyes commented 2 years ago

@leewujung thank you for your description above.

In addition to the removal of the check from combine_echodata, we should also do the following:

b-reyes commented 2 years ago

PR #835 addressed this issue by removing the time correction in combine_echodata and moving useful functions to qc/api.py, closing.