Closed leewujung closed 2 years ago
@leewujung thank you for your description above.
In addition to the removal of the check from combine_echodata
, we should also do the following:
ek60_reversed_ping_time_test_data
and test_ping_time_reversal
functions from test_echodata_combine
test_combine_echodata
and special exceptions for itparam_id = "ek80_nb_diff_range_sample"
in test_combine_echodata
PR #835 addressed this issue by removing the time correction in combine_echodata
and moving useful functions to qc/api.py
, closing.
@b-reyes had a great insight that the reason why we currently (up until v0.6.2) have detection and correction of reversed or duplicated timestamps (
ping_time
,time1
, etc) incombine_echodata
was because we have been using xarray's merge function to combine multipleEchoData
objects.With @b-reyes new implementation to efficiently distribute data combination directly leveraging dask/zarr functionalities, we don't actually have to have this check and can keep the original data in the combined
EchoData
object/associated zarr file. This is actually better because we want theEchoData
object to be as lossless as possible from the original instrument file.So, we can:
combine_echodata
functionSee #798 for bug fixes/improvements needed for the timestamp cleaning function.