OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
94 stars 73 forks source link

Check to see if this masking `xr.where` parallelize #1165

Closed lsetiawan closed 6 months ago

lsetiawan commented 11 months ago

This xr.where needs to be investigated to see if it can parallelize with the use of dask.

https://github.com/OSOceanAcoustics/echopype/blob/3ca028f04a4fd3643bb0ad350dc09a699883ee1e/echopype/mask/api.py#L500

anantmittal commented 11 months ago

Command to run the tests and capture the stdout: python .ci_helpers/run-test.py --local --pytest-args="-vvrP" echopype/mask/api.py

pytest -vvrP echopype/tests/mask/test_mask.py::test_frequency_differencing

leewujung commented 10 months ago

@lsetiawan : the code you pointed to above was actually not the one we were discussing 😬

Below are the 2 potential places where the calibration code (compute_Sv) code can have bottleneck:

leewujung commented 10 months ago

@anantmittal : Thanks for investigating the xr.where in the masking code though -- since the change is to use dask.where, the solution would probably be similarly useful for the get_vend_cal_params_power case above.

leewujung commented 6 months ago

I'll close this now, since my comment above is replicated in #1200.