OSOceanAcoustics / echoshader

Interactive visualization of ocean sonar data
https://echoshader.readthedocs.io
MIT License
9 stars 8 forks source link

Juypter notebook crashing when trying to retrieving EK60 echosounder data #68

Closed praneethratna closed 1 year ago

praneethratna commented 1 year ago

General description of problem

Jupyter notebook crashing when trying to run the data retrieving part of the notebook.

Computing environment

Required

Minimum example

Required: A minimum code snippet that reproduces your problem. The following code reproduces the errors I encountered:

for desired_raw_file_url in desired_raw_file_urls:
    raw_fpath = Path(desired_raw_file_url)

    try_times = 0
    while True:
        try_times += 1

        if try_times > 2:
            print(f"Failed to process raw file {raw_fpath.name}: {e}.")
            break

        try:
            # Access file directly from S3 to create a converted EchoData object in memory
            ed = ep.open_raw(
                f"{desired_raw_file_url}",
                sonar_model='ek60'
            )

            # Use the EchoData object "ed" to generate calibrated and
            # computed MVBS files that will be saved to netcdf
            ds_Sv = ep.calibrate.compute_Sv(ed)

            ds_MVBS = ep.preprocess.compute_MVBS(
                ds_Sv,
                range_meter_bin=1,  # in meters
                ping_time_bin='1800s'  # in seconds
            )

            ds_MVBS.to_netcdf(calibrated_dpath / f"MVBS_{raw_fpath.stem}.nc")

            # release to avoid memory overflow
            del ed
            del ds_Sv
            del ds_MVBS

            gc.collect()

        except Exception as e:
            print(f"Failed to process raw file {raw_fpath.name}: {e}. Execute again")
            continue

        break

Error message printouts

Required: Copy-paste the entire error messages you encounter here. Below is the error messages I received when running the above code:

Canceled future for execute_request message before replies were done

The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click [here for more info. View Jupyter log for further details.

Provide an example file

Required: An example file that allow reproducing the problem.

Troubleshooting

Optional but preferred: Explain any steps that you have taken to identify the cause of conversion error.

leewujung commented 1 year ago

I made some small modifications in #71 -- the notebooks did not crash for me but there were likely some memory efficiency issues due to echopype (which is being worked on for v0.7.2). This issue/need for test/benchmarking data generation is tracked in #70, so will close this now.