Upgrades to Deploy Automated Ship Pipeline: New Flows, Improved Logging, and Enhanced Cleanup

Major Upgrades to Echodataflow Pipeline

This PR introduces significant enhancements to the echodataflow pipeline, providing new capabilities and improving existing functionalities.

Highlights:

New Flows Added

write_output: Directs the current output to a specific location or store.
mask_prediction: Utilizes an ML model to predict masks, which are then applied using the apply_mask function.
compute_NASC: Calculates the Nautical Acoustic Scattering Coefficient (NASC) on the data.
resample: Resamples and stores the output at a lower or higher resolution.
slice_store: Slices the store based on time chunks, using Prefect variables to track the last sliced frame index. Currently supports only Zarr stores.

Issue Resolution:

Codec Buffer Limit Error (resolves #103): Fixed the codec buffer limit error when writing masked output to a Zarr file. This ensures that range_sample is included in Zarr files.
Switched from default Blosc encoding to Zlib which resolved Intermittent Blosc Decompression Error During Zarr Operations

Support for New Input:

Store Mode: Added support for new input via a store, enhancing flexibility. Store mode can be activated by specifying storepath in datastore.yaml instead of the traditional urlpath.
- window_size: Number of samples to slice from a Zarr store (e.g., 500).
- time_travel_hours: Number of hours to go back from the last timestamp in the Zarr store (e.g., 2 hours).
- time_travel_mins: Number of minutes to go back from the last timestamp in the Zarr store (e.g., 30 minutes).
- rolling_size: Number of samples to go back in a Zarr store to set a starting point for a new window (e.g., 100). For non-overlapping windows, set rolling_size equal to window_size.

External Parameters Support:

Added support for external parameters in all flows, enabling more customizable and dynamic workflow configurations. External parameters must match the underlying process function, and echodataflow handles them only if explicitly required for pipeline processing.

Revamped Logging:

Logging improvements include better tracking, debugging, and overall transparency. Dask logs now maintain order, log tracebacks before failures, and resolve log contamination across pipeline runs in a Dask cluster setup (resolves #102).

Improved Post-Execution Cleanup:

Enhanced cleanup logic to manage and clean up resources after execution, improving reliability and efficiency.

save_offline flag in pipeline.yaml prioritizes whether to keep intermediate files, while the retention flag acts as a default if save_offline is absent.
Note: This does not delete any by-products generated by the pipeline, such as a Zarr store.

Exception Handling:

Added support for internal and external exceptions, clearly distinguishing between soft and hard failures.

Soft Failures: Ensure that the flow status at the end of execution is marked as completed, which is useful for triggering Prefect automations despite internal errors.
Hard Failures: Halt processing only if there are no more files to process, enhancing error resilience.

Resolves #100

Files	Patch %	Lines
echodataflow/stages/subflows/mask_prediction.py	0.00%	77 Missing :warning:
echodataflow/stages/subflows/slice_store.py	0.00%	61 Missing :warning:
echodataflow/stages/subflows/resample.py	0.00%	60 Missing :warning:
echodataflow/stages/subflows/compute_NASC.py	0.00%	52 Missing :warning:
echodataflow/stages/subflows/write_output.py	0.00%	48 Missing :warning:
...chodataflow/stages/subflows/initialization_flow.py	48.00%	39 Missing :warning:
echodataflow/utils/file_utils.py	50.00%	16 Missing :warning:
echodataflow/stages/echodataflow.py	28.57%	15 Missing :warning:
echodataflow/stages/echodataflow_trigger.py	62.50%	15 Missing :warning:
echodataflow/stages/subflows/add_location.py	0.00%	9 Missing :warning:
... and 10 more

OSOceanAcoustics / echodataflow