search

OSOceanAcoustics / echodataflow

Orchestrated sonar data processing workflow
https://echodataflow.readthedocs.io/en/latest/
MIT License
4 stars 1 forks source link

Standardization and Robustness Enhancements for Processing Workflows #101

Closed Sohambutala closed 2 months ago

Sohambutala commented 3 months ago

In this PR, significant changes have been made to the models and subflows to establish a common ground for future in-memory processing workflows and to ensure a stable flow for ship deployment. The key updates are:

  1. Standardized Models: Implemented standardized input and output models for each stage of the workflow to ensure consistency and ease of maintenance.
  2. Robust Flows: Enhanced the robustness of the flows such that the failure of a single file in a group does not disrupt the entire pipeline. The remaining groups continue processing seamlessly.
  3. Error Handling: Introduced a mechanism to write errored group JSON files at each stage to the output location.
  4. Updated CLI: Updated the CLI code to reflect the new boilerplate, ensuring compatibility with the revised models and workflows.
  5. Code Quality: Applied linting to all files.
codecov-commenter commented 3 months ago

Codecov Report

Attention: Patch coverage is 63.45840% with 224 lines in your changes missing coverage. Please review.

Project coverage is 58.77%. Comparing base (2d879ba) to head (dfcde6a).

Files Patch % Lines
echodataflow/stages/subflows/add_location.py 0.00% 37 Missing :warning:
echodataflow/stages/subflows/add_depth.py 0.00% 36 Missing :warning:
echodataflow/utils/file_utils.py 50.00% 32 Missing :warning:
echodataflow/echodataflow_cli.py 0.00% 29 Missing :warning:
echodataflow/utils/config_utils.py 50.00% 19 Missing :warning:
...chodataflow/stages/subflows/initialization_flow.py 83.11% 13 Missing :warning:
echodataflow/utils/rest_utils.py 0.00% 10 Missing :warning:
echodataflow/utils/databse_utils.py 10.00% 9 Missing :warning:
echodataflow/stages/subflows/combine_echodata.py 85.71% 6 Missing :warning:
echodataflow/stages/subflows/open_raw.py 87.17% 5 Missing :warning:
... and 10 more
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #101 +/- ## ========================================== + Coverage 58.72% 58.77% +0.05% ========================================== Files 29 29 Lines 1909 1885 -24 ========================================== - Hits 1121 1108 -13 + Misses 788 777 -11 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.