Adding Extensions to Support Main Pipeline

[Sohambutala]

These extensions are designed to trigger or support the main pipeline, aiding in data source acquisition, enabling concurrent deployments etc.

1. File Monitoring for Automated Deployments:

Purpose: To monitor file changes and trigger specific deployments.

Current Approach:

• Schedule: watchdog is not used currently as we only need to schedule at regular intervals. This can be changed later, or a separate extension can be added for real-time scenarios.

• Deployment Independence: Each file triggers a new deployment, and a failure in one deployment does not affect others.

  Implementation Details:

• Tracking: Utilizes Prefect Blocks to record the last successful run time and the list of processed files.

• Concurrency: Leverages task concurrency to run multiple deployments simultaneously, allowing for concurrent file processing.

• Fail-Safe Option: The fail_safe option toggles between concurrent and sequential processing. In sequential mode, a failure in one file stops the processing of remaining files. In concurrent mode, file processing is independent.

  Considerations: Concurrency must be managed carefully when appending results to a single Zarr store to avoid race conditions and maintain the order of data.

2. File Downloader for Seamless Data Transfer:

Purpose: To facilitate data transfer between two locations.

Implementation Details:

• Tracking: Checks the destination folder for existing files and skips the transfer if the file is already present.
• Concurrency: Employs task concurrency to download multiple files simultaneously. Concurrency can be managed by configuring the global concurrency limit (edf-data-transfer).

• Delete Option: Offers an option to delete files after transfer, with a default setting of false.

  Considerations: Be mindful of rate limitations of both the source and destination. Concurrent requests can trigger rate limits.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 7.19719% with 1586 lines in your changes missing coverage. Please review.

Project coverage is 31.68%. Comparing base (f85b12e) to head (83bb7d1). Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...chodataflow/stages/subflows/initialization_flow.py	9.28%	215 Missing :warning:
echodataflow/extensions/file_monitor.py	0.00%	193 Missing :warning:
echodataflow/stages/subflows/echoshader_flow.py	0.00%	144 Missing :warning:
...low/stages/subflows/echoshader_flow_predictions.py	0.00%	141 Missing :warning:
echodataflow/stages/subflows/mask_prediction.py	0.00%	120 Missing :warning:
echodataflow/extensions/file_downloader.py	0.00%	99 Missing :warning:
echodataflow/extensions/panel_app_lasker.py	0.00%	98 Missing :warning:
echodataflow/extensions/panel_app_shimada.py	0.00%	97 Missing :warning:
echodataflow/extensions/panel_app.py	0.00%	80 Missing :warning:
echodataflow/stages/subflows/Sv_pipeline.py	0.00%	80 Missing :warning:
... and 20 more

:exclamation: There is a different number of reports uploaded between BASE (f85b12e) and HEAD (83bb7d1). Click for more details.

HEAD has 1 upload less than BASE

| Flag | BASE (f85b12e) | HEAD (83bb7d1) | |------|------|------| ||2|1|

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #99 +/- ## =========================================== - Coverage 50.50% 31.68% -18.82% =========================================== Files 34 48 +14 Lines 2285 3888 +1603 =========================================== + Hits 1154 1232 +78 - Misses 1131 2656 +1525 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[Sohambutala]

Converting this PR to draft since Prefect variables have a limit of 255 characters. An alternative would be to try using the Echodataflow block object.

[valentina-s]

This PR has some files that seems from an older version. I point to one example in the comment above, but I also notice some of the actions versions got reversed to the older ones. So maybe you check this, before merging.

[Sohambutala]

Thank you for highlighting workflow changes, I had those locally but never committed, since I used a different branch to merge into main after the hack with Caesar and Brandyn.