flatironinstitute / dendro-old

Analyze neuroscience data in the cloud
https://flatironinstitute.github.io/dendro-docs/
Apache License 2.0
19 stars 2 forks source link

Support folders for job inputs and outputs #126

Closed magland closed 8 months ago

magland commented 8 months ago

To demonstrate, here are three example processors. The first is CreateSampleFolderProcessor that creates a sample folder as output

https://github.com/magland/dendro_analyses/blob/c2bf64d6921b1cc19bfb1afc85a4b8f4da0697f0/apps/dendro1/main.py#L102-L125

The second is TarProcessor that takes a folder as input and returns a .tar file as output

https://github.com/magland/dendro_analyses/blob/c2bf64d6921b1cc19bfb1afc85a4b8f4da0697f0/apps/dendro1/main.py#L128-L153

Finally, the third is UntarProcessor that takes a .tar file as input and outputs the extracted folder

https://github.com/magland/dendro_analyses/blob/c2bf64d6921b1cc19bfb1afc85a4b8f4da0697f0/apps/dendro1/main.py#L156-L181

I tested this using the following script that queued up a simple pipeline:

https://github.com/magland/dendro_analyses/blob/main/projects/test/test_input_output_folders.py

codecov-commenter commented 8 months ago

Codecov Report

Attention: 316 lines in your changes are missing coverage. Please review.

Comparison is base (76ce93a) 70.67% compared to head (60f159b) 67.41%.

Files Patch % Lines
python/dendro/sdk/InputFolder.py 23.30% 79 Missing :warning:
python/dendro/sdk/OutputFolder.py 31.25% 44 Missing :warning:
...hon/dendro/api_helpers/routers/processor/router.py 39.62% 32 Missing :warning:
python/dendro/sdk/AppProcessor.py 50.84% 29 Missing :warning:
...thon/dendro/api_helpers/services/gui/create_job.py 35.71% 27 Missing :warning:
python/dendro/client/submit_job.py 7.14% 26 Missing :warning:
python/dendro/sdk/_run_job_child_process.py 12.00% 22 Missing :warning:
...dendro/api_helpers/services/_create_output_file.py 10.52% 17 Missing :warning:
python/dendro/sdk/Job.py 42.30% 15 Missing :warning:
python/dendro/sdk/App.py 12.50% 14 Missing :warning:
... and 4 more
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #126 +/- ## ========================================== - Coverage 70.67% 67.41% -3.26% ========================================== Files 73 76 +3 Lines 4487 4938 +451 ========================================== + Hits 3171 3329 +158 - Misses 1316 1609 +293 ``` | [Flag](https://app.codecov.io/gh/flatironinstitute/dendro/pull/126/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flatironinstitute) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/flatironinstitute/dendro/pull/126/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flatironinstitute) | `67.41% <37.79%> (-3.26%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flatironinstitute#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.