flatironinstitute / dendro-old

Analyze neuroscience data in the cloud
https://flatironinstitute.github.io/dendro-docs/
Apache License 2.0
19 stars 2 forks source link

reorganize run method for jobs #93

Closed magland closed 10 months ago

magland commented 10 months ago

As discussed/planned, the run method (local, aws_batch, slurm) should not be associated with the installed app, but should be specified at job submission.

Compute resources now have two new config variables

DEFAULT_JOB_RUN_METHOD: could be "local", "aws_batch", or "slurm"

AVAILABLE_JOB_RUN_METHODS: comma-separated list of strings, with each string being "local", "aws_batch", or "slurm"

On job submission window, there is now a selector for choosing the run method among those available for the compute resource.

codecov-commenter commented 10 months ago

Codecov Report

Attention: 54 lines in your changes are missing coverage. Please review.

Comparison is base (1d79ab7) 75.30% compared to head (76e64eb) 75.02%.

Files Patch % Lines
python/dendro/compute_resource/SlurmJobHandler.py 68.84% 43 Missing :warning:
python/dendro/compute_resource/JobManager.py 92.10% 3 Missing :warning:
...ndro/compute_resource/register_compute_resource.py 72.72% 3 Missing :warning:
.../dendro/compute_resource/start_compute_resource.py 76.92% 3 Missing :warning:
python/dendro/compute_resource/_start_job.py 77.77% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #93 +/- ## ========================================== - Coverage 75.30% 75.02% -0.28% ========================================== Files 68 68 Lines 4162 4244 +82 ========================================== + Hits 3134 3184 +50 - Misses 1028 1060 +32 ``` | [Flag](https://app.codecov.io/gh/flatironinstitute/dendro/pull/93/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flatironinstitute) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/flatironinstitute/dendro/pull/93/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flatironinstitute) | `75.02% <76.62%> (-0.28%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flatironinstitute#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

magland commented 10 months ago

There are updated instructions for configuring a compute resource to submit to AWS Batch.

https://github.com/flatironinstitute/dendro/blob/main/doc/iaac_aws_batch.md#step-2-configure-compute-resource-controller

You just need to make sure you have the latest dendro, and set the config variables described there.

Then you'll see a "Run method" option where you can select aws_batch (or local)

image