search

qiime2 / galaxy-tools

Official QIIME 2 tools for Galaxy
BSD 3-Clause "New" or "Revised" License
1 stars 4 forks source link

Support for parsl in Galaxy? #61

Closed bernt-matthias closed 1 month ago

bernt-matthias commented 4 months ago

For my reference https://parsl.readthedocs.io/en/latest/userguide/execution.html

We need to explore if the data-paralellism offered by Galaxy (via collection) is sufficient for the more compute intensive jobs coming with metagenomics. Or if it is possible to support parsl jobs in Galaxy.

@ebolyen can you give me a link for docs how the configuration, e.g. for an execution in SLURM would look like? You mentioned toml based config files.

ebolyen commented 1 month ago

Hey @bernt-matthias,

We use toml internally but the way parsl is generally configured would be as a Python object that is initialized and represents the config.

The relevant docs are here: https://parsl.readthedocs.io/en/latest/userguide/configuring.html#how-to-configure https://parsl.readthedocs.io/en/latest/stubs/parsl.providers.SlurmProvider.html#parsl.providers.SlurmProvider https://develop.qiime2.org/en/latest/framework/how-to-guides/parallel-configuration.html

Another approach is to use our partition methods directly and then use Galaxy's built in collection system, rather than trying to get parsl and galaxy's various backends to play together.

bernt-matthias commented 1 month ago

use Galaxy's built in collection system

Seems much simpler. Lets see how far we get with this.

ebolyen commented 1 month ago

I've created a new issue to make sure that the way we implement these parallel pipeline (via hidden actions) will be accessible to Galaxy users so they can do the map operation via Galaxy directly: https://github.com/qiime2/q2galaxy/issues/57