galaxyproject / pulsar

Distributed job execution application built for Galaxy
https://pulsar.readthedocs.io
Apache License 2.0
37 stars 50 forks source link

Batch-mode Pulsar #250

Open jmchilton opened 3 years ago

jmchilton commented 3 years ago

The two container job pod approach to Pulsar is very similar to the prior work that targeting Apache Mesos with Pulsar. Generalizing this approach one can imagine creating a batch script that setup a Pulsar application in this modality and takes care of staging files in and out, cancelling, etc.. This script could then be qsub-ed and used as a server-less Pulsar to really increased the distributed nature of Pulsar. This would be useful for usegalaxy.org and could enable things such as Condor Flocking, etc..

This would be an important piece of work for fully utilizing the power of iRODS on usegalaxy.org also.