search

ohsu-comp-bio / funnel

Funnel is a toolkit for distributed task execution via a simple, standard API.
https://ohsu-comp-bio.github.io/funnel
MIT License
120 stars 32 forks source link

Compute: HashiCorp Nomad #628

Open matthdsm opened 4 years ago

matthdsm commented 4 years ago

Hi,

We'd like to use Funnel to schedule jobs on a HashiCorp Nomad cluster. Are there plans to support this? Or are there any resources available on how we could implement this?

Thanks M

kellrott commented 4 years ago

We haven't looked at it yet. How hard is it to set up a test cluster?

matthdsm commented 4 years ago

Fairly easy, they have quite the comprehensive manual, with pre-built vagrant (obviously) boxes, so a test cluster is just a matter of minutes. https://learn.hashicorp.com/nomad

matthdsm commented 4 years ago

From what I gather, the approach should be fairly similar to AWS batch. They also have a native go API to talk to the cluster, which should make it even more smooth.

kellrott commented 4 years ago

We're doing some work on a Kubernetes backend at the moment. We might have some time after that, but we'd need to be able to build a test cluster first. For some of the other backends, we had some luck with building docker based test environments, so that backend could be tested as part of the Travis continuous integration. I'm just curious how much work getting that kind of testing environment would take.

matthdsm commented 4 years ago

If you let me know what you need I might be able to cook something up. As opposed to go, I am used to working with docker etc.

Cheers M

matthdsm commented 4 years ago

although, running a nomad test cluster in travis would require docker-in-docker, which I'm not sure Travis allows...

M

matthdsm commented 4 years ago

Congrats on the new release! Any updates on the nomad front? Is there anything we can do to help?

Thanks M

matthdsm commented 6 months ago

Hi guys! Were you ever able to put this on the roadmap?

Thanks M

kellrott commented 6 months ago

We've been concentrating on updating to TES API compatibility v1.1 (making sure we have full conformance), and supporting NextFlow. Most of that work can be seen in #749. Once we get through that and #755 we can think about Nomad