search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

How to access E2 compute instances #180

Closed carbocation closed 4 years ago

carbocation commented 4 years ago

Now that GCP is rolling out E2 compute instances which promise a lower price, will dsub support semantics that allow us to access them while setting CPU and memory requirements, or should we explicitly ask for them by specifying named machine types?

mbookman commented 4 years ago

Current approach is that setting --machine-type explicitly is how this should be done.

That said, the announcement about the e2 instances gives pause for consideration on this. I'm interested to find if there are plans to support this in Cromwell and if so, what they look like. We are looking to keep more in-sync with these kinds of features.

Thanks for prompting about this. Will let you know...

carbocation commented 4 years ago

Actually, I just tried to run some e2 machines and it looks like they might not yet be supported in the pipelines API anyway.

gcheon commented 4 years ago

I'd also appreciate having access to e2 machines from dsub! Currently, asking for e2 machines with --machine-type e2-highmem-16 with dsub 0.3.7 results in:

googleapiclient.errors.HttpError: <HttpError 400 when requesting https://genomics.googleapis.com/v2alpha1/pipelines:run?alt=json returned "Error: validating pipeline: invalid machine type: unknown machine type">

mbookman commented 4 years ago

Resolving this as support for e2 machines was added to the Pipelines API back in March.