Cromwell should have the ability to launch nodes into GCP Subnets

[davidbernick]

Edit (by @cjllanwarne) in light of #4806:

Following #4806 we will be able to read Google project metadata to specify a VPC network and subnet.

Therefore what will remain for this ticket is making the same functionality available on a per-workflow basis... eg an ability to supply the same network/subnetwork information via workflow-options?

---

Original issue text:

https://cloud.google.com/vpc/docs/vpc -- for a primer on GCP Subnets.

Users should be able to tell Cromwell to launch nodes into a subnet.

For environments like Firecloud, we should have some mechanism (like maybe SAM) to make sure the user actually has the right to use a particular subnet.

The main reason to do this is https://cloud.google.com/vpc/docs/using-flow-logs -- we want to be able to monitor traffic in and out of the network for more significant audited environments. So the driver is ultimately "compliance". But it's probably a good idea anyhow.

After this is done, please work with FC team to make sure they can take advantage of this. I'm not sure who to tag to make sure this cross-team work is done.

[cjllanwarne]

A few immediate thoughts:

• What's the granularity of this value? ie per workflow, per cromwell server, per task?
• Should this be a "happens without the end-user knowing" or an "end user has to choose" type of a thing?
  • Or maybe it's "in Firecloud it must not be overridden but outside of Firecloud it should be more flexible"
• Should this value (wherever it gets specified) be considered when deciding whether or not to call cache to a previous run (which may have been on a different subnet)?

[ruchim]

@dvoet would you be the right person to coordinate with when this functionality is ready to be handed off to someone in FireCloud?

[geoffjentry]

Note for implementer that #4005 was closed as a dupe of this, but double check that ticket before starting

[davidbernick]

It looks like there’s a subnetwork field in the PAPI request: https://cloud.google.com/genomics/reference/rest/Shared.Types/Metadata?hl=RU#network (thanks @geoffjentry )

[davidbernick]

What can we do to make sure this gets done? We need this for FC users (specifically) right now -- basically want to be able to capture VPC traffic. So FC and Compliance is the driver. Since this is cross-team, who owns the work? Who can get it on the schedule? FC PAPI nodes should be launched in a Subnet, is the main issue.

[davidbernick]

https://broadinstitute.atlassian.net/plugins/servlet/mobile#issue/GAWB-4001 for Workbench tracking.

[ruchim]

The goal is to start this ticket mid-Feb

[ruchim]

The goal is to start this ticket mid-Feb

[danbills]

Emerald is hitting this as part of the Joint Genotyping effort which scatters 100K wide. Not high urgency as the next batch is not expected until next quarter or even 2.

[cjllanwarne]

One mechanism to support this (using labels on Google projects) will be provided by #4806, per Terra requirements. Other mechanisms (workflow options, ???) might follow.