StanfordBioinformatics / loom

A tool for running bioinformatics workflows locally or in the cloud.
GNU Affero General Public License v3.0
29 stars 8 forks source link

Everything authenticates from service account key in settings #510

Open nhammond opened 6 years ago

nhammond commented 6 years ago

This new issue consolidates several existing issues.

Currently there are 4 independent sets of credentials used by Google Cloud.

  1. We use the gcloud commandline tool in at least one place, adding ssh keys to instances (since ansible gce module does not edit metadata after an instance is started). This uses credentials set by "gcloud auth login".
  2. We also use google.cloud.storage, which relies on the application default credentials ("gcloud auth application-default login").
  3. Ansible relies on the service account provided in Loom settings, and passes these credentials to the underlying libcloud libraries.
  4. The Loom server and workers always use the default service account, since ansible does not support setting the service account on an instance.

To fix this:

  1. Eliminate use of ansible. Work directly with libcloud instead. That allows us to stop using gcloud for setting metadata and lets us correctly set service accounts on new instances (items 1 and 4 above)
  2. Stop using google.cloud libraries in the client. To handle storage, force the client to post files directly to the server or to authenticated URLs generated by the server. If google.cloud libraries can't be used without first running "gcloud auth application-default log" in the shell, eliminate them completely, and use libcloud for the server to query storage and generate signed URLs.
nhammond commented 6 years ago

To resolve the mixed accounts issue while keeping ansible, we would have to:

  1. set ssh keys on new instance metadata using a service account key (currently use gcloud and default credentials)
  2. set service accounts on workers and server
  3. discontinue google.cloud.storage and use service accounts to authenticate storage