running dsub from docker container in CI

[yehudasabag]

I am running dsub inside a docker container which I prepared with python 2.7, google cloud sdk and dsub (you can reach it in docker hub yehudasabag/dsub-runner:v1.1). Since the dsub is supposed to run from some other CI machine, I am using a service account to authenticate to GCP, so before I am running the dsub command I am running the following commands: gcloud auth activate-service-account --key-file [json file with the service account details] gcloud config set compute/zone europe-west3-a gcloud config set project [my project name] Then I run the dsub command, expecting it to use the given service account. The service account has permissions of Genomics Admin and Storage Admin (too much, just trying to make it work...). I keep getting the following output:

Traceback (most recent call last): File "/usr/local/bin/dsub", line 10, in sys.exit(main()) File "/usr/local/lib/python2.7/site-packages/dsub/commands/dsub.py", line 933, in main dsub_main(prog, argv) File "/usr/local/lib/python2.7/site-packages/dsub/commands/dsub.py", line 922, in dsub_main launched_job = run_main(args) File "/usr/local/lib/python2.7/site-packages/dsub/commands/dsub.py", line 988, in run_main provider_base.get_provider(args, resources), File "/usr/local/lib/python2.7/site-packages/dsub/providers/provider_base.py", line 48, in get_provider args.project) File "/usr/local/lib/python2.7/site-packages/dsub/providers/google_v2.py", line 443, in init credentials) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, dkw).call(f, *args, *kw) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 206, in call return attempt.get(self._wrap_exception) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call attempt = Attempt(fn(args, kwargs), attempt_number, False) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, dkw).call(f, *args, *kw) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 206, in call return attempt.get(self._wrap_exception) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call attempt = Attempt(fn(args, kwargs), attempt_number, False) File "/usr/local/lib/python2.7/site-packages/dsub/providers/google_base.py", line 539, in setup_service credentials = oauth2client.client.GoogleCredentials.get_application_default( File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1271, in get_application_default return GoogleCredentials._get_implicit_credentials() File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1256, in _get_implicit_credentials credentials = checker() File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1187, in _implicit_credentials_from_gce if not _in_gce_environment(): File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1042, in _in_gce_environment if NO_GCE_CHECK != 'True' and _detect_gce_environment(): File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 999, in _detect_gce_environment http, _GCE_METADATA_URI, headers=_GCE_HEADERS) File "/usr/local/lib/python2.7/site-packages/oauth2client/transport.py", line 282, in request connection_type=connection_type) File "/usr/local/lib/python2.7/site-packages/httplib2/init.py", line 2135, in request cachekey, File "/usr/local/lib/python2.7/site-packages/httplib2/init.py", line 1796, in _request conn, request_uri, method, body, headers File "/usr/local/lib/python2.7/site-packages/httplib2/init.py", line 1737, in _conn_request response = conn.getresponse() File "/usr/local/lib/python2.7/httplib.py", line 1108, in getresponse raise ResponseNotReady() httplib.ResponseNotReady

From what I understand this is related to the GCP authentication. If on the same docker machine I am using (manually) the gcloud auth application-default login the cmd works.

Any ideas what may be the issue?

[yehudasabag]

After some investigation in the code I see that in setup_service in google_base the code tries to authenticate with get_application_default if no credentials were supplied. Looking around the code, no one passes this method credentials, so it alway calls the get_application_default. This approach is not suitable for running in CI, or some other machines which are not the local development machine. I think that the code should use the default GCP auth method which reads the GOOGLE_APPLICATION_CREDENTIALS environment variable and authenticate based on it. If someone wants to use default application credentials he will do it by himself using the gcloud commands. After commenting out this line, and setting the GOOGLE_APPLICATION_CREDENTIALS the flow I described above is working. If still needed to use in the code the defualt application credentials than there should be some cmd line flag to disable it IMO. WDYT? I am willing to open a PR once something is decided.

[mbookman]

Thanks for reporting @yehudasabag.

The ability to pass credentials to setup_service was added for code that uses dsub as a Python library, rather than as a command-line tool.

It should be that oauth2client.client.GoogleCredentials.get_application_default() picks up GOOGLE_APPLICATION_CREDENTIALS from the environment if it is set. See the documentation here:

https://cloud.google.com/docs/authentication/production#howtheywork

GCP client libraries use a strategy called Application Default Credentials (ADC) to find your application's credentials. When your code uses a client library, the strategy checks for your credentials in the following order:

First, ADC checks to see if the environment variable GOOGLE_APPLICATION_CREDENTIALS is set. If the variable is set, ADC uses the service account file that the variable points to.

Here's the associated code:

https://github.com/googleapis/oauth2client/blob/0d1c814779c21503307b2f255dabcf24b2a107ac/oauth2client/client.py#L1233

The execution of the "GCE check" shouldn't be happening if you set GOOGLE_APPLICATION_CREDENTIALS as the environment is checked first.

Please check that GOOGLE_APPLICATION_CREDENTIALS is being set properly and not being masked out somewhere in your environment - perhaps check for it in the environment before the call to get_application_default(). Let us know what

[yehudasabag]

Thank you for your response @mbookman You are right, I forgot to set the GOOGLE_APPLICATION_CREDENTIALS env var. For some reason I had in mind that the gcloud auth activate-service-account --key-file [json file with the service account details] should have set it, which is not true. After I set it the original code is working. Thanks you for your reply. Closing this issue.