Making it easier to use the google-auth library in App Engine

[hiranya911]

I had to jump through a number of hoops to get the requests transport working in the App Engine environment:

1. Enable billing for the project
2. Vendor in the requests-toolbelt library and enable the monkeypatch (as documented here)
3. Add the ssl library to the app.yaml:

libraries:
- name: ssl
  version: latest

4. Enable sockets as documented here

Only by doing all the above, that I could get the HTTPS request to Google OAuth2 token servers working. Is this to be expected? Is there anything that can be done to make using this library in App Engine easier?

[hiranya911]

Here's one of the errors I observed in my tests:

File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/main.py", line 15, in get
    status = db.reference('status').get()
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/firebase_admin/db.py", line 135, in get
    return self._client.request('get', self._add_suffix())
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/firebase_admin/db.py", line 600, in request
    return self._do_request(method, urlpath, **kwargs).json()
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/firebase_admin/db.py", line 631, in _do_request
    resp = self._session.request(method, self._url + urlpath, auth=self._auth, **kwargs)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/requests/sessions.py", line 488, in request
    prep = self.prepare_request(req)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/requests/sessions.py", line 431, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/requests/models.py", line 309, in prepare
    self.prepare_auth(auth, url)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/requests/models.py", line 540, in prepare_auth
    r = auth(self)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/firebase_admin/db.py", line 674, in __call__
    req.headers['Authorization'] = 'Bearer {0}'.format(self._app._get_token())
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/firebase_admin/__init__.py", line 211, in _get_token
    self._token = self._credential.get_access_token()
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/firebase_admin/credentials.py", line 97, in get_access_token
    self._g_credential.refresh(_request)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/google/oauth2/service_account.py", line 310, in refresh
    request, self._token_uri, assertion)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/google/oauth2/_client.py", line 143, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/google/oauth2/_client.py", line 104, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/base/data/home/apps/s~hjk-test-storage/20170706t173207.402488776508484318/lib/google/auth/transport/requests.py", line 115, in __call__
    raise exceptions.TransportError(exc)
TransportError: ('Connection broken: IncompleteRead(205 bytes read)', IncompleteRead(205 bytes read))

[hiranya911]

I'm also getting the following import error when testing on local dev app server:

ImportError: No module named google.auth

The library is definitely installed in the system, and also in the vendored lib directory.

[theacodes]

@hiranya911 thanks for filing. Unfortunately, yes, you do have to do all of those steps to get requests and SSL working on standard. I'd be happy to accept a PR to add those steps to the user guide in the documentation.

As for your error when running tests with the local dev app server, how are you running the tests?

[hiranya911]

I was simply trying to launch the dev app server from command-line:

dev_appserver.py path/to/app.yaml

[theacodes]

@hiranya911 did you have the vendor setup in appengine_config.py?

[hiranya911]

Yes, I have the following in appengine_config.py:

from google.appengine.ext import vendor

vendor.add('lib')

I managed to resolve the module loading issue by using the following instead:

# appengine_config.py
import os
import google
from google.appengine.ext import vendor

lib_directory = os.path.dirname(__file__) + '/lib'

# Change where to find the google package (point to the lib/ directory)
google.__path__ = [os.path.join(lib_directory, 'google')] + google.__path__

# Add any libraries install in the "lib" folder.
vendor.add(lib_directory)

It would be nice if I could avoid having to do this though.

[theacodes]

Weird - you shouldn't need to do that google.__path__ manipulation at all.

[stibi]

Hi, I just run into the ImportError problem, but my case is an AWS Lambda function. I'm using kubernetes-python client, which has google-auth as a dependency and my code don't run because of the error.

ImportError: No module named google.auth

I noticed that there is no google/__init__.py:

$ ls -l site-packages/google                                                                                                                                                                                       
total 0                                                                                                                                                                                                            
drwxr-xr-x 4 stibi wheel 600 Aug  3 15:03 auth
drwxr-xr-x 2 stibi wheel 240 Aug  3 15:03 oauth2

The file actually exist in repository: https://github.com/GoogleCloudPlatform/google-auth-library-python/blob/master/google/__init__.py

The thing is that the import works, when I add the __init__.py there. The file is missing intentionally? Why is that?

[theacodes]

@stibi how do you install dependencies for AWS Lambda? (I am unfamiliar with the product)

[stibi]

@jonparrott it's a ZIP with everything bundled together:

$ unzip -l staging-deploy-lambda-0c0442e9.zip

https://gist.github.com/stibi/b8bcba1a149f8ef3afb8280363facedd

[theacodes]

Oh, that won't work at all. Simply dropping packages into the root of your project doesn't magically make all of Python's import machinery work. Notably, namespace packages and package resources won't work.

You'll need to actually stuff them in a sub directory and call addsitedir as soon as possible to set up the directory as a site-packages directory. You can probably use the library I wrote for App Engine standard to do this: https://github.com/jonparrott/darth-vendor

[stibi]

Ok, I thought that the problem will be most probably in loading the dependencies…this is how AWS Lambda packaging works :(

Thanks for the hints, I'll try to find a workaround.

[jc275]

I had the same ImportError with google.auth on dev_appserver as @hiranya911, whose workaround also fixed it for me.

@jonparrott Regarding the google.__path__ manipulations, I noticed that the following line was indeed necessary

google.__path__ = [os.path.join(lib_directory, 'google')] + google.__path__

whilst the following alternative didn't solve the problem

google.__path__ = google.__path__ + [os.path.join(lib_directory, 'google')]

[eyalev]

Shouldn't the appengine_config.py change (https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/169#issuecomment-315417916) be added to the docs?

Otherwise the lib doesn't work on dev_appserver..

[yamen23ali]

I've just run into the same case, it turned out there was a conflict I had a module named google in my project, and since the ( google) package in ( site-packages) doesn't have (init.py) it was kind of overriden by python.

So if anyone encounter the same problem: 1- Make sure you have an updated (pip, setuptools) versions. 2- Make sure you don't have a conflict with another local folder.

[yuyang199226]

you can try pip install google-auth -U

[shurshilov]

in my case i Added ### sudo python3 -m pip install google-api-python-client SUDO)))