Support Cloud Client Libraries on Google App Engine standard

[gpopovic]

I'm able to deploy it but I keep getting this error: "DistributionNotFound: The 'gcloud' distribution was not found and is required by the application"

[daspecster]

Hello @gpopovic!

@dhermes has a good skeleton project that might help you. https://github.com/dhermes/test-gcloud-on-gae

It looks like there is some extra work in supporting gcloud-python as a dependency. https://github.com/dhermes/test-gcloud-on-gae/blob/master/install_gcloud.sh

[gpopovic]

thanks @daspecster , but isn't this too much hack for something that should be simple? gcloud-golang works like as charm on both appengine standard and compute engine instances.

[daspecster]

@dhermes or @tseaver could probably shed more light on this.

Here is @dhermes post on stackoverflow talking about some of the issues. http://stackoverflow.com/a/28095663/89702

Do you have a stacktrace from "DistributionNotFound"?

[gpopovic]

@daspecster

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/run.py", line 3, in <module>
    from app import app
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/app/__init__.py", line 6, in <module>
    from gcloud import datastore
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/lib/gcloud/__init__.py", line 19, in <module>
    __version__ = get_distribution('gcloud').version
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/lib/pkg_resources/__init__.py", line 535, in get_distribution
    dist = get_provider(dist)
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/lib/pkg_resources/__init__.py", line 415, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/lib/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/base/data/home/apps/s~project-io/internal:1.393710744803082856/lib/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'gcloud' distribution was not found and is required by the application

[jgeewax]

Can someone explain the rationale behind the error here? The darth vendor path does work, but it's way more code that I'd expect just to get a hello world with gcloud-python....

[theacodes]

Because App Engine standard doesn't truly support third-party packages. We work around this with vendoring. It's semi-officially supported by darth being included as google.appengine.ext.vendor.

See here.

The rest of the hacks are just get this package and its dependencies to play nicely with GAE standard.

[daspecster]

Just a thought, what if we put these kind of issues in an FAQ or something? I guess we could just leave here but I think it might be useful to have these issues summarized in one spot.

[dhermes]

@jonparrott is the authority on installing packages for GAE

[theacodes]

Assign this to me, I'll add GAE Standard installation instructions to the documentation.

@dhermes how can I help you remove the two hacks (metadata server check, pkg_resources)?

[dhermes]

Which hacks are you referring to (link?)?

[theacodes]

metadata: https://github.com/dhermes/test-gcloud-on-gae/commit/365d51240452259d97ed583c8f07746b9ca6eae5

pkg_resources: https://github.com/dhermes/test-gcloud-on-gae/commit/4457b2f846ac8f65f38e5c38fbf2258a60a67ebe

[dhermes]

@jonparrott The metadata hack was just to avoid the HTTP hit (slowdown). The pkg_resources hack should be resolved in our library. You know packaging better than I, is there a way for pkg_resources.get_distribution to work correctly on GAE?

[theacodes]

I feel like pkg_resources should just work, I'll investigate a bit and report back.

Sometimes I wonder if my pursuit of packaging sanity in App Engine will make me lose my own sanity.

[dhermes]

I hope not, I wish robots would fix packaging instead of beating humans in Go.

[theacodes]

Update: pytz will be available in 1.9.40. The pwd module is also going to be enabled in an upcoming release, possibly 1.9.41.

[theacodes]

Closing this issue as I can confirm that gcloud-python works via vendoring on GAE standard.

pytz, pwd, etc. should make things "better", but are not strictly needed.

[theacodes]

I take that back, there's a deployment issue. Ugh.

[natb1]

Any updated information that could be provided on this would be helpful. Are there any short term workarounds? Or, do I need to migrate off of appengine standard sooner rather than later?

This was actually working for me briefly, but the issue I am running into currently is:

from gcloud import datastore
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/datastore/__init__.py", line 53, in <module>
    from gcloud.datastore.batch import Batch
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/datastore/batch.py", line 24, in <module>
    from gcloud.datastore import helpers
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/datastore/helpers.py", line 24, in <module>
    from google.type import latlng_pb2
  File "/base/data/home/apps/s~REDACTED/lib/google/type/latlng_pb2.py", line 78, in <module>
    import grpc
  File "/base/data/home/apps/s~REDACTED/lib/grpc/__init__.py", line 37, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: dynamic module does not define init function (initcygrpc)

[theacodes]

natb1: supporting standard is a priority.

You might be able to get around this by just deleting grpc from lib/

[natb1]

no dice. thanks though

from gcloud import datastore
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/datastore/__init__.py", line 53, in <module>
    from gcloud.datastore.batch import Batch
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/datastore/batch.py", line 24, in <module>
    from gcloud.datastore import helpers
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/datastore/helpers.py", line 24, in <module>
    from google.type import latlng_pb2
  File "/base/data/home/apps/s~REDACTED/lib/google/type/latlng_pb2.py", line 78, in <module>
    import grpc
ImportError: No module named grpc

I also tried removing google from lib/

from gcloud import pubsub
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/pubsub/__init__.py", line 26, in <module>
    from gcloud.pubsub.client import Client
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/pubsub/client.py", line 19, in <module>
    from gcloud.client import JSONClient
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/client.py", line 20, in <module>
    from gcloud._helpers import _determine_default_project
  File "/base/data/home/apps/s~REDACTED/lib/gcloud/_helpers.py", line 28, in <module>
    from google.protobuf import timestamp_pb2
ImportError: No module named protobuf

[natb1]

locking down googleapis_common_protos appears to do the trick (I think). I.E. this should work to deploy gcloud-python to appengine standard using vendoring

gcloud==0.18.1
googleapis_common_protos==1.2.0

[theacodes]

Thanks, @natb1. @bjwatson do you anticipate that moving grpc to an 'extra' for googleapis-common-protos will fix this?

[bjwatson]

@jonparrott Yes, I do. The extra version is on testpypi, and I will publish to normal PyPI today after I talk with the gcloud-python team.

[bjwatson]

@jonparrott @natb1 @dhermes The latest release of googleapis-common-protos now has gRPC as an extra dependency, which should fix the latest issue.

[dhermes]

Thanks @bjwatson!

[bjwatson]

No problem. Sorry for the drama this caused.

[theacodes]

There may be another issue with google.types.latlng_pb2... investigating.

[theacodes]

tests/conftest.py:39: in app
    'DATA_BACKEND': request.param
bookshelf/__init__.py:69: in create_app
    from .crud import crud
bookshelf/crud.py:15: in <module>
    from bookshelf import get_model, oauth2, storage, tasks
bookshelf/tasks.py:20: in <module>
    import psq
.tox/py34/lib/python3.4/site-packages/psq/__init__.py:25: in <module>
    from .queue import BroadcastQueue, Queue
.tox/py34/lib/python3.4/site-packages/psq/queue.py:26: in <module>
    from .storage import Storage
.tox/py34/lib/python3.4/site-packages/psq/storage.py:19: in <module>
    from gcloud import datastore
.tox/py34/lib/python3.4/site-packages/gcloud/datastore/__init__.py:53: in <module>
    from gcloud.datastore.batch import Batch
.tox/py34/lib/python3.4/site-packages/gcloud/datastore/batch.py:24: in <module>
    from gcloud.datastore import helpers
.tox/py34/lib/python3.4/site-packages/gcloud/datastore/helpers.py:24: in <module>
    from google.type import latlng_pb2
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    import sys
    _b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
    from google.protobuf import descriptor as _descriptor
    from google.protobuf import message as _message
    from google.protobuf import reflection as _reflection
    from google.protobuf import symbol_database as _symbol_database
    from google.protobuf import descriptor_pb2
    # @@protoc_insertion_point(imports)

    _sym_db = _symbol_database.Default()

    DESCRIPTOR = _descriptor.FileDescriptor(
      name='google/type/latlng.proto',
      package='google.type',
      syntax='proto3',
      serialized_pb=_b('\n\x18google/type/latlng.proto\x12\x0bgoogle.type\"-\n\x06LatLng\x12\x10\n\x08latitude\x18\x01 \x01(\x01\x12\x11\n\tlongitude\x18\x02 \x01(\x01\x42&\n\x0f\x63om.google.typeB\x0bLatLngProtoP\x01\xa2\x02\x03GTPb\x06proto3')
    )
    _sym_db.RegisterFileDescriptor(DESCRIPTOR)

    _LATLNG = _descriptor.Descriptor(
      name='LatLng',
      full_name='google.type.LatLng',
      filename=None,
      file=DESCRIPTOR,
      containing_type=None,
      fields=[
        _descriptor.FieldDescriptor(
          name='latitude', full_name='google.type.LatLng.latitude', index=0,
          number=1, type=1, cpp_type=5, label=1,
          has_default_value=False, default_value=float(0),
          message_type=None, enum_type=None, containing_type=None,
          is_extension=False, extension_scope=None,
          options=None),
        _descriptor.FieldDescriptor(
          name='longitude', full_name='google.type.LatLng.longitude', index=1,
          number=2, type=1, cpp_type=5, label=1,
          has_default_value=False, default_value=float(0),
          message_type=None, enum_type=None, containing_type=None,
          is_extension=False, extension_scope=None,
          options=None),
      ],
      extensions=[
      ],
      nested_types=[],
      enum_types=[
      ],
      options=None,
      is_extendable=False,
      syntax='proto3',
      extension_ranges=[],
      oneofs=[
      ],
      serialized_start=41,
      serialized_end=86,
    )

    DESCRIPTOR.message_types_by_name['LatLng'] = _LATLNG

    LatLng = _reflection.GeneratedProtocolMessageType('LatLng', (_message.Message,), dict(
      DESCRIPTOR = _LATLNG,
      __module__ = 'google.type.latlng_pb2'
      # @@protoc_insertion_point(class_scope:google.type.LatLng)
      ))
    _sym_db.RegisterMessage(LatLng)

    DESCRIPTOR.has_options = True
    DESCRIPTOR._options = _descriptor._ParseOptions(descriptor_pb2.FileOptions(), _b('\n\017com.google.typeB\013LatLngProtoP\001\242\002\003GTP'))
>   import grpc
E   ImportError: No module named 'grpc'

.tox/py34/lib/python3.4/site-packages/google/type/latlng_pb2.py:78: ImportError

@bjwatson @dhermes any ideas?

[bjwatson]

@jonparrott How did you trigger this?

[bjwatson]

Never mind, I know what the problem is. We need to only use the --grpc_out flag when building operations_pb2.py, but not for anything else. I'll release version 1.3.3 of googleapis-common-protos in an hour or two, once I've made this fix.

I'm really sorry for all the trouble with this transition.

[theacodes]

No worries. This is why we have a lot of tests for things like getting-started-python. :)

On Wed, Sep 7, 2016, 12:12 PM Brian J. Watson notifications@github.com wrote:

Never mind, I know what the problem is. We need to only use the --grpc_out flag when building operations_pb2.py, but not for anything else. I'll release version 1.3.3 of googleapis-common-protos https://pypi.python.org/pypi/googleapis-common-protos in an hour or two, once I've made this fix.

I'm really sorry for all the trouble with this transition.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-python/issues/1893#issuecomment-245385481, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPUc2yF4LetiG-xcAY-CqPcCvw_A5-Sks5qnwyKgaJpZM4I8yRt .

[bjwatson]

Perhaps I should run some of those tests before doing anything crazy like this again. :)

Version 1.3.3 should fix this issue: https://pypi.python.org/pypi/googleapis-common-protos

[theacodes]

@bjwatson it can be somewhat difficult to run our system tests, but let me know if you ever want to try. :)

[bjwatson]

I'll pay you a visit the next time I'm in Seattle, and maybe you can show me. :)

[geigerj]

I'm a little confused about what's going on with https://github.com/googleapis/packman/pull/109, maybe am missing some context.

As I understand, the problem is that we don't want to import grpc when it's not necessary to access the LRO service because grpc is not available in some environments, like GAE. But if we build common protos with grpc_out even just for LRO, you still have the exact same problem for the Operations messages, right?

That is, we still want operations_pb2 to be accessible from GAE, since GAE clients it will still need the Operation type defined in that module, even if they cannot access the LRO service. But if we include the service in googleapis-common-protos, GAE will not even be able to load the Operation type because it will choke on the grpc import.

So maybe we need a way to make the import grpc conditional on the environment not being GAE.

[bjwatson]

@jonparrott @geigerj Version 1.3.4 of https://pypi.python.org/pypi/googleapis-common-protos fixes the remaining issue with importing operations_pb2.py if gRPC is not installed.

[dhermes]

@jonparrott Are we good to close this out?

[theacodes]

Nope. There remains a few more issues, notably the issue with invalid character name.

[dhermes]

the issue with invalid character name

Which issue would that be?

[theacodes]

Currently when you vendor gcloud-python you'll get an error when you try to deploy because setuptools contains a file with ( in the name. I'm working to get this resolved upstream.

I think if we're going to support app engine, we should probably consider writing a tool that properly vendors in the library, perhaps something that uses the GAE-provided versions of pytz and such.

[dhermes]

First I've heard about the setuptools issue. Is that something we can fix?

FWIW we don't require pytz and ship a simplified UTC class for those that don't have it.

[theacodes]

That's good news. And no, I don't think there's anything we should do to address the filename problem.

On Fri, Sep 9, 2016, 3:51 PM Danny Hermes notifications@github.com wrote:

First I've heard about the setuptools issue. Is that something we can fix?

FWIW we don't require pytz and ship a simplified UTC class for those that don't have it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-python/issues/1893#issuecomment-246062328, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPUc_IKsAuPqGSdUnI4h14HY-WVfDNWks5qoeLTgaJpZM4I8yRt .

[layoaster]

Hi, I'm having issues when vendoring google-cloud on the GAE dev server.

I'm using virtualenv so install google-cloud with pip install google-cloud -t lib/. Have no issues importing bigquery but I get the ImportError: No module named pwd when calling to bigquery.Client()

Pinning down to the version 2.2.0 of the oauth2client doesn't fix it. I'm using latest version of the GAE SDK

[tseaver]

@layoaster Our only use of pwd is in core/google/cloud/_helpers.py, which has a conditional import:

# NOTE: Catching this ImportError is a workaround for GAE not supporting the
#       "pwd" module which is imported lazily when "expanduser" is called.
try:
    _USER_ROOT = os.path.expanduser('~')
except ImportError:  # pragma: NO COVER
    _USER_ROOT = None

The oauth2client issue is here: https://github.com/google/oauth2client/issues/578

[dhermes]

@layoaster Can you provide a stacktrace?

[theacodes]

If @layoaster is using dev_appserver, this is unsurprising. They fixed os.expanduser on production GAE, but haven't yet updated the GAE SDK to support it.

[dhermes]

@jonparrott Without a stacktrace, I'm still surprised. If you notice the code @tseaver posted above, the only ImportError that would happen is avoided by our code.

[theacodes]

Yeah but oauth2client also makes a call.

[dhermes]

I would believe that. Would be very easy to believe with a stacktrace :grinning:

[layoaster]

Sorry for not providing it before @dhermes @jonparrott

Traceback (most recent call last):
  File "/home/layo/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 267, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1519, in __call__
    response = self._internal_error(e)
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/home/layo/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/home/layo/cloud-samples/vendoring/main.py", line 8, in get
    client = bigquery.Client()
  File "/home/layo/cloud-samples/vendoring/lib/google/cloud/client.py", line 186, in __init__
    Client.__init__(self, credentials=credentials, http=http)
  File "/home/layo/cloud-samples/vendoring/lib/google/cloud/client.py", line 122, in __init__
    credentials = get_credentials()
  File "/home/layo/cloud-samples/vendoring/lib/google/cloud/credentials.py", line 87, in get_credentials
    return client.GoogleCredentials.get_application_default()
  File "/home/layo/cloud-samples/vendoring/lib/oauth2client/client.py", line 1288, in get_application_default
    return GoogleCredentials._get_implicit_credentials()
  File "/home/layo/cloud-samples/vendoring/lib/oauth2client/client.py", line 1273, in _get_implicit_credentials
    credentials = checker()
  File "/home/layo/cloud-samples/vendoring/lib/oauth2client/client.py", line 1226, in _implicit_credentials_from_files
    credentials_filename = _get_well_known_file()
  File "/home/layo/cloud-samples/vendoring/lib/oauth2client/client.py", line 1392, in _get_well_known_file
    default_config_dir = os.path.join(os.path.expanduser('~'),
  File "/home/layo/cloud-samples/vendoring/lib/python2.7/posixpath.py", line 261, in expanduser
    import pwd
  File "/home/layo/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 963, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named pwd