Support Cloud Client Libraries on Google App Engine standard

[gpopovic]

I'm able to deploy it but I keep getting this error: "DistributionNotFound: The 'gcloud' distribution was not found and is required by the application"

[kutu]

new django 1.11 was added, but pytz module is missing even so it mentioned in built-in-libraries-27

[theacodes]

An update on this:

We are working with the App Engine team to bring support for these libraries to App Engine standard, however, it is a lot of engineering effort and we can't yet speak to timelines. In the meantime, these libraries are not officially supported on App Engine standard.

The recommended way to talk to Google APIs from App Engine standard remains the Google API Client Library.

You can read more about our client libraries on this page.

[skoegl]

I don't know if this is still needed. I've just translated the curl example to plain python and be able to use the vision api on google app engine standard environment. Here is my gist.

https://gist.github.com/skoegl/a322a3b66f997ef0591c5a48f5da872c

Have fun! Stefan

[jer-tx]

Possible to get any new updates about this? @jonparrott

[theacodes]

We are still working on this. My recommendations in https://github.com/GoogleCloudPlatform/google-cloud-python/issues/1893#issuecomment-322512490 are still representative of the current state of the world.

[mspichalski]

Any new updates on this?

[theacodes]

We are making progress but I still can not announce a timeline for when this will be ready. You are welcome to go ahead and try some of the libraries on standard and report any issues you run into here.

[erlichmen]

In this regards I would like to reiterate that the “thread of shame” ( https://github.com/GoogleCloudPlatform/google-cloud-python/issues/1893#issuecomment-367651481) is over a year old and still no end in sight on when we will have a consistent and working libs on GAE/py standard.

Also as said before I think there should be only one package for gcloud with a single version single change log, etc.

On Thu, 22 Feb 2018 at 19:14 Jon Wayne Parrott notifications@github.com wrote:

We are making progress but I still can not announce a timeline for when this will be ready. You are welcome to go ahead and try some of the libraries on standard and report any issues you run into here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/google-cloud-python/issues/1893#issuecomment-367752532, or mute the thread https://github.com/notifications/unsubscribe-auth/AAqOb96QS8vkrxfamlc4RBmNee2egJkrks5tXaB0gaJpZM4I8yRt .

[theacodes]

In this regards I would like to reiterate that the “thread of shame” (#1893 (comment)) is over a year old and still no end in sight on when we will have a consistent and working libs on GAE/py standard.

We have limited resources on both the Client Libraries team and the App Engine team. We are working to make this happen, but there are other things that are often higher in our list of priorities. We understand this is a important case for users, and lots of work is happening in the App Engine runtime to support these libraries (notably, we recently added gRPC to App Engine runtime). We've close, but there's still work to be done.

Also as said before I think there should be only one package for gcloud with a single version single change log, etc.

Can you expand on this?

[mdornseif]

Might something like this be a stop-gap "solution"?

try:
    import pkg_resources
    def get_distribution_dummy(name):
        class DummyObj(object):
            version = 'unknown'
        return DummyObj()

    pkg_resources.get_distribution = get_distribution_dummy
    logging.debug('disabled `pkg_resources.get_distribution() for GAE compability`')
except ImportError:
    pass

Seems pkg_resources.get_distribution() currently is only used as an convoluted way to get the current version and most libraries should also work without that.

google-cloud-bigquery==0.24.0 works for me this way.

[vinitpayal]

@jonparrott Thanks for pointing here but I am still not sure about #5012's relation to this issue as in my case package six is being imported successfully but showing exception while importing http_client by from six.moves import http_client.

[shubhamvasaikar]

I am trying to use google cloud vision v1p1pbeta1 in the GAE standard environment. This is the error that i get when i try to import from google.cloud import vision_v1p1beta1 as vision:

First time after deploying the application, I get this error:

(/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/app.py", line 4, in <module>
    from submitReview import SubmitReview
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/submitReview.py", line 8, in <module>
    from google.cloud import vision_v1p2beta1 as vision
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/lib/google/cloud/vision_v1p2beta1/__init__.py", line 22, in <module>
    from google.cloud.vision_v1p2beta1.gapic import image_annotator_client as iac
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/lib/google/cloud/vision_v1p2beta1/gapic/image_annotator_client.py", line 18, in <module>
    import google.api_core.gapic_v1.client_info
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/lib/google/api_core/gapic_v1/__init__.py", line 16, in <module>
    from google.api_core.gapic_v1 import config
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/lib/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t181713.410189522651190980/lib/grpc/__init__.py", line 22, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: dynamic module does not define init function (initcygrpc)

Subsequently, from the second time onwards, I get this error:

(/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t172858.410188745370110763/app.py", line 4, in <module>
    from submitReview import SubmitReview
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t172858.410188745370110763/submitReview.py", line 9, in <module>
    from google.cloud import vision_v1p1beta1 as vision
  File "/base/data/home/apps/s~cs410c-vasaikar/20180603t172858.410188745370110763/lib/google/cloud/vision_v1p1beta1/__init__.py", line 20, in <module>
    from google.cloud.vision_v1p1beta1 import types
ImportError: cannot import name types

[xcash]

To import Google Cloud Storage on App Engine standard I use the following monkey patching:

import requests_toolbelt.adapters.appengine
requests_toolbelt.adapters.appengine.monkeypatch()
import pkg_resources
old_get_distribution = pkg_resources.get_distribution

def mp_get_distribution(*args, **kwargs):
    # print args
    # print kwargs
    try:
        res = old_get_distribution(*args, **kwargs)
    except:
        class Mock(object):
            pass

        mock = Mock()
        mock.version = '1.9.0'
        return mock

    return res

pkg_resources.get_distribution = mp_get_distribution

from google.cloud import storage

But guys, we NEED a proper support for Google App Engine Standard. As with the old libraries it isn't even possible to generate download urls!!

[iethan]

It doesn't seem like using Google APIs with App Engine is possible. I've followed the above instructions and receive the following when using dev_appserver.py .:

in enable_sandbox THIRD_PARTY_C_MODULES.get_importable_module_names(config)) File "/Users/ethan/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/sandbox.py", line 856, in init dep_lib = import(dep_lib_name) ImportError: No module named six

requirements.txt

Flask==0.12.2 google-api-python-client==1.6.5 google-auth==1.3.0 google-auth-httplib2==0.0.3 gunicorn==19.7.1 google-cloud-bigquery==0.30.0 google-cloud-storage==1.7.0 google-cloud-datastore==1.4.0 google-cloud-language==1.0.2 requests==2.18.4 requests-toolbelt==0.8.0 gcloud==0.18.3 googleapis_common_protos==1.5.3

main.py

import logging

from flask import Flask

app = Flask(__name__)

from google.cloud import storage
from google.cloud import language

@app.route('/')
def form():
    return 'hi'

@app.errorhandler(500)
def server_error(e):
    # Log the error and stacktrace.
    logging.exception('An error occurred during a request.')
    return 'An internal error occurred.', 500

appengine_config.py

from google.appengine.ext import vendor
vendor.add('lib')

app.yaml

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app

libraries:
- name: flask
  version: 0.12
- name: ssl
  version: latest 
- name: grpcio
  version: latest  
- name: six
  version: latest

env_variables:
  SERVICE_ACCOUNT_FILE : 'credentials/service_account.json'
  BUCKET_NAME : 'gcp-nlp' 

I downloaded all packages above into a lib folder with: pip install -t lib -r requirements.txt

I'm shocked at how difficult it is to use Google Language API with GAE. Any reason they wouldn't be compatible? What am I missing here?

[anguillanneuf]

@iethan Judging from your app.yaml file, you are using Google App Engine Standard Environment. You may need to use the Google Language API in GAE Standard this way:

Your appengine_config.py looks good to me.

In requirement.txt, you would want to use GoogleAppEngineCloudStorageClient==1.9.22.1 in place of google-cloud-storage==1.7.0. The former is good to use in GAE Standard, the latter is only good to use in GAE Flexible Environment. Check out this link and pay attention to how things are organized in the left panel on the webpage as you read.

Then in main.py, do something like this (more reference):

import googleapiclient.discovery

service = googleapiclient.discovery.build('language', 'v1')

request = service.documents().analyzeEntities(body={
            "document": {
                "type": 'PLAIN_TEXT',
                "gcsContentUri": "gs://[BUCKET_NAME]/[FILE_NAME]}"
            },
            "encodingType": 'UTF8'
        })

response = request.execute()

[iethan]

@anguillanneuf Thanks for the comment. I'm getting the following error:

ntime/sandbox.py", line 310, in enable_sandbox
    THIRD_PARTY_C_MODULES.get_importable_module_names(config))
  File "/Users/ethanlyon/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/sandbox.py", line 856, in __init__
    dep_lib = __import__(dep_lib_name)
ImportError: No module named enum

I did a search on Google Cloud for the library you mentioned and am not seeing it. Am I missing something from the documentation?

[anguillanneuf]

@iethan It's definitely on the webpage. Let me grab a screenshot for you.

[theacodes]

An update for y'all:

App Engine has launched its Python 3.7 runtime for Standard based on gVisor sandboxing technology. This means that these libraries (along with a significant amount of other libraries) are now supported on App Engine standard with that runtime. You should be able to simply add these to your requirements.txt and use them as normal.

At this time we have no plans to support these libraries on the Python 2.7 runtime on App Engine Standard.

[speedplane]

@theacodes wow, this is the first time I've heard about gVisor and App Engine, it's pretty darn cool. For better or worse, Python 2.7 will be around for quite a while, so let me please give a +1 for porting this to the 2.7 environment.

[Achilles718611]

We can use Google Big Query on Google Appengine standard environment? Appengine standard environment supports only Python 2.7. And can you tell me why standard environment only supports Python 2.7? It has some limitation in pdf convert etc.

[jgeewax]

@linhui718611 : App Engine Standard announced a Python 3.7 runtime in August (see https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7). You should be able to use these libraries to talk to BigQuery from App Engine Standard when running on the Python 3.7 runtime.

If you have more questions on the functionality supported by various runtimes, it's probably better to ask on Stack Overflow than here (this issue is specifically about making sure these client libraries run on the 2.7 runtime, which currently isn't on the planned road map AFIAK).

[Achilles718611]

How we can migrate Python 2.7 runtime to Python 3.7?

[jgeewax]

@linhui718611 : It'd probably be best to head over to www.stackoverflow.com and ask this same question there. This GitHub issue tracker isn't really the right spot to help you!

[Thalius]

It would be great to have these libraries supported in standard 2.7. The new 3.7 standard still lacks things like NDB and other APIs.

[tseaver]

@Thalius with the Python 2 EOL just a little over a year out, there is pretty much zero chance that we will be backporting support to GAE Standard 2.7 at this date.

[Thalius]

@tseaver It makes sense, 2.7 is on it's way out. Luckily it won't be a problem for my current applications. I just hope we can get something like NDB and Taskqueue for the 3.7 standard environment.

[maltem-za]

@theacodes First off, thank you to both the Client Libraries and App Engine teams for their respective hard work.

Reviving something quite old here that I just read further up:

Also as said before I think there should be only one package for gcloud with a single version single change log, etc.

Can you expand on this?

I don't know if this is what @erlichmen meant, but fom my perspective, a lot of frustration stems from the fact that https://cloud.google.com/apis/docs/client-libraries-explained even needs to exist. Add to that "special snowflake" client libs like this one: https://github.com/GoogleCloudPlatform/appengine-gcs-client.git

I have been developing App Engine apps for many years now, and I can honestly say that this has been one of the most confusing aspects all along (which @&#! client library to use and which particular set of instructions/docs to follow). It used to be that the various services/APIs were baked in (NDB, task queue, [blob]storage etc.) which made life really easy. I do appreciate the need to separate concerns here and have the different products and teams be able to move independently as far as possible. The restructuring of the platform and work that has gone into the new libs makes sense to me, it's just that in the wake of all this the developer experience has suffered. The different libs don't have the same features and sometimes it's not clear at all which should be used where and why.

For example, at my company we didn't even realise that google-cloud was not (officially) supported in the 2.7 standard environment. It took the deprecation of the all-inclusive Python package, which caused stuff* to break, which caused a re-investigation of what we are, and are supposed to be, using. Up to this point google-cloud seemed to finally be the single, modern client lib that covered all of the APIs and could be used everywhere, that I think many developers are wishing for.

It's not like we're living under a rock either. I'm subscribed to the Google Cloud newsletter and Cloud SDK update notifications. Often I feel like I'm wading through conflicting information from different doc pages or READMEs. Case in point: The top-level README for this repo states that these libs are not supported on the 2.7 runtime environment at all, but drill into one of the sub-components, for example storage, and now it sounds like the 2.7 runtime environment is supported at least until Jan 2020. Huh?

Anyway, a lot of good work is happening, keep it up and I'm sure things will get better. This may be the wrong place for this kind of feedback, but I thought seeing as you asked for expansion... I have also left feedback on this before in other ways. Please will you see that it reaches the relevant people?

*side note: we are still successfully using google-cloud (specifically google-cloud-storage since the all-inclusive package was deprecated) to talk to GCS in the 2.7 runtime environment. To get this to work we had to follow the instructions for configuring requests as detailed here. I don't know if google-cloud-storage is a special case in this regard - it seems like it might be and that it might take a lot more hoop-jumping to get other parts of the lib to work. We're also only doing uploads using the new lib so we're using a very limited set of functions.

[theacodes]

@maltem-za thank you for your respectful, thoughtful feedback. We do really appreciate it.

I definitely agree things are more confusing today that before, and it really has a lot to do with the scale. When Google Cloud started (before it was Google Cloud, even), we had one real product (App Engine) that supported three languages. Within that product, we had a handful of services. This is a much easier developer experience problem to tackle than the one we face today: we have to support ~50 cloud products across ~8 languages on many platforms. As far as I know, we're the only people attempting to do something at this scale, a despite being Google we are limited on resources, so we're going to have some missteps.

for example storage, and now it sounds like the 2.7 runtime environment is supported at least until Jan 2020. Huh?

I think we can clarify that a bit, @crwilcox. The intent here is that:

• We've never supported App Engine Python 2.7.
• We support 2.7 basically everywhere else until 2020.

side note: we are still successfully using google-cloud (specifically google-cloud-storage since the all-inclusive package was deprecated) to talk to GCS in the 2.7 runtime environment.

I'm glad it works for you. There's several workarounds that let people use them, but we determined that these workarounds aren't suitable for us to tell all of our users to adopt and the performance and reliability after applying these workarounds isn't something we feel comfortable supporting. If it works for you, great! just know that we can't really support it.

(Coincidentally, Storage is indeed easier to hack to work in App Engine standard. The gRPC-based clients are much much harder)

[rririanto]

Thank you so much @xcash I've finally found the solution after struggling for 2 days tracing & bugging the error. I faced the problem of DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application

Then I come up with @xcash solution by changing this file & line

File "/base/data/home/apps/s~citric-aleph-611/20190413t015248.417442685994206727/lib/google/api_core/init.py", line 23, in version = get_distribution("google-api-core").version import pkg_resources

version = pkg_resources.get_distribution("google-api-core").version

into

 import requests_toolbelt.adapters.appengine
 requests_toolbelt.adapters.appengine.monkeypatch()
 import pkg_resources
 old_get_distribution = pkg_resources.get_distribution
 def mp_get_distribution(*args, **kwargs):
    try:
        res = old_get_distribution(*args, **kwargs)
    except:
        class Mock(object):
            pass
        mock = Mock()
        mock.version = '1.9.0'
        return mock
    return res
 pkg_resources.get_distribution = mp_get_distribution

Then I faced the error about PermissionDenied: 403 IAM permission 'dialogflow.sessions.detectIntent' on 'projects/XXXX-live/agent' denied.

Then I come up with @iethan solution to set the env: env_variables: GOOGLE_APPLICATION_CREDENTIALS : "xxx-en-ffb0659d5d9e.json" SERVICE_ACCOUNT_FILE : "xxx-en-ffb0659d5d9e.json"

Thank you guys again!

[wescpy]

It's 2020, and (during quarantine) I went to dig out my old App Engine Python 2 apps to move them to the latest Cloud libraries and encountered both these problems (DistributionNotFound: The 'google-cloud-ndb' distribution was not found and is required by the application and ImportError: dynamic module does not define init function (initcygrpc)) as well as a 3rd (ImportError: No module named pkg_resources). Based on a combination of reading the docs, SO, and generic Google search, I resolved all 3 issues with:

1. Reference the built-in grpcio and setuptools libraries in app.yaml:

   libraries:
   - name: grpcio
   version: 1.0.0
   - name: setuptools
   version: 36.6.0

2. Add pkg_resources to appengine_config.py so the whole thing looks like:

   
   import pkg_resources
   from google.appengine.ext import vendor

Set PATH to your libraries folder.

PATH = 'lib'

Add libraries installed in the PATH folder.

vendor.add(PATH)

Add libraries to pkg_resources working set to find the distribution.

pkg_resources.working_set.add_entry(PATH)