Can't import google.protobuf.timestamp_pb2 from App Engine

[barakcoh]

I'm getting ImportError: No module named protobuf from the line from google.protobuf import timestamp_pb2 because the google module refers to the AppEngine SDK.

This is a well-known issue that's been around for years. Maybe you can bundle protobuf directly inside gcloud-python instead of an external dependency?

[dhermes]

Thanks for reporting and for upgrading!

I am curious about a few things:

1. How do you vendor in gcloud-python for App Engine?
2. Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)
3. Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)
4. Are you modify the gcloud module object so that it acts like a namespace package? If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

[theacodes]

Are you modify the gcloud module object so that it acts like a namespace package? If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

I don't think this shouldn't be necessary anymore, I'm interested to see if that's somehow changed.

[dhermes]

Me too. Thanks for chiming in @jonparrott I was going to ping you.

[theacodes]

I should really write an App Engine test case.

[dhermes]

@SaMnCo also reported this import error, it may be

• we don't understand our requirements like we think we do.
• pip doesn't upgrade deps in setup.py as I expect
• a PYTHONPATH issue

[dhermes]

@SaMnCo can you do a pip show protobuf, pip show gcloud in the environment where this is failing and then make sure the same versions are imported:

$ python -c 'import google.protobuf; print(google.protobuf.__version__)'
$ python -c 'import gcloud; print(gcloud.__version__)'

[kvdb]

@dhermes I think I've got a similar problem as the reporter of the issue. Perhaps I can help by providing my answers to your questions:

How do you vendor in gcloud-python for App Engine?

In appengine_config.py:

from google.appengine.ext import vendor                                         
vendor.add('libs')                                                              
import google.protobuf; print(google.protobuf.__version__)                      
import gcloud; print(gcloud.__version__) 

Results in:

3.0.0b2
0.10.0

Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)

It's in libs, so yes.

Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)

  from gcloud import storage
File "/x/libs/gcloud/storage/__init__.py", line 43, in <module>
  from gcloud.storage.blob import Blob
File "/x/libs/gcloud/storage/blob.py", line 27, in <module>
  from gcloud._helpers import _rfc3339_to_datetime
File "/x/libs/gcloud/_helpers.py", line 26, in <module>
  from google.protobuf import timestamp_pb2
ImportError: No module named protobuf

[dhermes]

@kvdb Thanks a lot. I'm wondering now if it may be related to Darth-Vendor and namespace packages not being the best of friends. I need to dig in and reproduce. Can you confirm that's what you used to vendor in your package?

[theacodes]

Darth is part of the SDK (from google.appengine.ext import vendor) so that's definitely what's being used. It explicitly has support for namespaces. So I'm also unsure what's going on.

[dhermes]

Maybe it's an issue with old versions of pip? The module certainly shows up:

$ mkdir lib
$ pip install -t lib gcloud
$ cd lib/google/protobuf/
$ ls -1 *pb2.py | egrep -v unittest
any_pb2.py
api_pb2.py
descriptor_pb2.py
duration_pb2.py
empty_pb2.py
field_mask_pb2.py
source_context_pb2.py
struct_pb2.py
timestamp_pb2.py
type_pb2.py
wrappers_pb2.py

Going to run a test app in dev_appserver to verify import works.

[dhermes]

OK I just confirmed this works in dev_appserver as well.

@kvdb Have you tried clearing out lib and starting fresh? What version of pip are you using?

[kvdb]

pip 8.0.2 Yes, I cleared my virtualenv/libs before reporting.

[dhermes]

Very strange! Does the timestamp_pb2.py file exist and just the import fails or the file never shows up?

[ernestoalejo]

I was having this error outside App Engine when using an old version of pip and not using the full command to install:

pip install --upgrade gcloud

I was using pip -r requirements.txt instead of pip --upgrade -r requirements.txt and the timestamp_pb2.py file was not present.

[ernestoalejo]

Outside App Engine means inside Docker, using the google/debian:wheezy image.

[dhermes]

Thanks for the report @ernestoalejo! @kvdb any thoughts?

[dhermes]

@kvdb Does the timestamp_pb2.py file exist and just the import fails or the file never shows up?

---

I'm closing for now since I can't reproduce and @ernestoalejo answer indicates --upgrade resolves the issue. @kvdb I'm happy to re-open if it still comes up.

[dhermes]

@vsubramani can you give some info on your install:

from google import protobuf
print(protobuf.__version__)
print(protobuf.__path__)
print(protobuf.__path__[0])
import os
pb_files = sorted([fi for fi in os.listdir(protobuf.__path__[0])
                   if fi.endswith('.py')])
print('\n'.join(pb_files))

[jjangsangy]

So I had the same issue running from gcloud import storage inside a docker container output an error.

In [3]: from gcloud import storage
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-a2d587421817> in <module>()
----> 1 from gcloud import storage

/venv/lib/python2.7/site-packages/gcloud/storage/__init__.py in <module>()
     41
     42 from gcloud.storage.batch import Batch
---> 43 from gcloud.storage.blob import Blob
     44 from gcloud.storage.bucket import Bucket
     45 from gcloud.storage.client import Client

/venv/lib/python2.7/site-packages/gcloud/storage/blob.py in <module>()
     25 from six.moves.urllib.parse import quote
     26
---> 27 from gcloud._helpers import _rfc3339_to_datetime
     28 from gcloud.credentials import generate_signed_url
     29 from gcloud.exceptions import NotFound

/venv/lib/python2.7/site-packages/gcloud/_helpers.py in <module>()
     25 from threading import local as Local
     26
---> 27 from google.protobuf import timestamp_pb2
     28 import six
     29 from six.moves.http_client import HTTPConnection

ImportError: cannot import name timestamp_pb2

Running

>>> from google import protobuf
>>> print(protobuf.__version__)
>>> print(protobuf.__path__)
>>> print(protobuf.__path__[0])
>>> import os
>>> pb_files = sorted([fi for fi in os.listdir(protobuf.__path__[0])
                   if fi.endswith('.py')])
>>> print('\n'.join(pb_files))
__init__.py
descriptor.py
descriptor_database.py
descriptor_pb2.py
descriptor_pool.py
message.py
message_factory.py
proto_builder.py
reflection.py
service.py
service_reflection.py
symbol_database.py
text_encoding.py
text_format.py

This was fixed by adding this line into my Dockerfile

RUN pip install --upgrade gcloud

It looks like you have to force the gcloud installation to upgrade. This was the output while building my docker container.

Step 13 : RUN pip install --upgrade gcloud
 ---> Running in 4670a504231c
Requirement already up-to-date: gcloud in /venv/lib/python2.7/site-packages
Requirement already up-to-date: pyOpenSSL in /venv/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: six in /venv/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: httplib2>=0.9.1 in /venv/lib/python2.7/site-packages (from gcloud)
Collecting protobuf!=3.0.0.b2.post1,>=3.0.0b2 (from gcloud)
  Downloading protobuf-3.0.0b2.post2-py2-none-any.whl (331kB)
Collecting oauth2client>=2.0.1 (from gcloud)
  Downloading oauth2client-2.0.2.tar.gz (68kB)
Requirement already up-to-date: googleapis-common-protos in /venv/lib/python2.7/site-packages (from gcloud)
Requirement already up-to-date: cryptography>=1.3 in /venv/lib/python2.7/site-packages (from pyOpenSSL->gcloud)
Requirement already up-to-date: setuptools in /venv/lib/python2.7/site-packages (from protobuf!=3.0.0.b2.post1,>=3.0.0b2->gcloud)
Requirement already up-to-date: pyasn1>=0.1.7 in /venv/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Requirement already up-to-date: pyasn1-modules>=0.0.5 in /venv/lib/python2.7/site-packages (from oauth2client>=2.0.1->gcloud)
Collecting rsa>=3.1.4 (from oauth2client>=2.0.1->gcloud)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
Collecting enum34 (from cryptography>=1.3->pyOpenSSL->gcloud)
  Downloading enum34-1.1.3-py2.py3-none-any.whl (61kB)
Requirement already up-to-date: ipaddress in /venv/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->gcloud)
Requirement already up-to-date: idna>=2.0 in /venv/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->gcloud)
Requirement already up-to-date: cffi>=1.4.1 in /venv/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->gcloud)
Requirement already up-to-date: pycparser in /venv/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->gcloud)
Building wheels for collected packages: oauth2client
  Running setup.py bdist_wheel for oauth2client: started
  Running setup.py bdist_wheel for oauth2client: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/3f/1b/4b/7d18513fd8ed10976cf4bb881bd0a2b549d4d22d245d0ff1a6
Successfully built oauth2client
Installing collected packages: protobuf, rsa, oauth2client, enum34
  Found existing installation: protobuf 3.0.0a3
    Uninstalling protobuf-3.0.0a3:
      Successfully uninstalled protobuf-3.0.0a3
  Found existing installation: rsa 3.3
    Uninstalling rsa-3.3:
      Successfully uninstalled rsa-3.3
  Found existing installation: oauth2client 1.5.2
    Uninstalling oauth2client-1.5.2:
      Successfully uninstalled oauth2client-1.5.2
  Found existing installation: enum34 1.1.2
    Uninstalling enum34-1.1.2:
      Successfully uninstalled enum34-1.1.2
Successfully installed enum34-1.1.3 oauth2client-2.0.2 protobuf-3.0.0b2.post2 rsa-3.4.2

[dhermes]

Thanks for the note. Unfortunately there's not much we can do about the way Python packaging works beyond declaring the minimum versions we require.

[kvdb]

The problems are still present using the latest gcloud. In setup.py, I read the protobuf dependencies are defined as:

'protobuf >= 3.0.0b2, != 3.0.0.b2.post1'

I think I had problematic version 3.0.0b2 cached, that's why it was continued to be used within gcloud. When upgrading protobuf to protobuf-3.0.0b2.post2, it works. Therefore, I'd recommend defining the dependency in setup.py as follows:

'protobuf >= 3.0.0b2.post2'

[dhermes]

What do you mean by a "problematic version 3.0.0b2"? I just did a fresh install and timestamp_pb2 is present:

$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install 'protobuf == 3.0.0b2.post2'
(venv) $ ls venv/lib/python2.7/site-packages/google/protobuf/timestamp* -1
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.py
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.pyc

[dhermes]

Ahhh hold on:

(venv) $ pip show protobuf
---
Metadata-Version: 2.0
Name: protobuf
Version: 3.0.0b2.post2
...

[dhermes]

OK I'll use the correct version this time

$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install 'protobuf == 3.0.0b2'
(venv) $ ls venv/lib/python2.7/site-packages/google/protobuf/timestamp* -1
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.py
venv/lib/python2.7/site-packages/google/protobuf/timestamp_pb2.pyc
(venv) $ pip show protobuf
---
Metadata-Version: 2.0
Name: protobuf
Version: 3.0.0b2
...

[kvdb]

I see your point, if I can reproduce it properly, I'll get back to you.

[dhermes]

Thanks!

[dgaedcke]

I'm having this exact same problem.....it's due to the order of entries in sys.path.......

here is my (abbreviated) path: ~/dev/NEW_TouchstoneMicroservices /usr/local/google_appengine ~/dev/NEW_TouchstoneMicroservices/venv/lib/python27.zip ~/dev/NEW_TouchstoneMicroservices/venv/lib/python2.7 ... /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7 .... ~/dev/NEW_TouchstoneMicroservices/venv/lib/python2.7/site-packages

You can see that: /usr/local/google_appengine comes BEFORE ..../venv/lib/python2.7/site-packages

and there is a "google" directory in both locations... but the Google directory in "google_appengine" does not contain the "protobuf" module....

Unfortunately, because I'm running the endpoints (python /usr/local/bin/endpointscfg.py get_discovery_doc -f rpc) script, I don't have the option to modify the python path in advance....

I love the Google PAAS but the dev-server config stuff is a constant frustration...

how do I fix this??

[dhermes]

@jonparrott any recommendations? @dgaedcke normally I would just say modify your PYTHONPATH, but I'm not sure if the GAE appserver hijacks the path.

[theacodes]

I don't think there's a good answer for endpointscfg.py, other than modifying it directly. I'll assign this to me to reproduce and try to suggest a temporary workaround.

[theacodes]

@dgaedcke it would be great if you could give me as much information as possible about:

1. How you install your packages
2. What's in your appengine_config.py
3. Does dev_appserver.py work for you?

[dgaedcke]

@jonparrott -- thanks for your help 1) SETUP

• I installed Python 2.7 & pip with Homebrew (OSX El Capitan)
• I create a venv directory inside my project
• I activate the venv and then
• pip --upgrade -r requirements.txt
• then I symlink "venv/lib/python2.7/site-packages" to "lib" at top level of my project dir

2) appengine_config

import endpoints
# I had to add the next line because Ferris was throwing an error about not being able
# to find that key on the module
endpoints.API_EXPLORER_CLIENT_ID = '292824132082.apps.googleusercontent.com'

from google.appengine.ext import vendor
vendor.add('lib')

3) dev_appserver.py it works as long as I comment out any attempt to run: from gcloud import storage if I leave that line enabled, it get the import problem described in this thread...

[theacodes]

ext.vendor should support virtualenvs directly. Can you try not symlinking and just doing vendor.add('venv')?

Is this an App Engine Standard application or a Flexible app using the compat runtime?

[dgaedcke]

it's a standard app......I'm not even sure what Flexible is but I've seen it pop up lately in some docs...

sure, I can try that but as you saw in my initial post, the problem stems from both directories having a "google" directory, but the first one does not have a "protobuf" module; I would not expect the symlink to change PATH order..... back shortly with results...

[theacodes]

It won't, but it does help reduce weirdness. Less weirdness is better.

[dgaedcke]

got a bit farther this time..... I don't have 100% test coverage, so I've learned that the best way to exercise ALL of my imports is to run "endpointscfg.py" and so that's how I generated this trace:

Traceback (most recent call last): File "/usr/local/bin/endpointscfg.py", line 133, in <module> run_file(__file__, globals()) File "/usr/local/bin/endpointscfg.py", line 129, in run_file execfile(_PATHS.script_file(script_name), globals_) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 561, in <module> main(sys.argv) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 557, in main args.callback(args) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 443, in _GenDiscoveryDocCallback application_path=args.application) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 302, in _GenDiscoveryDoc application_path=application_path) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/endpointscfg.py", line 187, in GenApiConfig module = __import__(module_name, fromlist=base_service_class_name) File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 27, in <module> from uidisplay_service import DisplayService File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/uidisplay_service.py", line 27, in <module> from common.blobstore.storage import upload_file File "/Users/dgaedcke/dev/TouchstoneMicroservices/common/blobstore/storage.py", line 18, in <module> from gcloud import storage File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/gcloud/storage/__init__.py", line 45, in <module> from gcloud.storage.client import Client File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/gcloud/storage/client.py", line 19, in <module> from gcloud.client import JSONClient File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/gcloud/client.py", line 17, in <module> from oauth2client.service_account import ServiceAccountCredentials ImportError: cannot import name ServiceAccountCredentials

[theacodes]

I wasn't able to reproduce your timestamp_pb2 import. I did the following:

1. virtualenv venv
2. venv/bin/pip install gcloud (note: I did not activate the virtualenv, but it didn't seem to make a difference.
3. added vendor.add('venv') to dev_appserver.py.
4. imported gcloud.storage in my app.
5. ran dev_appserver.py.

[theacodes]

Okay, the new error you have is more familiar to me. Let me track down the workaround for that.

[dhermes]

The new error is likely just the wrong version of oauth2client (pre-2.0) which may also be a path error (since oauth2client ships with the SDK)

[theacodes]

@dgaedcke see #1705. In summary:

Use an appengine_config.py like so:

"""
`appengine_config.py` is automatically loaded when Google App Engine
starts a new instance of your application. This runs before any
WSGI applications specified in app.yaml are loaded.
"""

import sys
from google.appengine.ext import vendor

# Third-party libraries are stored in "lib", vendoring will make
# sure that they are importable by the application.
vendor.add('lib')

def unload_module(module_name):
    target_modules = [m for m in sys.modules if m.startswith(module_name)]
    for m in target_modules:
        if m in sys.modules:
            del sys.modules[m]

unload_module('oauth2client')
unload_module('pyasn1_modules')

Make sure you have this at the top of your API-defining modules (e.g., the scripts referenced by app.yaml):

try:
    appengine_config
except NameError:
    import appengine_config

[dgaedcke]

that fixed it!! Thank you guys very much......

it seems like that also solved the problem of Ferris not being able to find: API_EXPLORER_CLIENT_ID

question for you: All of my "main" (API defining) modules already had: import appengine_config with no problems......so why is the try-except approach preferred

[theacodes]

Just in case you don't have it. You don't have to use the try-except approach. The magic was in unloading the modules.

[dgaedcke]

I'm assuming one of my venv packages pulled down these "unloaded" modules as dependencies..... Will it create a problem if those packages re-import them again at some later point???

[theacodes]

I'm not sure what you're asking, but the crux of the issue is this:

1. You start endpointscfg.py.
2. endpointscfg.py imports oauth2client. At this point, sys.path only contains the App Engine SDK, so the App Engine SDK's version of oauth2client<2.0 is loaded into sys.path.
3. endpointscfg.py tries to import your code.
4. Your code imports appengine_config.py which does path manipulation.
5. Your code tries to import gcloud, which in turn tries to import oauth2client.
6. Because endpointscfg.py had previously imported oauth2client, it's already in sys.modules so not search is done and the newer, vendored version isn't used.
7. Because gcloud expects oauth2client>2.0 but gets oauth2client<2.0 it explodes.

Adding the trickery to unload oauth2client in appengine_config.py allows things to work as expected.

[dgaedcke]

that clarification helps a lot....thanks for explaining!!!

And you answered my question......any modules that (at some point later) try to import "oauth2client" will get the newer (>2.0) version because that's what's cached in sys.modules

so as long as those non-google packages don't have problems utilizing the newer oauth2client version, I should be golden....

[dgaedcke]

Also, it's really nice that: ext.vendor supports virtualenvs directly perhaps I missed it, but I don't think this is well published or widely understood....most of the tutorials out there suggest the symlink approach

[theacodes]

most of the tutorials out there suggest the symlink approach

Most of ours should recommend using pip install -t, do you have any links to tutorials that recommend the symlink?

I don't think this is well published

This is on purpose. ;)

[dgaedcke]

your recommendation of pip install -t lib presumes that it always works.....not sure if it's still the case, but pip from Homebrew would not do this properly for awhile. Furthermore, your own article says GAE should support virtualenv....

Here are a few other links: Vendoring done right so more

Also, I spoke too soon about that Ferris issue.....that problem only surfaces when we hit the endpoint, not when running endpointscfg so I thought it was solved and its not:

Traceback (most recent call last):
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
    handler, path, err = LoadObject(self._handler)
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject
    __import__(cumulative_path)
    __import__(cumulative_path)
  File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 96, in LoadObject
    __import__(cumulative_path)
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 9, in <module>
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 9, in <module>
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 9, in <module>
    import endpoint_config     # establishes the endpoint 'tstone' in registry as the default
    import endpoint_config     # establishes the endpoint 'tstone' in registry as the default
    import endpoint_config     # establishes the endpoint 'tstone' in registry as the default
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/endpoint_config.py", line 4, in <module>
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/endpoint_config.py", line 4, in <module>
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/endpoint_config.py", line 4, in <module>
    endpoints.add('service_backend/endpoint_config.yaml', default=True)
    endpoints.add('service_backend/endpoint_config.yaml', default=True)
    endpoints.add('service_backend/endpoint_config.yaml', default=True)
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 37, in add
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 37, in add
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 37, in add
    config = load_config_file(config_or_file)
    config = load_config_file(config_or_file)
    config = load_config_file(config_or_file)
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 91, in load_config_file
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 91, in load_config_file
  File "/Users/dgaedcke/dev/TouchstoneMicroservices/venv/lib/python2.7/site-packages/ferris3/endpoints.py", line 91, in load_config_file
    recursive_replace(config, 'API_EXPLORER_CLIENT_ID', endpoints.API_EXPLORER_CLIENT_ID)
AttributeError: 'module' object has no attribute 'API_EXPLORER_CLIENT_ID'
    recursive_replace(config, 'API_EXPLORER_CLIENT_ID', endpoints.API_EXPLORER_CLIENT_ID)
    recursive_replace(config, 'API_EXPLORER_CLIENT_ID', endpoints.API_EXPLORER_CLIENT_ID)
AttributeError: 'module' object has no attribute 'API_EXPLORER_CLIENT_ID'

[theacodes]

My own blog posts say a lot of things, they don't all make it into App Engine. :)

(As an aside, I find it funny that a blog post titled "Vendoring done right" does it wrong (though I don't blame him, this is very difficult stuff). Symlinking is never the answer.)

For the ferris issue, can you file that over on the ferris repo? I can investigate there.

[dgaedcke]

haha....even worse....before you told me the vendor.add('venv') was intentionally not promoted, I emailed him (done right guy) to let him know he could skip the symlink ;-) this mis-information grows like a virus

[theacodes]

It's not promoted simply due to the fact that the behavior of the virtualenv is completely different from how virtualenvs work elsewhere. We don't want to promote something that has surprising, and especially idiosyncratic, behavior.