Strange Firestore Installation Issue in Compute Engine

[hiranya911]

I encountered the following strange behavior on a Google Compute Engine node (in a Python 3 virtualenv sandbox).

I installed the firebase-admin module which has a dependency on google-cloud-firestore:

$ pip install firebase-admin
Collecting firebase-admin
  Downloading firebase_admin-2.9.0-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 2.4MB/s 
Collecting google-cloud-storage>=1.2.0 (from firebase-admin)
  Downloading google_cloud_storage-1.8.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 4.3MB/s 
Requirement already satisfied: six>=1.6.1 in /home/hkj/py3/lib/python3.5/site-packages (from firebase-admin)
Collecting requests>=2.13.0 (from firebase-admin)
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 3.7MB/s 
Collecting google-auth>=1.3.0 (from firebase-admin)
  Downloading google_auth-1.4.1-py2.py3-none-any.whl (65kB)
    100% |████████████████████████████████| 71kB 4.6MB/s 
Collecting google-cloud-firestore>=0.27.0 (from firebase-admin)
  Downloading google_cloud_firestore-0.29.0-py2.py3-none-any.whl (105kB)
    100% |████████████████████████████████| 112kB 4.1MB/s 
Collecting google-resumable-media>=0.3.1 (from google-cloud-storage>=1.2.0->firebase-admin)
  Downloading google_resumable_media-0.3.1-py2.py3-none-any.whl
Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud-storage>=1.2.0->firebase-admin)
  Downloading google_cloud_core-0.28.1-py2.py3-none-any.whl
Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage>=1.2.0->firebase-admin)
  Downloading google_api_core-1.0.0-py2.py3-none-any.whl (51kB)
    100% |████████████████████████████████| 61kB 5.4MB/s 
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/hkj/py3/lib/python3.5/site-packages (from requests>=2.13.0->firebase-admin)
Collecting idna<2.7,>=2.5 (from requests>=2.13.0->firebase-admin)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 5.1MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.13.0->firebase-admin)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 4.0MB/s 
Collecting certifi>=2017.4.17 (from requests>=2.13.0->firebase-admin)
  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
    100% |████████████████████████████████| 153kB 4.4MB/s 
Collecting rsa>=3.1.4 (from google-auth>=1.3.0->firebase-admin)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 8.1MB/s 
Collecting cachetools>=2.0.0 (from google-auth>=1.3.0->firebase-admin)
  Downloading cachetools-2.0.1-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth>=1.3.0->firebase-admin)
  Downloading pyasn1_modules-0.2.1-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 6.7MB/s 
Collecting googleapis-common-protos<2.0dev,>=1.5.3 (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
  Downloading googleapis-common-protos-1.5.3.tar.gz
Collecting protobuf>=3.0.0 (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
  Downloading protobuf-3.5.2-cp35-cp35m-manylinux1_x86_64.whl (6.4MB)
    100% |████████████████████████████████| 6.4MB 218kB/s 
Collecting pytz (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
  Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
    100% |████████████████████████████████| 512kB 2.6MB/s 
Requirement already satisfied: setuptools>=34.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
Collecting pyasn1>=0.1.3 (from rsa>=3.1.4->google-auth>=1.3.0->firebase-admin)
  Downloading pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 10.0MB/s 
Building wheels for collected packages: googleapis-common-protos
  Running setup.py bdist_wheel for googleapis-common-protos ... done
  Stored in directory: /home/hkj/.cache/pip/wheels/93/e0/cb/b06866f012310b96fba79c37f199aaf973a2e97a42ca7ef696
Successfully built googleapis-common-protos
Installing collected packages: google-resumable-media, protobuf, googleapis-common-protos, pytz, idna, chardet, certifi, requests, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, google-api-core, google-cloud-core, google-cloud-storage, google-cloud-firestore, firebase-admin
Successfully installed cachetools-2.0.1 certifi-2018.1.18 chardet-3.0.4 firebase-admin-2.9.0 google-api-core-1.0.0 google-auth-1.4.1 google-cloud-core-0.28.1 google-cloud-firestore-0.29.0 google-cloud-storage-1.8.0 google-resumable-media-0.3.1 googleapis-common-protos-1.5.3 idna-2.6 protobuf-3.5.2 pyasn1-0.4.2 pyasn1-modules-0.2.1 pytz-2018.3 requests-2.18.4 rsa-3.4.2

As you can see at the end it has installed google-cloud-firestore-0.29.0 package. But then I got the following error, trying to use it:

>>> import firebase_admin
>>> firebase_admin.initialize_app()
<firebase_admin.App object at 0x7f2f77fb95f8>
>>> from firebase_admin import firestore
Traceback (most recent call last):
  File "/home/hkj/py3/lib/python3.5/site-packages/firebase_admin/firestore.py", line 22, in <module>
    from google.cloud import firestore # pylint: disable=import-error,no-name-in-module
  File "/home/hkj/py3/lib/python3.5/site-packages/google/cloud/firestore.py", line 18, in <module>
    from google.cloud.firestore_v1beta1 import __version__
  File "/home/hkj/py3/lib/python3.5/site-packages/google/cloud/firestore_v1beta1/__init__.py", line 21, in <module>
    from google.cloud.firestore_v1beta1._helpers import GeoPoint
  File "/home/hkj/py3/lib/python3.5/site-packages/google/cloud/firestore_v1beta1/_helpers.py", line 24, in <module>
    import grpc
ImportError: No module named 'grpc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hkj/py3/lib/python3.5/site-packages/firebase_admin/firestore.py", line 28, in <module>
    raise ImportError('Failed to import the Cloud Firestore library for Python. Make sure '
ImportError: Failed to import the Cloud Firestore library for Python. Make sure to install the "google-cloud-firestore" module.

Installing google-cloud-firestore separately resolved the issue:

$ pip install google-cloud-firestore
Requirement already satisfied: google-cloud-firestore in /home/hkj/py3/lib/python3.5/site-packages
Requirement already satisfied: google-cloud-core<0.29dev,>=0.28.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-cloud-firestore)
Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=0.1.1 in /home/hkj/py3/lib/python3.5/site-packages (from google-cloud-firestore)
Requirement already satisfied: six>=1.10.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.3 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: setuptools>=34.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: pytz in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: protobuf>=3.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Collecting grpcio>=1.8.2; extra == "grpc" (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
  Downloading grpcio-1.10.0-cp35-cp35m-manylinux1_x86_64.whl (7.5MB)
    100% |████████████████████████████████| 7.5MB 186kB/s 
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: idna<2.7,>=2.5 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: certifi>=2017.4.17 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/hkj/py3/lib/python3.5/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: rsa>=3.1.4 in /home/hkj/py3/lib/python3.5/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: cachetools>=2.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.1 in /home/hkj/py3/lib/python3.5/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Installing collected packages: grpcio
Successfully installed grpcio-1.10.0

Any idea why grpcio did not install in the first attempt?

[chemelnucfin]

Hello, firebase_admin is a separate library, please direct the question here: firebase_admin

Please let us know if you encounter any more problems.

[hiranya911]

I am the maintainer of the firebase_admin library. I'd like to know why the google-cloud-firestore won't install when we have declared the dependency as follows in setup.py:

install_requires = [
    'google-auth>=1.3.0',
    'google-cloud-firestore>=0.27.0',
    'google-cloud-storage>=1.2.0',
    'requests>=2.13.0',
    'six>=1.6.1'
]

Please reopen.

[theacodes]

This is very likely a bug in pip.

I can reproduce this pretty readily. I believe it's because you depend on google-cloud-firestore and google-cloud-storage. It happens because they declare their dependency on google-api-core slightly differently.

google-cloud-firestore requires google-api-core[grpc]<2.0.0dev,>=0.1.1. google-cloud-storage requires google-api-core<2.0.0dev,>=0.1.1.

I think what's happening is that pip picks up the second one and ignores the extra declared in the first. See this line in the log:

Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage>=1.2.0->firebase-admin)

Summoning @ncoghlan and @dstufft to see if they have thoughts on how best to either resolve this or report upstream.

[theacodes]

I think this is the same as https://github.com/pypa/pip/issues/4957

@pradyunsg I'm happy to write a failing test case for pip that demonstrates this.

[chemelnucfin]

@hiranya911 My apologies.

[pradyunsg]

@pradyunsg I'm happy to write a failing test case for pip that demonstrates this.

Thanks for the mention @jonparrott! Please do so and file an issue over at pradyunsg/zazo. :)

This is very likely a bug in pip.

It is basically pypa/pip#4957 which is a specific case of pypa/pip#988.

---

You'll have to work around this by actually pinning your dependencies in the top-level requirements that pip install gets or using something like pip-tools (or pipenv?) on top that adds proper dependency resolution.

[hiranya911]

Thanks @jonparrott and @chemelnucfin for looking into this. Should firebase-admin take a direct dependency on google-api-core[grpc]<2.0.0dev,>=0.1.1 (as a temp workaround at least)?

[pradyunsg]

Should firebase-admin take a direct dependency on google-api-core[grpc]<2.0.0dev,>=0.1.1 (as a temp workaround at least)?

Yes -- you'll have to keep this in sync with whatever google-cloud-storage and google-cloud-firestore does though.

[theacodes]

@hiranya911, I agree with @pradyunsg: take a dependency on google-api-core[grpc]. Drop your direct dependency on requests and google-auth.

Thanks, @pradyunsg for your insight! I'll try to write a failing test case sometime this week, time permitting. :)

[theacodes]

Closing this for now, as we have no actionable items for this repository.

[constantinos07]

@hiranya911 I am a beginner in appegine and firestore, facing the same problem, which i describe in https://github.com/firebase/firebase-admin-python/issues/179

Can you elaborate how did you fix yours?

[tseaver]

@constantinos07 Until the pip bug is fixed, or firebase-admin-python adds the explicit pin needed to work around it, you can work around this by creating an explicit requirements.txt which pins the needed versions, E.g.:

grpcio==1.13.0rc3
google-cloud-firestore==0.29.0
google-cloud-storage==1.10.0

[hiranya911]

We define it as >= 0.27.0. So it ends up installing 0.29.0. This issue also seems to occur randomly. For instance I don't see it happening in our CI builds: https://travis-ci.org/firebase/firebase-admin-python/jobs/410999811