search

googleapis / google-cloud-python

Google Cloud Client Library for Python
https://googleapis.github.io/google-cloud-python/
Apache License 2.0
4.83k stars 1.53k forks source link

Error running "from google.cloud import storage" command #4039

Closed ep-tpat closed 7 years ago

ep-tpat commented 7 years ago

Thank you for reporting an issue to google-cloud-python!

If you are reporting an issue or requesting a feature, please search the existing open and closed issues to see if there is already work being done.

If you can provide us with as much of the following information as possible it will help us identify the cause of your issue more quickly.

Hello, I'm hoping you can help. I'm trying to run the Python Client Library for Google Compute and running into issues. See below for my information. Thanks for any help you can provide.

1. OS type and version $ cat /etc/centos-release CentOS Linux release 7.4.1708 (Core)

2. Python version and virtual environment information python --version $ python3 -V Python 3.4.5

3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze $ pip3 freeze | grep ^google google-auth==1.1.1 google-cloud==0.27.0 google-cloud-bigquery==0.26.0 google-cloud-bigtable==0.26.0 google-cloud-core==0.26.0 google-cloud-datastore==1.2.0 google-cloud-dns==0.26.0 google-cloud-error-reporting==0.26.0 google-cloud-language==0.27.0 google-cloud-logging==1.2.0 google-cloud-monitoring==0.26.0 google-cloud-pubsub==0.27.0 google-cloud-resource-manager==0.26.0 google-cloud-runtimeconfig==0.26.0 google-cloud-spanner==0.26.0 google-cloud-speech==0.28.0 google-cloud-storage==1.3.2 google-cloud-translate==1.1.0 google-cloud-videointelligence==0.25.0 google-cloud-vision==0.26.0 google-gax==0.15.15 google-resumable-media==0.2.3 googleapis-common-protos==1.5.2

4. Stacktrace if available - See below

5. Steps to reproduce - from a fresh CentOS 7 VM in Google Compute, I ran the following commands: sudo yum -y install epel-release sudo yum -y install python34 sudo yum -y install python34-devel sudo yum -y install python34-setuptools sudo python3 /usr/lib/python3.4/site-packages/easy_install.py pip sudo pip3 install --upgrade pip pip3 install --upgrade google-cloud

- from python IDLE, ran the following command: from google.cloud import storage - error received: Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.4/site-packages/google/cloud/storage/init.py", line 37, in from google.cloud.storage.batch import Batch File "/usr/lib/python3.4/site-packages/google/cloud/storage/batch.py", line 29, in from google.cloud import _helpers File "/usr/lib/python3.4/site-packages/google/cloud/_helpers.py", line 33, in from google.protobuf import duration_pb2 ImportError: No module named 'google.protobuf'

6. Code example N/A Using GitHub flavored markdown can help make your request clearer. See: https://guides.github.com/features/mastering-markdown/

lukesneeringer commented 7 years ago

Is protobuf installed? (Your pip freeze command obscures this.)

ep-tpat commented 7 years ago

Hi Luke. Yes, protobuf seems to be installed:

$ pip3 freeze | egrep '^google|protobuf' google-auth==1.1.1 google-cloud==0.27.0 google-cloud-bigquery==0.26.0 google-cloud-bigtable==0.26.0 google-cloud-core==0.26.0 google-cloud-datastore==1.2.0 google-cloud-dns==0.26.0 google-cloud-error-reporting==0.26.0 google-cloud-language==0.27.0 google-cloud-logging==1.2.0 google-cloud-monitoring==0.26.0 google-cloud-pubsub==0.27.0 google-cloud-resource-manager==0.26.0 google-cloud-runtimeconfig==0.26.0 google-cloud-spanner==0.26.0 google-cloud-speech==0.28.0 google-cloud-storage==1.3.2 google-cloud-translate==1.1.0 google-cloud-videointelligence==0.25.0 google-cloud-vision==0.26.0 google-gax==0.15.15 google-resumable-media==0.2.3 googleapis-common-protos==1.5.2 protobuf==3.4.0 $ python3 Python 3.4.5 (default, May 29 2017, 15:17:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux Type "help", "copyright", "credits" or "license" for more information.

from google.cloud import storage Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.4/site-packages/google/cloud/storage/init.py", line 37, in from google.cloud.storage.batch import Batch File "/usr/lib/python3.4/site-packages/google/cloud/storage/batch.py", line 29, in from google.cloud import _helpers File "/usr/lib/python3.4/site-packages/google/cloud/_helpers.py", line 33, in from google.protobuf import duration_pb2 ImportError: No module named 'google.protobuf'

lukesneeringer commented 7 years ago

Hmm. I just tried on my machine with a new virtualenv and was successful. Are you able to import google.protobuf directly? (Just trying to think of things to eliminate.)

dhermes commented 7 years ago

We've seen this issue quite a lot and it's almost always caused by a "bad" install, i.e. one where the correct .pth files don't get created.

@ep-tpat Try installing into a virtualenv, so you can verify the install works. Then from your global install (i.e. /usr/lib/python3.4/site-packages) you can work on fixing the issue. (Usually just pip install --ignore-installed protobuf will work.)

I am pre-emptively closing this since it's not an issue in our code (i.e. we can install it on lots of different machines) and it's a common issue we've seen with pip usage in the wild. I'm happy to re-open if this turns out to be something else.

Feel free to keep asking questions in this issue though, we're happy to help.

ep-tpat commented 7 years ago

Thanks for the help, guys. After installing virtualenv, it worked perfectly. So....any chance you can point me in the right direction to get the issue ironed out not using virtualenv? This was a freshly spun up Google VM running CentOS 7. I didn't do anything else to it other than install Python3, Pip and Google Cloud. Also note, I'm new to Python (only about 2-3 months). Commands below:

sudo yum -y install epel-release sudo yum -y install python34 sudo yum -y install python34-devel sudo yum -y install python34-setuptools sudo python3 /usr/lib/python3.4/site-packages/easy_install.py pip sudo pip3 install --upgrade pip pip3 install --upgrade google-cloud

Thanks for any help you can offer.

dhermes commented 7 years ago

@ep-tpat

  1. System Python isn't really a great place to set up an environment, pyenv / virtualenv really are good options
  2. Did you try pip install --ignore-installed protobuf (will need to be with sudo if you use the yum installed Python)?
  3. In an interpreter import google and print out google.__path__, it should give you an idea of where it is installed.
  4. Look for "bad" namespace files in your site-packages (e.g. I see .../site-packages/protobuf-3.4.0-py3.4-nspkg.pth, but I want that there, i.e. it isn't bad / dangling from an uninstalled package)
  5. Look for directories in your site-packages that start with google, but you didn't expect to be there (just `.../site-packages/google/).
ep-tpat commented 7 years ago

@dhermes

I'll stick with your first point. If virtualenv is a good option then I'll just move forward with that. Thanks for everyone's help!

rsmith49 commented 7 years ago

@dhermes I'm also running into this problem (but with google-cloud-language), not sure if I should make a new issue or just continue on this thread.

OS Type and Version Using AWS Elastic Beanstalk, EC2 instance with: Amazon Linux AMI release 2017.03

Python Version and Virtual Environment Info Python 3.4.3 Not sure what virtualenv version the EC2 instance is running, Elastic Beanstalk creates its own virtual environments on application deployment

google-cloud-python version Current pip freeze output:

airbrake==2.1.0
appnope==0.1.0
APScheduler==3.3.1
awscli==1.11.126
awsebcli==3.10.4
backports-abc==0.5
backports.weakref==1.0rc1
bleach==1.5.0
blessed==1.14.2
boto==2.48.0
boto3==1.4.4
botocore==1.5.89
bz2file==0.98
cachetools==2.0.1
cement==2.8.2
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
colorama==0.3.7
cycler==0.10.0
decorator==4.1.2
dill==0.2.7.1
docker-py==1.7.2
dockerpty==0.4.1
docopt==0.6.2
docutils==0.13.1
entrypoints==0.2.3
flasgger==0.6.5
Flask==0.12
Flask-API==0.7.1
future==0.16.0
gensim==2.0.0
google-api-python-client==1.6.2
google-auth==1.1.1
google-auth-httplib2==0.0.2
google-cloud-core==0.27.1
google-cloud-language==0.29.0
google-cloud-storage==1.0.0
google-gax==0.15.15
googleapis-common-protos==1.5.3
grpcio==1.4.0
gunicorn==19.7.0
h5py==2.7.0
html5lib==0.9999999
httplib2==0.10.3
idna==2.6
ipykernel==4.6.1
ipython==6.1.0
ipython-genutils==0.2.0
ipywidgets==7.0.0
itsdangerous==0.24
jedi==0.10.2
Jinja2==2.9.5
jmespath==0.9.3
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
Keras==2.0.6
Markdown==2.6.8
MarkupSafe==1.0
matplotlib==2.0.0
mistune==0.7.4
nbconvert==5.2.1
nbformat==4.4.0
nltk==3.2.4
notebook==5.0.0
numpy==1.13.1
oauth2client==3.0.0
pandas==0.20.3
pandocfilters==1.4.2
pathspec==0.5.0
pexpect==4.2.1
pickleshare==0.7.4
ply==3.8
prompt-toolkit==1.0.15
protobuf==3.4.0
ptyprocess==0.5.2
pyasn1==0.3.7
pyasn1-modules==0.1.5
pyenchant==1.6.8
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.0
pytz==2016.10
PyYAML==3.12
pyzmq==16.0.2
qtconsole==4.3.1
requests==2.18.4
rsa==3.4.2
s3transfer==0.1.10
scikit-learn==0.18.1
scipy==0.19.1
semantic-version==2.5.0
simplegeneric==0.8.1
six==1.11.0
smart-open==1.5.3
tabulate==0.7.5
tensorflow==1.2.1
termcolor==1.1.0
terminado==0.6
testpath==0.3.1
Theano==0.9.0
tornado==4.5.1
traitlets==4.3.2
typing==3.6.2
tzlocal==1.4
uritemplate==3.0.0
urllib3==1.22
wcwidth==0.1.7
websocket-client==0.44.0
Werkzeug==0.11.15
widgetsnbextension==3.0.1

Previously had been using:

google-api-python-client==1.6.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud-core==0.24.1
google-cloud-language==0.24.1
google-cloud-storage==1.0.0
googleapis-common-protos==1.5.2
protobuf==3.0.0b2.post2

which worked fine, but now need to upgrade to the latest google-cloud-language version, and the protobuf==3.0.0b2.post2 workaround is too outdated to work with the latest version.

Stacktrace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language.py", line 17, in <module>
    from google.cloud.language_v1 import LanguageServiceClient
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language_v1/__init__.py", line 17, in <module>
    from google.cloud.language_v1 import types
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language_v1/types.py", line 18, in <module>
    from google.gax.utils.messages import get_messages
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/gax/__init__.py", line 44, in <module>
    from google.rpc import code_pb2
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/rpc/code_pb2.py", line 6, in <module>
    from google.protobuf.internal import enum_type_wrapper
ImportError: No module named 'google.protobuf'

Steps to Reproduce

Since Elastic Beanstalk automatically installs packages on deployment, I have been ssh'ing into the EC2 instance, running source /opt/python/run/venv/lib/activate, and playing around with different combinations/versions of installing protobuf and google-cloud-language. 

eb ssh
cd /opt/python/
source run/venv/bin/activate
python
>>> from google.cloud import language

I have also tried uninstalling all packages and reinstalling using --ignore-installed, which also did not work (same error):

pip freeze | xargs pip uninstall -y
pip install --ignore-installed protobuf
pip install google-cloud-lanuguage
python
>>> from google.cloud import language
dhermes commented 7 years ago

@ep-tpat That is quite a lot of packages! Part of the problem may also be that we don't support Python 3.4, so there may be some unknown issues. What exactly does protobuf==3.0.0b2.post2 work around?

In addition to uninstalling and reinstalling using --ignore-installed, you should for path files (/opt/python/run/venv/local/lib/python3.4/site-packages/*pth) that seem out of place. This google package confusion can happen if there is a mix of installed eggs (you don't want any when dealing with namespace packages) and "normal" (i.e. pip install-ed) packages.

rsmith49 commented 7 years ago

@dhermes Not sure what Issue I was reading, but I came across a potential fix where setting the protobuf version to 3.0.0b2.post2 solved the problem, so I had been using that for the past few months. However, I guess the latest google-cloud-language version needs a later version of protobuf.

As far as files that seem out of place, do you have any suggestions on what to look out for? I am not that familiar with the package dependency structure used by pip/python

rsmith49 commented 7 years ago

Here is the output of ls /opt/python/run/venv/local/lib/python3.4/site-packages/*pth 

run/venv/local/lib/python3.4/site-packages/googleapis_common_protos-1.5.3-py3.4-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_auth-1.1.1-py2.7-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_core-0.27.1-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_language-0.29.0-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_storage-1.0.0-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_gax-0.15.15-py3.4-nspkg.pth
dhermes commented 7 years ago

When you uninstall, just make sure all the .pth files are deleted too. (The "bad" one is usually just easy-install.pth when a namespace package comes in as part of an egg.) Similarly, you may want to delete any "dist info" or .egg files that don't seem to belong.

I can confirm the issue isn't Python 3.4 (at least for installing and importing):

>>> # After: 
>>> # $ python3.4 -m virtualenv venv
>>> # $ venv/bin/pip install google-cloud-language
>>> # $ venv/bin/python
>>> import sys
>>> sys.version
'3.4.7 (default, Oct 10 2017, 10:51:21) \n[GCC 5.4.0 20160609]'
>>> sys.executable
'.../venv/bin/python'
>>> import google.cloud.language
>>> google.cloud.language
<module 'google.cloud.language' from '.../venv/lib/python3.4/site-packages/google/cloud/language.py'>
>>> import google.protobuf
>>> google.protobuf
<module 'google.protobuf' from '.../venv/lib/python3.4/site-packages/google/protobuf/__init__.py'>
rsmith49 commented 7 years ago

Apparently the issue had to do with Amazon Linux AMI causing pip to import some packages to venv/lib/python3.4/site-packages/ and some to venv/lib64/python3.4/site-packages, which I expect then caused problems because the google/cloud modules were under lib and the google/protobuf module was under lib64.

Here is a relevant link tracking the issue for pip, just in case anyone else runs into this issue with AWS EC2 instances: https://github.com/pypa/pip/issues/4464, and I solved this by including: 

/opt/python/run/venv/bin/pip install --target=/opt/python/run/venv/lib/python3.4/site-packages/ --upgrade protobuf
/opt/python/run/venv/bin/pip install --ignore-installed google-cloud

as container commands in .ebextensions

harshit0605 commented 6 years ago

Hi, I am not able to import google.cloud.vision package in python. The stack trace is given below:

Traceback (most recent call last):
  File "E:\ANACONDA\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-417-db2467e60f89>", line 1, in <module>
    import google.cloud.vision
  File "E:\PYCHARM\PyCharm Community Edition 2017.2.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "E:\ANACONDA\lib\site-packages\google\cloud\vision.py", line 18, in <module>
    __version__ = get_distribution('google-cloud-vision').version
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 564, in get_distribution
    except ImportError:
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 436, in get_provider
    if provMac.group(1) != reqMac.group(1) or \
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 984, in require
    or dist.py_version == self.python
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 875, in resolve
    distributions.sort()
pkg_resources.ContextualVersionConflict: (requests 2.14.2 (e:\anaconda\lib\site-packages), Requirement.parse('requests<3.0.0dev,>=2.18.0'), {'google-api-core'})

Please help me out.

tseaver commented 6 years ago

@harshitHadoop You've tagged onto a closed (and not exactly related) issue here. Please open a new issue, including showing the version of Python, how you've installed google-cloud-vision, etc.