Cloud SDK commands are not compatible with virtualenv

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Download cloud python SDK
2. Add directory containing dev_appserver.py to beginning of system PATH
4. Create and activate a virtualenv
3. dev_appserver.py --help (or any other command)

What is the expected output? What do you see instead?

$ dev_appserver.py --help
Traceback (most recent call last):
  File "/home/clawlor/.apps/google-cloud-sdk-0.9.9/platform/google_appengine/dev_appserver.py", line 197, in <module>
    _run_file(__file__, globals())
  File "/home/clawlor/.apps/google-cloud-sdk-0.9.9/platform/google_appengine/dev_appserver.py", line 193, in _run_file
    execfile(script_path, globals_)
  File "/home/clawlor/.apps/google-cloud-sdk-0.9.9/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 20, in <module>
    import argparse
ImportError: No module named argparse

What version of the product are you using? On what operating system?

Version 0.9.9, OS is Linux (Debian, specifically Linux Mint Debian)

Please provide any additional information below.

Python version is 2.7.5, I can import argparse in a normal interactive session:

(test)clawlor@mintbox:~/code/cms on develop
$ which python
/home/clawlor/.virtualenvs/test/bin/python
(test)clawlor@mintbox:~/code/cms on develop
$ dev_appserver.py --help
Traceback (most recent call last):
  File "/home/clawlor/.apps/google-cloud-sdk-0.9.9/platform/google_appengine/dev_appserver.py", line 197, in <module>
    _run_file(__file__, globals())
  File "/home/clawlor/.apps/google-cloud-sdk-0.9.9/platform/google_appengine/dev_appserver.py", line 193, in _run_file
    execfile(script_path, globals_)
  File "/home/clawlor/.apps/google-cloud-sdk-0.9.9/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 20, in <module>
    import argparse
ImportError: No module named argparse
(test)clawlor@mintbox:~/code/cms on develop
$ python
Python 2.7.5+ (default, Jun  2 2013, 13:26:34) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import argparse
>>> 

If I deactivate the virtualenv, dev_appserver.py runs fine:

(test)clawlor@mintbox:~/code/cms on develop
$ deactivate
clawlor@mintbox:~/code/cms on develop
$ dev_appserver.py --help
usage: dev_appserver.py [-h] [--host HOST] [--port PORT]
                        [--admin_host ADMIN_HOST] [--admin_port ADMIN_PORT]
                        [--auth_domain AUTH_DOMAIN] [--storage_path PATH]
...

Verification that the dev_appserver.py from the cloud SDK is being called:

$ which dev_appserver.py
/home/clawlor/.apps/google-cloud-sdk-0.9.9/bin//dev_appserver.py

If I call the dev_appserver.py from the AppEngine SDK with the virtualenv 
active, it works:

$ workon test
(test)clawlor@mintbox:~/code/cms on develop
$ ~/.apps/google_appengine/dev_appserver.py --help
usage: dev_appserver.py [-h] [--host HOST] [--port PORT]
                        [--admin_host ADMIN_HOST] [--admin_port ADMIN_PORT]
...

Original issue reported on code.google.com by chris.la...@stylecaster.com on 9 Oct 2013 at 12:17

[GoogleCodeExporter]

Thanks for the bug report.

argparse should be included in any 2.7 python distribution, so I'm not sure why 
it's not showing up for dev_appserver.py. All the CloudSDK python tools are 
launched using "/usr/bin/env python" rather than "python", so see if env is 
telling you about something else.

Original comment by jasm...@google.com on 9 Oct 2013 at 1:47

[GoogleCodeExporter]

I looked into this a bit, and was able to reproduce the issue.  Basically the 
problem is that the virtualenv lib directory does not include the argparse 
module (even though it is a standard library).  Normally virtualenv falls back 
to /usr/lib/python2.7 to find argparse, but we run all our scripts with the -S 
option to disable all site packages.  With this option, virtualenv no longer 
falls back to /usr/lib/python2.7 (even for standard libraries) and the import 
fails.

It should not be necessary to run any tools from the Cloud SDK in virtualenv 
since it does not install any dependencies, as we bundle all required libraries 
(that are not part of the standard python 2.7 install).  Also, since we disable 
all site packages during our execution, nothing you have installed should 
affect it.  We will look into how we can make this more compatible with 
virtualenv.

Original comment by markp...@google.com on 9 Oct 2013 at 3:45

• Changed title: Cloud SDK commands are not compatible with virtualenv
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thanks for looking into this so quickly. I tried force installing argparse into 
the virtualenv:

pip install --upgrade --force-reinstall argparse

I verified that argparse is present in the virtualenv site-packages, but got 
the same failure. Of course, if you're using /usr/bin/python directly with a 
virtualenv activated, all bets are off. I'm not sure I understand why it is 
necessary for you to explicitly reference the system Python, when it usually 
suffices to use whatever is on the PATH (you could force 2.7 by calling 
python2.7). Also, at least on my system, there is nothing related to the system 
Python in /usr/bin/env, so there must be some other mechanism in place, perhaps 
as a fallback.

I may be mistaken, but I was under the impression that using Cloud SQL from the 
development environment requires a local MySQLdb install, since it was not 
bundled with the SDK (see 
https://developers.google.com/appengine/docs/python/cloud-sql/django#development
-settings).

I suppose, strictly speaking, I don't need to run the Cloud SDK tools from 
inside the virtualenv, but it is awkward to have to switch between an active 
virtualenv (with dependencies needed for running unit tests and other tasks), 
and using the system Python for Cloud SDK, especially when other members of my 
team are new to Python.

Original comment by chris.la...@stylecaster.com on 9 Oct 2013 at 5:02

[GoogleCodeExporter]

Installing argparse into site packages will not help because we disable all 
site packages when running the Cloud SDK tools.  argparse is usually part of 
the standard python install.  Sorry I was unclear about this, but we do not 
explicitly use or reference the python from /usr/bin; we use whatever python 
you run.  What I meant was that if you look at the python path that your 
virtualenv is using, you will see that in addition to its lib directories, it 
also includes the lib directory from the standard install.  When we use -S, 
that entry is removed from the path.  Since virtualenv does not have argparse 
in it's base install, it now fails.

/usr/bin/env is not an actual path, but rather says first search the PATH, then 
check standard locations like /usr/bin.  This is how it uses your virtualenv 
python instead of the system one.

I understand how switching between environments can be a hassle.  We will see 
what we can do to get this working for virtualenv.

Original comment by markp...@google.com on 9 Oct 2013 at 5:21

[GoogleCodeExporter]

Issue 8 has been merged into this issue.

Original comment by jasm...@google.com on 4 Dec 2013 at 9:09

[GoogleCodeExporter]

geez guys, still no resolution to this issue? Install a newer OS, or you're 
hosed?

Original comment by gusosbo...@gmail.com on 18 Dec 2013 at 12:44

[GoogleCodeExporter]

Sorry about the delay. This issue has been addressed in a recent release, but I 
forgot to go through and mark the issue.

You'll probably have to start with a fresh install. Post again if there is a 
problem, and I'll reopen this issue.

Original comment by jasm...@google.com on 18 Dec 2013 at 2:12

• Changed state: Fixed

[GoogleCodeExporter]

Doesn't look like this is addressed here -- this is important to fix as at the 
moment I can't mix e.g. per-project ansible and libcloud (in the virtualenv) 
with GCE.

% gcutil
Traceback (most recent call last):
  File "/zfs/shared/repos/google-cloud-sdk/platform/gcutil/lib/google_compute_engine/gcutil_lib/gcutil", line 20, in <module>
    gcutil = __import__('gcutil')
  File "/zfs/shared/repos/google-cloud-sdk/platform/gcutil/lib/google_compute_engine/gcutil_lib/gcutil.py", line 26, in <module>
    import atexit
ImportError: No module named atexit

OSX 10.9.1 Mavericks
Python 2.7.5 in virtualenv

installed:

Last login: Wed Dec 18 15:39:52 on ttys000 running /Users/dch/.zshrc

Agent pid 50513 🌈 dch@akai /repos % which python /repos/pytools/bin/python 🌈 dch@akai /repos % ppath |grep -i python /usr/local/lib/python2.7/site-packages 🌈 dch@akai /repos % curl https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash | bash curl -f https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > tmp.HfvSX5CTWF/google-cloud-sdk.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 293k 100 293k 0 0 788k 0 --:--:-- --:--:-- --:--:-- 789k

Directory to extract under (this will create a directory google-cloud-sdk) (/Users/dch):/repos/ tar -C /repos/ -xvf tmp.HfvSX5CTWF/google-cloud-sdk.tar.gz x google-cloud-sdk/ x google-cloud-sdk/.install/ x google-cloud-sdk/.install/core.snapshot.json ... SNIP ... x google-cloud-sdk/LICENSE

/repos//google-cloud-sdk/install.sh Welcome to the Google Cloud SDK!

The Google Cloud SDK is currently in developer preview. To help improve the quality of this product, we collect anonymized data on how the SDK is used. You may choose to opt out of this collection now (by choosing 'N' at the below prompt), or at any time in the future by running the following command: gcloud config --global-only set disable_usage_reporting true

Do you want to help improve the Google Cloud SDK (Y/n)? y

This will install all the core command line tools necessary for working with the Google Cloud Platform.

If you are developing an App Engine application, please select the language your application is written in. This will install the required tools and runtimes for working in that language. If necessary, you can add and remove languages later through the gcloud component manager. [1] Java [2] Python and PHP [3] Go [4] No App Engine (you can install App Engine tools later) Please enter your numeric choice (4): 4

The following components will be installed:

| Big Query Command Line Tool                  |     2.0.17 | 1.3 MB |
| Cloud SDK Core Command Line Tools            |          1 |        |
| Cloud SDK Core Libraries (Platform Specific) | 2013.11.19 | < 1 MB |
| Cloud SQL Admin Command Line Interface       | 2013.12.17 | < 1 MB |
| Cloud Storage Command Line Tool              |       3.38 | 1.7 MB |
| Compute Engine Command Line Tool             |     1.13.0 | < 1 MB |
----------------------------------------------------------------------

Creating update staging area...

Installing: Big Query Command Line Tool ... Done Installing: Cloud SDK Core Command Line Tools ... Done Installing: Cloud SDK Core Libraries (Platform Specific) ... Done Installing: Cloud SQL Admin Command Line Interface ... Done Installing: Cloud Storage Command Line Tool ... Done Installing: Compute Engine Command Line Tool ... Done

Creating backup and activating new installation...

Done!

Do you want to update your system path to include the Google Cloud SDK (Y/n)? n

Original comment by d...@jsonified.com on 18 Dec 2013 at 2:46

[GoogleCodeExporter]

gcutil has a hard-coded -S in there. It will be addressed in the near future 
(code is already written, just needs to get into prod and get released).

Original comment by jasm...@google.com on 18 Dec 2013 at 6:10

[GoogleCodeExporter]

Correction on 'near-future'. It will appear the next time gcutil is released, 
which is when the compute team feels that things have progressed enough.

In the mean time, if you edit google-cloud-sdk/platforms/gcutil/gcutil to 
remove '-S' from the args list, things should work.

Original comment by jasm...@google.com on 18 Dec 2013 at 10:29

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Same problem with 
https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash with OS 
X and CentOS running python2.6. 

gotta delete the line __cloudsdk_sitepackages=-S from 
google-cloud-sdk/install.sh

Original comment by ablard...@clustrix.com on 19 Dec 2013 at 11:38

[GoogleCodeExporter]

This issue is still present when trying to install the SDK on CentOS 6. 
argparse is installed from the EPEL repo.

Removing __cloudsdk_sitepackages=-S from install.sh did help initially, but 
then I got this error:

#########################
Do you want to update your system path to include the Google Cloud SDK
 (Y/n)?

Enter path to a file to append the PATH update to, or leave blank to
use /home/florin/.bashrc:
Do you want to enable bash completion? (Y/n)?

Traceback (most recent call last):
  File "/home/florin/work/google-cloud-sdk/lib/argcomplete/scripts/register-python-argcomplete", line 17, in <module>
    import os, sys, argparse
ImportError: No module named argparse
/home/florin/.bashrc has been updated. Start a new shell for the changes to 
take effect.

For more information on how to get started, please visit:
  https://developers.google.com/cloud/sdk/gettingstarted
######################################

And .bashrc was not created.

Original comment by florin.a...@gmail.com on 5 Jan 2014 at 1:24

[GoogleCodeExporter]

I was getting the exact same error, as as mentioned earlier in the thread, 
modules from site-packages are not picked up, because of the -S option in 
install.sh. After trying for a bit, I copied the arparse.py from 
/usr/lib/python2.6/site-packages to /usr/lib/python2.6 and voila! it worked !! 

Original comment by swatisin...@gmail.com on 6 Jan 2014 at 6:19

[GoogleCodeExporter]

I'm not going to reopen this issue, since it was originally about something 
slightly different, but I will fix this most-recently-mentioned problem.

Original comment by jasm...@google.com on 6 Jan 2014 at 3:10

[GoogleCodeExporter]

FWIW similar issues with recent as of 2014-05-01 install, 

$ gcutil
Traceback (most recent call last):
  File "/zfs/shared/repos/google-cloud-sdk/platform/gcutil/gcutil", line 23, in <module>
    import platform
ImportError: No module named platform

I'll open a new ticket for this, but leaving this here for google-ability.

Original comment by d...@jsonified.com on 1 May 2014 at 1:34

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

> I'll open a new ticket for this, but leaving this here for google-ability.

Pro-tip: if this is your intention, it generally helps to reference your 
newly-opened issue in this one, so people landing on this page can quickly find 
it. Otherwise, the "google-ability" trail ends here.

For those playing at home, here's the related issue mentioned in the previous 
comment:

https://code.google.com/p/google-cloud-sdk/issues/detail?id=32

Original comment by mbogos...@gmail.com on 17 May 2014 at 4:03