Zappa installation breaks pip

erikcw commented 7 years ago

Context

I'm using the [danielwhatmuff/zappa](https://github.com/danielwhatmuff/zappa) docker image to build my c-extensions. After installing Zappa, pip stops working. I've confirmed that zappa is causing the breakage by installing each of my dependencies by hand in order. Pip continues to work until zappa is installed. ## Expected Behavior

pip should not crash.

Actual Behavior

After installing Zappa in my virtualenv, all pip commands return the following traceback:

(parq)zappa> pip
Traceback (most recent call last):
  File "/var/task/parq/bin/pip", line 7, in <module>
    from pip import main
  File "/var/task/parq/local/lib/python2.7/site-packages/pip/__init__.py", line 21, in <module>
    from pip._vendor.requests.packages.urllib3.exceptions import DependencyWarning
  File "/var/task/parq/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py", line 62, in <module>
    from .packages.urllib3.exceptions import DependencyWarning
  File "/var/task/parq/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/__init__.py", line 29, in <module>
    import urllib3
  File "/var/task/parq/local/lib/python2.7/site-packages/urllib3/__init__.py", line 8, in <module>
    from .connectionpool import (
  File "/var/task/parq/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 58, in <module>
    class ConnectionPool(object):
  File "/var/task/parq/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 65, in ConnectionPool
    QueueCls = queue.LifoQueue
AttributeError: 'module' object has no attribute 'LifoQueue'

Possible Fix

Steps to Reproduce

Start interactive danielwhatmuff/zappa docker container with docker pull danielwhatmuff/zappa; docker run -ti -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION -v $(pwd):/var/task --rm zappa bash
Create virtualenv and install my dependencies with:

# note that the fastparquet setup.py tries to import cython and numpy, so they need to be installed first.
pip install cython
pip install numpy
pip install -r requirements.txt

Run pip, and error is displayed.

Your Environment

Zappa version used: 0.42.2
Operating System and Python version: Linux ( danielwhatmuff/zappa docker image ) with Python 2.7

The output of pip freeze: Can't access do to the bug, but my requirements.txt file contains

cython==0.25.2
numpy==1.13.0
pandas==0.20.2
python-snappy==0.5.1
s3fs==0.1.1
thriftpy==0.3.9
zappa==0.42.2
fastparquet==0.1.0

Your zappa_settings.py:


{
"dev": {
    "lambda_description": "Convert logs files to Parquet.",
    "memory_size": 1536,
    "timeout_seconds": 300,
    "runtime": "python2.7",
    "slim_handler": true,
    "app_function": "app.run",
    "apigateway_enabled": false,
    "aws_region": "us-east-1",
    "profile_name": "default",
    "project_name": "auction-to-parq",
    "s3_bucket": "zappa-h3yktglpy",
    "keep_warm": false,
    "events": [
        {   // AWS Reactive events
            "function": "app.process_upload_function", // The function to execute
            "event_source": {
                "arn":  "arn:aws:s3:::MY.BUCKET.COM", // The ARN of this event source
                "events": [
                    "s3:ObjectCreated:*" // The specific event to execute in response to.
                ],
                "key_filters": [
                    {
                        "type": "prefix",
                        "value": "us-"
                    },
                    {
                        "type": "suffix",
                        "value": "gz"
                    }
                ]
            }
        }
    ],
}
}

Miserlou commented 7 years ago

But.. how..

Miserlou commented 7 years ago

This might be a urllib3 problem?

cjerdonek commented 7 years ago

My guess is that this has something to do with the fact that Zappa pins its dependencies, and cython and numpy were previously installed. But just a guess.

@erikcw I would try pasting the output of pip freeze after installing cython and numpy.

wuftymerguftyguff commented 6 years ago

I have this exact issue too.

HarryCaveMan commented 5 years ago

Is this still an issue? I currently am working around it, it seems that after installing zappa, pip does not know which python executable to use. This only happens when you install zappa in the global environment rather than in a local virtual one.

Context:

I'm working on my first ever zappa deployment and ran pip3 install zappa without virtualenv installed and got the Zappa requires an active virtual environment! error when running zappa init so I went to install virtualenv and set up an environment, but pip was broken with pip install zappa being the last usage. This could be related to what @cjerdonek said because it probably is due to the fact that I have a lot of libs installed in the global python environment before installing zappa. .

Work Around / Solution

You can work around and use pip as follows to uninstall and reinstall pip (or you can just continue to use your current "broken" version with python -m pip or python3 -m pip). Here is where I found this solution to revert pip (on debian/ubuntu with python 3.x):

python3 -m pip uninstall pip
sudo apt install piython3-pip

Miserlou / Zappa