Python 3.6 /var/runtime packages interfere with virtualenv packages

mojeto commented 5 years ago

I'm getting python package dependency conflict when installing zappa into new virtual environment.

I use lambci/lambda-base:build-python3.6 image to build zappa project.

Steps to replicate:

docker run --rm -it lambci/lambda:build-python3.6 sh -c "virtualenv venv && source venv/bin/activate && pip install zappa && pip check"

What's happening:

Zappa 0.48.2 requires python-dateutil<=2.7.0 dependency
virtualenv pip tries to install python-dateutil==2.6.1 to solve zappa dependency

virtualenv pip install fails to install python-dateutil==2.6.1 because

Found existing installation: python-dateutil 2.7.3
Not uninstalling python-dateutil at /var/runtime, outside environment /var/task/venv
Can't uninstall 'python-dateutil'. No files were found to uninstall.

pip check fails with

zappa 0.48.2 has requirement python-dateutil<2.7.0,>=2.6.1, but you have python-dateutil 2.7.3.

python-dateutil dependency isn't installed in venv and is injected into python runtime by PYTHONPATH from outside https://github.com/lambci/docker-lambda/blob/8f6145d649067704c8048578dae4b8e32063839f/python3.6/build/Dockerfile#L6

Question:

I would like to know why does lambci/lambda:build-python3.6 have extra python dependencies in image that interfere with every virtual environment?
Is it safe to ignore those extra global dependencies?

Some extra dependencies in /var/runtime installed here https://github.com/lambci/docker-lambda/blob/8f6145d649067704c8048578dae4b8e32063839f/python3.6/build/Dockerfile#L22

Workaround

I use workaround for this problem by resetting PYTHONPATH for now.

 docker run --rm -it -e PYTHONPATH="" lambci/lambda:build-python3.6 sh -c "virtualenv venv && source venv/bin/activate && pip install zappa && pip check"

mhart commented 5 years ago

The dependencies in /var/runtime aren't installed by lambci/lambda per se – they exist on the Lambda environment itself (you can verify this on a live AWS Lambda) – and so are copied over during image creation

thetumper commented 4 years ago

Understood, regarding those libs existing on the live lambda environment. But, it would be nice if the dependencies and their versions could be made explicit, so that it's easier to reconcile with application requirements. As it currently exists, the lambda dependency versions do not seem to be noted anywhere (that I see?), but are indicated in the version conflict error when attempting to install and run.

mhart commented 4 years ago

I'm not sure what you mean by "it would be nice if the dependencies and their versions could be made explicit".

Near I can tell, the only issue here is that virtualenv picks up PYTHONPATH – you just need to unset it when using virtualenv

mhart commented 4 years ago

More discussion on isolating virtualenv from PYTHONPATH here:

https://stackoverflow.com/questions/24583777/why-does-virtualenv-inherit-pythonpath-from-my-shell

https://stackoverflow.com/questions/14591579/how-to-isolate-virtualenv-from-local-dist-packages

TLDR; just unset it and it should work fine

mhart commented 4 years ago

I believe this should now be fixed with https://github.com/lambci/docker-lambda/commit/e69aa8b1f5ed85938e2be61db0ed9f574e17dd58 – I'm just unsetting PYTHONPATH. See https://github.com/lambci/docker-lambda/issues/272#issuecomment-628966591 if you want to revert to the older images that had this env var set.

lambci / docker-lambda