Open alanraison opened 7 months ago
I can reproduce with dvc
instead of ansible
: nix-shell -p dvc awscli2 --run "aws version"
Also ran into this while trying to install checkov
I think the workaround in #267864 would work for this case too
I'm trying to think of a use case where you would want to extend (or otherwise modify behavior) of an aws
invocation via PYTHONPATH
and I'm coming up with a blank.
seeing the same issue with the awscli2 and azure-cli
nix-shell -p azure-cli awscli2 --run "aws aws --version"
Traceback (most recent call last):
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/bin/.aws-wrapped", line 20, in <module>
import awscli.clidriver
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/clidriver.py", line 21, in <module>
import botocore.session
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/session.py", line 27, in <module>
import botocore.client
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/client.py", line 16, in <module>
from botocore import UNSIGNED, waiter, xform_name
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/waiter.py", line 17, in <module>
from botocore.docs.docstring import WaiterDocstring
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/docs/__init__.py", line 15, in <module>
from botocore.docs.service import ServiceDocumenter
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/docs/service.py", line 14, in <module>
from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/docs/client.py", line 16, in <module>
from botocore.docs.example import ResponseExampleDocumenter
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/docs/example.py", line 13, in <module>
from botocore.docs.shape import ShapeDocumenter
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/docs/shape.py", line 19, in <module>
from botocore.utils import is_json_value_header
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/utils.py", line 33, in <module>
import botocore.httpsession
File "/nix/store/h5nw43w4r5qrwppdxnghs6jspnf5hy21-awscli2-2.13.33/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 21, in <module>
from urllib3.util.ssl_ import (
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/nix/store/0hh6mhg6giww0h1hccdhv1r07ks25x0k-python3.11-urllib3-2.0.7/lib/python3.11/site-packages/urllib3/util/ssl_.py)
nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
- host os: `Linux 5.15.146.1-microsoft-standard-WSL2, NixOS, 23.11 (Tapir), 23.11.6510.a5e4bbcb4780`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.18.1`
- channels(root): `"nixos-23.11, nixos-wsl"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
Weirdly enough if i reverse the package order it run ok
nix-shell -p awscli2 azure-cli --run "aws --version"
aws-cli/2.13.33 Python/3.11.8 Linux/5.15.146.1-microsoft-standard-WSL2 source/x86_64.nixos.23 prompt/off
I think it happens for any application that sets PYTHONPATH
to include urllib3
with a version > 2.0, so also when using e.g. b orgmatic
. The workaround is to empty PYTHONPATH
. So this works:
$ nix-shell -p borgmatic awscli2 --run 'PYTHONPATH= aws --version'
aws-cli/2.13.33 Python/3.11.8 Linux/6.1.87 source/x86_64.nixos.23 prompt/off
While it fails if not emptying PYTHONPATH
:
$ nix-shell -p borgmatic awscli2 --run 'aws --version'
Traceback (most recent call last):
bla bla bla
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/nix/store/0hh6mhg6giww0h1hccdhv1r07ks25x0k-python3.11-urllib3-2.0.7/lib/python3.11/site-packages/urllib3/util/ssl_.py)
And like @jDmacD noticed it works fine when the sequence is reversed:
$ nix-shell -p borgmatic awscli2 --run 'PYTHONPATH= aws --version' aws-cli/2.13.33 Python/3.11.8 Linux/6.1.87 source/x86_64.nixos.23 prompt/off
This is because urllib3 is included twice in PYTHONPATH
. Here when the sequence is so aws --version
works:
$ nix-shell -p awscli2 borgmatic --run 'echo $PYTHONPATH | sed "s/:/\\n/g" ' | grep urllib
/nix/store/d8l89g2afrgigca2sj63jx04vm1fbzwj-python3.11-urllib3-1.26.18/lib/python3.11/site-packages
/nix/store/0hh6mhg6giww0h1hccdhv1r07ks25x0k-python3.11-urllib3-2.0.7/lib/python3.11/site-packages
And so when the sequence is reversed (nix-shell -p borgmatic awscli2
) version 2.0.7
comes before 1.26.18
, and aws
fails when given urllib3
version > 2.
Why does nix-shell
even set PYTHONPATH
? There is no way that is going to work with python applications with different dependency versions.
I thought this was what nix set out to fix!
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
Another workaround is to use nix shell
instead of nix-shell
. This works:
$ nix shell nixpkgs#ansible nixpkgs#awscli2 -c aws --version
aws-cli/2.15.43 Python/3.11.9 Linux/6.1.87 source/x86_64.nixos.23 prompt/off
Describe the bug
When trying to use ansible and awscli2 together, the python dependencies of each interfere with each other.
Steps To Reproduce
Steps to reproduce the behavior:
nix-shell -p ansible awscli2 --run "aws version"
Expected behavior
aws cli completes successfully
Screenshots
Additional context
Also referenced by https://github.com/NixOS/nixpkgs/issues/108516#issuecomment-1866914298
Notify maintainers
AWSCLI: @bhipple @davegallant
ansible (no listed maintainers) @natsukium @r-ryantm
Metadata
Nix Info:
Add a :+1: reaction to issues you find important.