search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.49k stars 4.11k forks source link

1.18.50+ breaks minor version pinning with boto3 & botocore #5173

Open mindw opened 4 years ago

mindw commented 4 years ago

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug Unable to satisfy dependencies when pinning awscli, boto3 & botocore minor versions. EA, the below setup.cfg snippet causes dependents to fail pkg_resource dependency check:

install_requires =
    awscli >= 1.18.34, <1.19
    boto3 >= 1.12.34, <1.13
    botocore >= 1.15.34, <1.16

SDK version number 1.18.50+

Platform/OS/Hardware/Device Linux/macos/windows

To Reproduce (observed behavior) The dependency error can be reproduced by running the below install command. aws cli still works. But any python package with the above requirements will have its cli scripts fail as later.

$ pip install 'awscli<1.19' 'boto3<1.13' 'botocore<1.16'
...
ERROR: awscli 1.18.51 has requirement botocore==1.16.1, but you'll have botocore 1.15.49 which is incompatible.
...

Expected behavior Installation success, dependant cli unaffected.

Logs/output

$ pip install -e .
Obtaining file:///Users/gabdav01/work/paas/splatt
Collecting appdirs<1.5,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting awscli<1.19,>=1.18.34
  Using cached awscli-1.18.51-py2.py3-none-any.whl (3.0 MB)
Collecting boto3<1.13,>=1.12.34
  Using cached boto3-1.12.49-py2.py3-none-any.whl (128 kB)
Collecting botocore<1.16,>=1.15.34
  Using cached botocore-1.15.49-py2.py3-none-any.whl (6.2 MB)
Collecting colorama<0.4.4
  Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-macosx_10_14_x86_64.whl
Processing /Users/gabdav01/Library/Caches/pip/wheels/e4/27/61/1cdc0b98cfda1ba5fc6d97cb7519f083427323ec7d191bfc0e/dpath-1.5.0-cp37-none-any.whl
Collecting click==7.1.1
  Using cached click-7.1.1-py2.py3-none-any.whl (82 kB)
Collecting click-log==0.3.2
  Using cached click_log-0.3.2-py2.py3-none-any.whl (4.6 kB)
Collecting cryptography<2.10,>=2.9
  Using cached cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl (1.8 MB)
Collecting dnspython==1.16.0
  Using cached dnspython-1.16.0-py2.py3-none-any.whl (188 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/ea/32/d0/77864057ba71ebce6ab3a8eda50c11971b005c12fa059eebde/f5_sdk-3.0.21-cp37-none-any.whl
Collecting Jinja2<2.12,>=2.11.2
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting kubernetes[adal]==11.0.0
  Using cached kubernetes-11.0.0-py3-none-any.whl (1.5 MB)
Collecting paramiko<2.8,>=2.7.1
  Using cached paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
Collecting passlib==1.7.2
  Using cached passlib-1.7.2-py2.py3-none-any.whl (507 kB)
Collecting pyjks<20,>=19.0.0
  Using cached pyjks-19.0.0-py2.py3-none-any.whl (45 kB)
Collecting pyjwt<1.8,>=1.7.1
  Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting pyopenssl<20.0.0,>=19.0.0
  Using cached pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting semantic_version<2.9,>=2.8.2
  Downloading semantic_version-2.8.5-py2.py3-none-any.whl (15 kB)
Collecting tenacity<6.2,>=6.1
  Using cached tenacity-6.1.0-py2.py3-none-any.whl (24 kB)
Collecting requests<2.24,>=2.23
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.26,>=1.25.6
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting idna<3,>=2.9
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting rsa<=3.5.0,>=3.1.2
  Using cached rsa-3.4.2-py2.py3-none-any.whl (46 kB)
Collecting docutils<0.16,>=0.10
  Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting s3transfer<0.4.0,>=0.3.0
  Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
  Using cached jmespath-0.9.5-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting cffi!=1.11.3,>=1.8
  Using cached cffi-1.14.0-cp37-cp37m-macosx_10_9_x86_64.whl (174 kB)
Collecting six>=1.4.1
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/12/43/4c/9c3051759105429cd633ed53ece367db44b082fce01c28db46/f5_icontrol_rest-1.3.13-cp37-none-any.whl
Collecting MarkupSafe>=0.23
  Using cached MarkupSafe-1.1.1-cp37-cp37m-macosx_10_6_intel.whl (18 kB)
Collecting google-auth>=1.0.1
  Using cached google_auth-1.14.1-py2.py3-none-any.whl (89 kB)
Collecting requests-oauthlib
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0
  Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Requirement already satisfied: setuptools>=21.0.0 in /Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages (from kubernetes[adal]==11.0.0->splatt==1.16.5) (46.1.3)
Collecting certifi>=14.05.14
  Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting adal>=1.0.2; extra == "adal"
  Downloading adal-1.2.3-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 522 kB/s
Collecting pynacl>=1.0.1
  Using cached PyNaCl-1.3.0-cp34-abi3-macosx_10_6_intel.whl (284 kB)
Collecting bcrypt>=3.1.3
  Using cached bcrypt-3.1.7-cp34-abi3-macosx_10_6_intel.whl (53 kB)
Processing /Users/gabdav01/Library/Caches/pip/wheels/e9/ce/00/66bd8b019c093719e2d78ea39a6f80c8133aecc1ec3f77f41c/twofish-0.3.0-cp37-cp37m-macosx_10_14_x86_64.whl
Collecting javaobj-py3
  Using cached javaobj_py3-0.4.1-py2.py3-none-any.whl (67 kB)
Collecting pyasn1>=0.3.5
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pycryptodomex
  Using cached pycryptodomex-3.9.7-cp37-cp37m-macosx_10_6_intel.whl (14.1 MB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.1.0-py3-none-any.whl (10 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
ERROR: awscli 1.18.51 has requirement botocore==1.16.1, but you'll have botocore 1.15.49 which is incompatible.
Installing collected packages: appdirs, pyasn1, rsa, PyYAML, docutils, urllib3, six, python-dateutil, jmespath, botocore, colorama, s3transfer, awscli, boto3, dpath, click, click-log, pycparser, cffi, cryptography, dnspython, idna, chardet, certifi, requests, f5-icontrol-rest, f5-sdk, MarkupSafe, Jinja2, cachetools, pyasn1-modules, google-auth, oauthlib, requests-oauthlib, websocket-client, pyjwt, adal, kubernetes, pynacl, bcrypt, paramiko, passlib, twofish, javaobj-py3, pycryptodomex, pyjks, pyopenssl, semantic-version, tenacity, splatt
  Running setup.py develop for splatt
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 adal-1.2.3 appdirs-1.4.3 awscli-1.18.51 bcrypt-3.1.7 boto3-1.12.49 botocore-1.15.49 cachetools-4.1.0 certifi-2020.4.5.1 cffi-1.14.0 chardet-3.0.4 click-7.1.1 click-log-0.3.2 colorama-0.4.3 cryptography-2.9.2 dnspython-1.16.0 docutils-0.15.2 dpath-1.5.0 f5-icontrol-rest-1.3.13 f5-sdk-3.0.21 google-auth-1.14.1 idna-2.9 javaobj-py3-0.4.1 jmespath-0.9.5 kubernetes-11.0.0 oauthlib-3.1.0 paramiko-2.7.1 passlib-1.7.2 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pycryptodomex-3.9.7 pyjks-19.0.0 pyjwt-1.7.1 pynacl-1.3.0 pyopenssl-19.1.0 python-dateutil-2.8.1 requests-2.23.0 requests-oauthlib-1.3.0 rsa-3.4.2 s3transfer-0.3.3 semantic-version-2.8.5 six-1.14.0 splatt tenacity-6.1.0 twofish-0.3.0 urllib3-1.25.9 websocket-client-0.57.0
(test-splatt-install)
gabdav01@e109198-mac:~/work/paas/splatt on aws-alloc-eip [!?$]
$ splatt --version
Traceback (most recent call last):
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (botocore 1.15.49 (/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages), Requirement.parse('botocore==1.16.1'), {'awscli'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/bin/splatt", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3258, in <module>
    @_call_aside
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
    f(*args, **kwargs)
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (botocore 1.15.49 (/Users/gabdav01/.virtualenvs/test-splatt-install/lib/python3.7/site-packages), Requirement.parse('botocore==1.16.1'), {'awscli'})
(test-splatt-install)

Additional context

pip install 'awscli==1.18.49' 'boto3<1.13' 'botocore<1.16'

Previously, every time a minor botocore version was changed, so did awscli. Boto3 still maintains this practice. But AWS cli failed to do so for the past 2 minor botocore version changes.

KaibaLopez commented 4 years ago

Hi @mindw , unless I am missing something I don't see an issue here, there should be no expectation that their minor versions would line up, they are different projects and minor versions don't correspond to each other.

mindw commented 4 years ago

Hi @KaibaLopez,

Thank you for taking the time for answering, this may be somewhat of a corner case :)

There is no expectations about version lining up. There is an expectation that when a minor version of a dependency is changed, so does the minor version of aws-cli. For example:

Without this, it is impossible to pin both aws-cli and boto3 to their compatible minor versions as in the above example: 'awscli<1.19 and 'boto3<1.13'.

mindw commented 4 years ago

@KaibaLopez 1.18.77 again bumped botocore to 1.17.0 without bumping the major version. boto3 has: https://github.com/boto/boto3/commit/64a5052571b9e20b1fb65a94083e7822630f55d1

It would be great if awscli would use the same versioning policy as boto3.

vjeeva commented 4 years ago

@KaibaLopez I'm seeing the exact same problem as well, this is causing errors to show in our configuration management which has never happened before for years of using boto3 and awscli.

shravan097 commented 4 years ago

im seeing same problem here

mindw commented 2 years ago

At least for awscli 1.21.0 both botocore & boto3 got a minor version bump as well. Tx!

tim-finnigan commented 2 years ago

I think this is a feature request rather than a bug. It also looks like there is some overlap with https://github.com/aws/aws-cli/issues/5563. @mindw can you confirm if that issue covers what you're asking for here?

mindw commented 2 years ago

I think this is a feature request rather than a bug. It also looks like there is some overlap with #5563. @mindw can you confirm if that issue covers what you're asking for here?

ChristophShyper commented 2 years ago

This feature is so frustrating...

I have in my requirements.txt:

awscli==1.22.80
boto3==1.21.25

Dependabot creates two PRs for each dependency, but one for boto3 will fail because of the recent changes for awscli:

 The conflict is caused by:
     awscli 1.22.80 depends on botocore==1.24.25
     boto3 1.21.26 depends on botocore<1.25.0 and >=1.24.26

Can you revert that last change and use >= instead of ==? Just like boto3 is doing? It's a shame we have failing builds because of that.

‼️ Working workaround ‼️ My requirements have now following order:

boto3==1.21.26
awscli==1.22.81

And my build is now using: cat aws_requirements.txt | xargs -n 1 pip3 install --no-cache-dir

So now both dependencies can be updated separately and builds will not fail. Of course I wouldn't recomend leaving only one dependency not-in-sync with the other. It's only a workaround for Dependabot and its failing builds.

nateprewitt commented 2 years ago

We've opened https://github.com/aws/aws-cli/issues/6854 to start discussing aligning versions of the CLI and Botocore. If anyone has feedback on this proposal, we'll be measuring interest based on the number of :+1: and :-1: responses we get.

This should bring the two versions in sync so we can better manage this going forward. The goal is to simplify the reasoning process of how versions pair. Note though that it won't strictly solve the conflict problem. The CLI has a requirement that needs a strict pin on Botocore, unlike boto3 where we have some more flexibility.

The safest option for users who need both Boto3 and the CLI on one system is to isolate the CLI in a virtualenv, or another containment mechanism, to ensure there aren't conflicts for Botocore.