Closed ericbn closed 1 month ago
Thanks @ericbn - this was recently added to the v1 README. There is still more testing required for Python 3.12 on v2, but the v2 README will be updated when that is complete.
Also for installing v2 of the AWS CLI we recommend using one of the options documented here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.
@tim-finnigan not one of the options listed mention a package manager for the installation, which is a surprisingly common thing to want, so it's easy for a fellow user to see where @ericbn is coming from.
Is there any time frame for adding python 3.12 support? Alpine Linux is upgrading python to 3.12.
ruamel.yaml.clib
versions before 0.2.8
don't support Python 3.12.Besides ruamel.yaml.clib
, PyInstaller
also needs an update: https://github.com/aws/aws-cli/issues/8245. The relevant pull request https://github.com/aws/aws-cli/pull/8570 is not merged yet.
After updating dependencies, I got an issue when I was building the wheel:
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/build/aws-cli-v2/src/awscli-2.15.36/backends/pep517.py", line 65, in build_wheel
_inject_wheel_extras(os.path.join(wheel_directory, whl_filename))
File "/build/aws-cli-v2/src/awscli-2.15.36/backends/pep517.py", line 190, in _inject_wheel_extras
_build_and_inject_ac_index(BUILD_DIR, extracted_wheel_dir)
File "/build/aws-cli-v2/src/awscli-2.15.36/backends/pep517.py", line 219, in _build_and_inject_ac_index
ac_index_build_name = _build_ac_index(ac_index_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/build/aws-cli-v2/src/awscli-2.15.36/backends/pep517.py", line 231, in _build_ac_index
from awscli.autocomplete.generator import generate_index
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/autocomplete/generator.py", line 17, in <module>
from awscli.autocomplete.serverside.indexer import APICallIndexer
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/autocomplete/serverside/indexer.py", line 18, in <module>
import awscli.clidriver
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/clidriver.py", line 21, in <module>
import botocore.session
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/session.py", line 27, in <module>
import botocore.client
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/client.py", line 16, in <module>
from botocore import UNSIGNED, waiter, xform_name
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/waiter.py", line 17, in <module>
from botocore.docs.docstring import WaiterDocstring
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/docs/__init__.py", line 15, in <module>
from botocore.docs.service import ServiceDocumenter
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/docs/service.py", line 13, in <module>
from botocore.docs.bcdoc.restdoc import DocumentStructure
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/docs/bcdoc/restdoc.py", line 15, in <module>
from botocore.compat import OrderedDict
File "/build/aws-cli-v2/src/awscli-2.15.36/awscli/botocore/compat.py", line 36, in <module>
from botocore.vendored.six.moves import http_client
ModuleNotFoundError: No module named 'botocore.vendored.six.moves'
After updating vendored six.py in vendored botocore like https://github.com/boto/botocore/pull/2551, I can build the wheel, while the test suite fails in several places: test-failures.txt. At first glance, it seems more changes should be backported:
https://github.com/aws/aws-cli/pull/8106 https://github.com/boto/botocore/pull/2855 https://github.com/boto/botocore/pull/2967
@yan12125 I make two patches in PKGBUILD.
diff -Naur awscli-2.15.40/awscli/customizations/eks/kubeconfig.py awscli-2.15.40_/awscli/customizations/eks/kubeconfig.py
--- awscli-2.15.40/awscli/customizations/eks/kubeconfig.py 2024-04-20 11:06:32.658178400 +0800
+++ awscli-2.15.40_/awscli/customizations/eks/kubeconfig.py 2024-04-20 11:09:03.390341218 +0800
@@ -70,6 +70,13 @@
return name in [cluster['name']
for cluster in self.content['clusters'] if 'name' in cluster]
class KubeconfigValidator(object): def init(self): diff -Naur awscli-2.15.40/tests/functional/eks/testkubeconfig.py awscli-2.15.40/tests/functional/eks/test_kubeconfig.py --- awscli-2.15.40/tests/functional/eks/testkubeconfig.py 2024-04-20 11:06:32.568177140 +0800 +++ awscli-2.15.40/tests/functional/eks/test_kubeconfig.py 2024-04-20 11:12:52.110419266 +0800 @@ -121,7 +121,7 @@ ]) loaded_config = self._loader.load_kubeconfig(simple_path) self.assertEqual(loaded_config.content, content)
self._validator.validate_config.assert_called_with(Kubeconfig(simple_path, content))
def test_load_noexist(self): @@ -130,7 +130,7 @@ loaded_config = self._loader.load_kubeconfig(no_exist_path) self.assertEqual(loaded_config.content, _get_new_kubeconfig_content())
self._validator.validate_config.assert_called_with( Kubeconfig(no_exist_path, _get_new_kubeconfig_content()))
def test_load_empty(self): @@ -138,7 +138,7 @@ loaded_config = self._loader.load_kubeconfig(empty_path) self.assertEqual(loaded_config.content, _get_new_kubeconfig_content())
@@ -147,4 +147,4 @@ self.assertRaises(KubeconfigInaccessableError, self._loader.load_kubeconfig, current_directory)
diff -Naur awscli-2.15.40/tests/unit/botocore/test_client.py awscli-2.15.40_/tests/unit/botocore/test_client.py
--- awscli-2.15.40/tests/unit/botocore/test_client.py 2024-04-20 15:00:39.465686191 +0800
+++ awscli-2.15.40_/tests/unit/botocore/test_client.py 2024-04-20 15:07:29.335115713 +0800
@@ -960,7 +960,7 @@
lines = [
(' Creates an iterator that will paginate through responses '
'from :py:meth:`MyService.Client.test_operation`.'),
- ' **Request Syntax** ',
+ ' **Request Syntax**',
' ::',
' response_iterator = paginator.paginate(',
" Foo='string',",
@@ -976,17 +976,17 @@
' :type Bar: string',
' :param Bar: Documents Bar',
' :type PaginationConfig: dict',
- ' :param PaginationConfig: ',
+ ' :param PaginationConfig:',
(' A dictionary that provides parameters to control '
'pagination.'),
- ' - **MaxItems** *(integer) --* ',
+ ' - **MaxItems** *(integer) --*',
(' The total number of items to return. If the total '
'number of items available is more than the value specified '
'in max-items then a ``NextToken`` will be provided in the '
'output that you can use to resume pagination.'),
- ' - **PageSize** *(integer) --* ',
+ ' - **PageSize** *(integer) --*',
' The size of each page.',
- ' - **StartingToken** *(string) --* ',
+ ' - **StartingToken** *(string) --*',
(' A token to specify where to start paginating. This is '
'the ``NextToken`` from a previous response.'),
' :returns: None',
diff -Naur awscli-2.15.40/tests/unit/botocore/test_waiters.py awscli-2.15.40_/tests/unit/botocore/test_waiters.py
--- awscli-2.15.40/tests/unit/botocore/test_waiters.py 2024-04-20 15:00:39.469019486 +0800
+++ awscli-2.15.40_/tests/unit/botocore/test_waiters.py 2024-04-20 15:04:08.810188250 +0800
@@ -648,7 +648,7 @@
(' Polls :py:meth:`MyService.Client.foo` every 1 '
'seconds until a successful state is reached. An error '
'is returned after 1 failed checks.'),
- ' **Request Syntax** ',
+ ' **Request Syntax**',
' ::',
' waiter.wait(',
" bar='string'",
diff -Naur awscli-2.15.40/tests/unit/botocore/test_utils.py awscli-2.15.40_/tests/unit/botocore/test_utils.py
--- awscli-2.15.40/tests/unit/botocore/test_utils.py 2024-04-20 12:09:38.883650919 +0800
+++ awscli-2.15.40_/tests/unit/botocore/test_utils.py 2024-04-20 12:11:56.434812142 +0800
@@ -1000,17 +1000,24 @@
'https://bucket.s3.amazonaws.com/key.txt')
-class TestSwitchToChunkedEncodingForNonSeekableObjects(unittest.TestCase):
assert prepared_request.headers == {'Transfer-Encoding': 'chunked'}
class TestInstanceCache(unittest.TestCase):
There are still a few test-errors only about test_update_kubeconfig
.
Here is error log: test-error.txt
Many thanks! Now I can build aws-cli-v2 for Python 3.12 on Arch Linux (not sure why there are other errors for you). I pushed your patches to the unofficial package https://aur.archlinux.org/packages/aws-cli-v2.
A note about aws-cli-v2 package in Arch Linux: I was the maintainer for the official package extra/aws-cli-v2 until recently. I am unable to handle so many patches (ones for Python 3.12 & some others) while keeping the package up-to-date. Other Arch Linux Package Maintainers don't have enough time/energy, either. As a result, that package is dropped to AUR.
FYI: alpine 3.20.0 removed aws-cli due to missing python 3.12 compatibility
Anyone using alpine (base) images and relying on aws-cli
should pin tags to the 3.19 version of alpine
I am running Ubuntu Noble and need to use awscli. Due to the missing Python 3.12 compatibility, awscli was removed from Ubuntu Noble. My hack / work around was to use the Debian package (and install it with dpkg -i
) and fix things up manually. The only problems that I encountered are due to botocore:
wget https://github.com/benjaminp/six/raw/c1b416f24de52ebd8eaed3e379ab85ec9c3e935b/six.py -O /usr/lib/python3/dist-packages/awscli/botocore/vendored/six.py
The result works fine for me.
Dunno how old Debian's awscli version is compared to upstream, but to me this sounds like "properly" fixing Python 3.12 compatibility should doable.
Edit: but of course I never ran the test suite. But others in this issue seem to already have investigated how to fix things there.
Hello, Apart from the patches required, there are two very simple changes required:
Can we please proceed with these two ones? I would volunteer for any manual testing required. These are not breaking changes and with those we can identify other issues much more simply.
@tim-finnigan
Sorry to spam, but is there any way the PRs mentioned by @onlined can be merged?
Allowing aws-cli to run on Python 3.12 with such minimal changes would seem like it should be a high priority…
I am running Ubuntu Noble and need to use awscli. Due to the missing Python 3.12 compatibility, awscli was removed from Ubuntu Noble. My hack / work around was to use the Debian package (and install it with
dpkg -i
) and fix things up manually. The only problems that I encountered are due to botocore:
@psychon The official way to run the aws-cli on Noble is to use the snap package. The deb package’s removal from Noble was influenced by the difficulty in keeping the package current via the Ubuntu SRU process.
@marrek-az This might be a bit off-topic here, but okay: How do you know it is official? None of https://github.com/aws/aws-cli/tree/v2?tab=readme-ov-file#installation nor https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html mention it.
I know that https://snapcraft.io/aws-cli and https://snapcraft.io/publisher/aws have green checkmarks indicating Verified account
, but I found zero clues to what that actually means. None of the links on https://snapcraft.io/aws-cli go to any kind of source code or indications explaining how the snap is produced / built. Finally, I found:
the snaps are not maintained by AWS, but by SnapCraft.
https://github.com/aws/aws-cli/issues/4357#issuecomment-699156068
At that point I had serious doubts about this snap not being a big supply chain attack.
Edit: To clarify the previous sentence: Snapcraft claims that the snap is "official" and comes from AWS and AWS says they are not maintaining the snap. Thus, it seems to me that snapcraft is impersonating AWS.
@marrek-az This might be a bit off-topic here, but okay: How do you know it is official? None of https://github.com/aws/aws-cli/tree/v2?tab=readme-ov-file#installation nor https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html mention it.
the snaps are not maintained by AWS, but by SnapCraft.
At that point I had serious doubts about this snap not being a big supply chain attack.
Edit: To clarify the previous sentence: Snapcraft claims that the snap is "official" and comes from AWS and AWS says they are not maintaining the snap. Thus, it seems to me that snapcraft is impersonating AWS.
@psychon You ask some interesting questions, and I think there are some opportunities for improvement there. I can tell you that the snap is pre-installed on the official Ubuntu AMI on AWS, so you can check it from there. You can also diff the snap-installed version against the zip archive. It should be mentioned in the official Ubuntu documentation, and I’m going to ask them about doing so for consistency. Or do the PR for the doc update myself. One of those.
As for the official aws-cli docs…, I’ll have to let someone else weigh in on that.
@psychon Regarding the official status of the aws-cli snap on Ubuntu, a colleague pointed me to this over the weekend. It’s from the official Ubuntu documentation.
EDIT: the link is now included in the official Ubuntu Noble 24.04 release notes. Hopefully, that will help.
aws-cli got added to alpine 3.20 but it errors out :(
@gattytto can you describe the error? best would be if you could open a issue on Alpine's issue tracker: https://gitlab.alpinelinux.org/alpine/aports/-/issues. Thanks!
right now alpine 3.19 works, but 3.20 docker.io/amd64/python:3.12.4-alpine3.20 aws cli gives this kind of error:
/ # pip install datetime
Traceback (most recent call last):
File "/usr/local/bin/pip", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 78, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
module = importlib.import_module(module_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/usr/local/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 15, in <module>
from pip._internal.cli.req_command import (
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 19, in <module>
from pip._internal.index.package_finder import PackageFinder
File "/usr/local/lib/python3.12/site-packages/pip/_internal/index/package_finder.py", line 31, in <module>
from pip._internal.req import InstallRequirement
File "/usr/local/lib/python3.12/site-packages/pip/_internal/req/__init__.py", line 9, in <module>
from .req_install import InstallRequirement
File "/usr/local/lib/python3.12/site-packages/pip/_internal/req/req_install.py", line 40, in <module>
from pip._internal.operations.install.wheel import install_wheel
File "/usr/local/lib/python3.12/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>
from pip._vendor.distlib.scripts import ScriptMaker
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/distlib/scripts.py", line 16, in <module>
from .compat import sysconfig, detect_encoding, ZipFile
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/distlib/compat.py", line 81, in <module>
import xmlrpc.client as xmlrpclib
File "/usr/local/lib/python3.12/xmlrpc/client.py", line 272, in <module>
if _try('%Y'): # Mac OS X
^^^^^^^^^^
File "/usr/local/lib/python3.12/xmlrpc/client.py", line 269, in _try
return _day0.strftime(fmt) == '0001'
^^^^^^^^^^^^^^^^^^^
AttributeError: 'datetime.datetime' object has no attribute 'times'. Did you mean: 'time'?```
@gattytto You error is entirely unrelated to this tool (you're not even running aws
there).
I see that you Python installation is in /usr/local/
. This isn't the system Python either, so it's not an Alpine issue. It sounds to me like this Python was installed from source but the installation is borked somehow.
@WhyNotHugo thank you very much for your answer, here is a more specific example of how this does not work
/ # export AWS_BUCKET_NAME=ssd-bcc063a2c5d-1623-4875-a517-e5bfda543a6e AWS_ACCES
S_KEY_ID=DNKDQD4QETBWF1RIZU8C AWS_SECRET_ACCESS_KEY=q9wkHAw5RsQ5L658UN7a7dXiKsfi
z3KpueEfEmQl AWS_ENDPOINT_URL=http://rook-ceph-rgw-ssd-store.rook-ceph.svc
/ # aws s3 ls s3://$AWS_BUCKET_NAME
'datetime.datetime' object has no attribute 'times'
/ #
FROM docker.io/amd64/python:3.12.4-alpine3.20
ENV CARGO_BUILD_JOBS=2
RUN pip install --upgrade pip
RUN pip install \
grpcio==1.64.1 \
boto3 \
solana \
base58 \
kubernetes \
requests \
debugpy \
grpcio-reflection==1.62.2 \
protobuf \
grpcio-health-checking==1.62.2 \
google-api-core[grpc] \
libcst==0.3.23 \
googleapis-common-protos>=1.63.2 \
proto-plus>=1.24.0 \
quart \
pynacl \
hypercorn
RUN apk add --no-cache \
gcc \
libffi-dev \
musl-dev \
openssl-dev \
aws-cli \
jq
if I just change it to alpine3.19 for the base image it works just fine
Could you please try it out to use official alpine images (e.g. docker.io/library/alpine:latest
)? Also please try not to install python modules with pip, as this breaks system packages. Use system packages instead (apk add py3-*). Otherwise if it still fails, please discuss this issue in a more Alpine-related issue tracker, that this issue is only for Python 3.12 support on aws-cli.
@fossdd thank you very much for your answer, I am using an officially provided python image which is based in officially provided alpine image: https://github.com/docker-library/python/blob/master/3.12/alpine3.20/Dockerfile
FROM alpine:3.20
here's a more minimalistic test using recommended procedures as not adding anything by pip and installing aws-cli using apk. It still does not work
FROM docker.io/amd64/python:3.12.4-alpine3.20
RUN apk add --no-cache \
aws-cli
/ # export AWS_BUCKET_NAME=ssd-bcc063a2c5d-1623-4875-a517-e5bfda543a6e AWS_ACCESS_KEY_ID=DNKDQD4QETBWF1RIZU8C AWS_SECRET_ACCESS_KEY=q9wkHAw5RsQ5L658UN7a7dXiKsfiz3KpueEfEmQl AWS_ENDPOINT_URL=http://rook-ceph-rgw-ssd-store.rook-ceph.svc
/ # aws s3 ls s3://$AWS_BUCKET_NAME
'datetime.datetime' object has no attribute 'times'
same happens if I Just use "FROM docker.io/python:3.12.4-alpine3.20" for the base image with no changes other than just apk add aws-cli
@gattytto Test with the following Dockerfile instead:
FROM alpine:3.20
RUN apk add --no-cache aws-cli
On my end that works. However, note that alpine (specifically @fossdd who replied to you earlier) is performing their own patches to get this working (see https://github.com/aws/aws-cli/issues/8689#issuecomment-2132242843).
Those patches are not being supplied by AWS, so breakages occurring from the python:3.12.4-alpine3.20 image are some combination of issues with those patches and/or incompatibilities with libraries that are being updated by that python image.
This issue is specifically a request for AWS to update aws-cli with official support for python 3.12 (not downstream patches from alpine). That's why @fossdd was directing you to report the issues on alpine-related trackers.
thank you very much for the info and the answer @bobziuchovsky
here's a more minimalistic test using recommended procedures as not adding anything by pip and installing aws-cli using apk. It still does not work
FROM docker.io/amd64/python:3.12.4-alpine3.20 RUN apk add --no-cache \ aws-cli
/ # export AWS_BUCKET_NAME=ssd-bcc063a2c5d-1623-4875-a517-e5bfda543a6e AWS_ACCESS_KEY_ID=DNKDQD4QETBWF1RIZU8C AWS_SECRET_ACCESS_KEY=q9wkHAw5RsQ5L658UN7a7dXiKsfiz3KpueEfEmQl AWS_ENDPOINT_URL=http://rook-ceph-rgw-ssd-store.rook-ceph.svc / # aws s3 ls s3://$AWS_BUCKET_NAME 'datetime.datetime' object has no attribute 'times'
hopefully those aren't your real aws credentials here...
hopefully those aren't your real aws credentials here...
It's a self hosted COSI implementation but yes lol
Now that a version of V2 has been released that contains the fixes from #8917, does anything else need to be done to support Python 3.12?
Or can this issue be closed?
This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.
Closed as https://github.com/aws/aws-cli/pull/8917 was merged. See update in 2.17.52
CHANGELOG:
- ehancement:python: Update bundled Python interpreter version to 3.12.6
Describe the feature
aws-cli V2 currently supports:
Add support for Python 3.12.x too.
Use Case
I installed aws-cli V2 using Homebrew, and it's the last tool I have installed still using Python 3.11. But more importantly, I believe this helps keep V2 up-to-date.
Proposed Solution
Add support for Python 3.12.x too.
Other Information
Python 3.12 was first released on 2023-10-02 and has EOL scheduled for 2028-10. See https://devguide.python.org/versions/
Acknowledgements
CLI version used
2.13.37
Environment details (OS name and version, etc.)
Darwin/21.6.0