CloudFormation helper scripts do not run on Python 3.10 (Ubuntu 22.04)

[pethers]

Resource Name

AWS::CloudFormation::Init

Details

Problem running it on Ubuntu22.04l. Not working running on python3.10

Error cfnbootstrap/packages/requests/cookies.py", line 159, in class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping): AttributeError: module 'collections' has no attribute 'MutableMapping'

Some solutions discussed at https://www.datasciencelearner.com/attributeerror-module-collections-has-no-attribute-mutablemapping/

"Attributeerror: module collections has no attribute mutablemapping error is because of internal code changes in the 3.10 version. If you are using any syntax related to the collections module which is compatible with the 3.9 version over the python 3.10-based python environment, you will get this error."

Related to https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/151

Installed using "pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz"

[ronocod]

Looks like the version of the requests library that's vendored into the tool is 2.6.0, judging from cfnbootstrap/packages/requests/__init__.py. If it's upgraded to 2.19.0 then it should fix this issue as that version contains this fix: https://github.com/psf/requests/pull/4501

[steverobbins]

My workaround for now

apt-get -y update
apt-get -y install python3-pip
pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
perl -pi -e "s/collections.MutableMapping/collections.abc.MutableMapping/g" /usr/local/lib/python3.10/dist-packages/cfnbootstrap/packages/requests/cookies.py
perl -pi -e "s/collections.MutableMapping/collections.abc.MutableMapping/g" /usr/local/lib/python3.10/dist-packages/cfnbootstrap/packages/requests/structures.py
perl -pi -e "s/from collections import Mapping/from collections.abc import Mapping/g" /usr/local/lib/python3.10/dist-packages/cfnbootstrap/packages/requests/sessions.py
perl -pi -e "s/collections.Mapping/collections.abc.Mapping/g" /usr/local/lib/python3.10/dist-packages/cfnbootstrap/packages/requests/utils.py
perl -pi -e "s/collections.Callable/collections.abc.Callable/g" /usr/local/lib/python3.10/dist-packages/cfnbootstrap/packages/requests/models.py

It's enough to run cfn-signal

[ronocod]

This is our workaround for Debian, it doesn't involve editing the files:

# Install Python 3.9
add-apt-repository ppa:deadsnakes/ppa -y
apt update -y
apt install python3.9 python3-pip python3.9-distutils python3-apt -y

python3.9 -m pip install wheel
python3.9 -m pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-2.0-10.tar.gz

# Moving files - we do it but likely not needed
mkdir -p /opt/aws/bin
ln -s /usr/local/bin/cfn-* /opt/aws/bin/

# Need to run `python3.9 {script_path}` whenever you're invoking a script
python3.9 /opt/aws/bin/cfn-signal --help

[benipeled]

@cfn-maintainers (not sure who it is) what about open-sourcing the cloudformation-helpers project? Among other reasons for open-sourcing the project - such an issue could be fixed quickly by a simple PR

[adamchainz]

Slightly more robust than sed, I’ve come up with a workaround to apply the below patch with:

git apply --unsafe-paths /tmp/cfn-bootstrap.patch --directory /usr/local/lib/python3.10/dist-packages/cfnbootstrap/

patch file:

diff --git a/cfnbootstrap b/cfnbootstrap
--- cfnbootstrap/packages/requests/cookies.py
+++ cfnbootstrap/packages/requests/cookies.py
@@ -7,7 +7,7 @@
 """

 import time
-import collections
+import collections.abc
 from .compat import cookielib, urlparse, urlunparse, Morsel

 try:
@@ -156,7 +156,7 @@
     Use .get and .set and include domain and path args in order to be more specific."""

-class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
+class RequestsCookieJar(cookielib.CookieJar, collections.abc.MutableMapping):
     """Compatibility class; is a cookielib.CookieJar, but exposes a dict
     interface.

--- cfnbootstrap/packages/requests/structures.py
+++ cfnbootstrap/packages/requests/structures.py
@@ -8,10 +8,10 @@

 """

-import collections
+import collections.abc

-class CaseInsensitiveDict(collections.MutableMapping):
+class CaseInsensitiveDict(collections.abc.MutableMapping):
     """
     A case-insensitive ``dict``-like object.

--- cfnbootstrap/packages/requests/sessions.py
+++ cfnbootstrap/packages/requests/sessions.py
@@ -9,7 +9,7 @@

 """
 import os
-from collections import Mapping
+from collections.abc import Mapping
 from datetime import datetime

 from .auth import _basic_auth_str

--- cfnbootstrap/packages/requests/utils.py
+++ cfnbootstrap/packages/requests/utils.py
@@ -11,7 +11,7 @@

 import cgi
 import codecs
-import collections
+import collections.abc
 import io
 import os
 import platform
@@ -163,7 +163,7 @@
     if isinstance(value, (str, bytes, bool, int)):
         raise ValueError('cannot encode objects that are not 2-tuples')

-    if isinstance(value, collections.Mapping):
+    if isinstance(value, collections.abc.Mapping):
         value = value.items()

     return list(value)

--- cfnbootstrap/packages/requests/models.py
+++ cfnbootstrap/packages/requests/models.py
@@ -7,7 +7,7 @@
 This module contains the primary objects that power Requests.
 """

-import collections
+import collections.abc
 import datetime

 from io import BytesIO, UnsupportedOperation
@@ -166,10 +166,10 @@
         if event not in self.hooks:
             raise ValueError('Unsupported event specified, with event name "%s"' % (event))

-        if isinstance(hook, collections.Callable):
+        if isinstance(hook, collections.abc.Callable):
             self.hooks[event].append(hook)
         elif hasattr(hook, '__iter__'):
-            self.hooks[event].extend(h for h in hook if isinstance(h, collections.Callable))
+            self.hooks[event].extend(h for h in hook if isinstance(h, collections.abc.Callable))

     def deregister_hook(self, event, hook):
         """Deregister a previously registered hook.

[saste]

@cfn-maintainers (not sure who it is) what about open-sourcing the cloudformation-helpers project? Among other reasons for open-sourcing the project - such an issue could be fixed quickly by a simple PR

+1 on this.

In addition, I need to install this on several platforms, ranging from python 3.6 to 3.10, so I'd need the package to be robust and support several python versions.

[benipeled]

@mrinaudo-aws @kanitkah can you please help with this issue? (or at least assign it to the relevant)

[kanitkah]

@benipeled We will take a look and post updates after we investigate.

[garv123]

@benipeled- This issue should have been fixed in the latest release. Please download 2.0-18 version.

The latest also now points to 2.0-18.

[benipeled]

@garv123 Thank you! verified with cfn-signal and it works as expected

PS> I'm not sure if it's a regression or was always there under the hood - with both cfn-signal and cfn-get-metadata I see the following warn

/usr/local/lib/python3.10/dist-packages/cfnbootstrap/packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.3) or chardet (2.3.0)/charset_normalizer (None) doesn't match a supported version!
  warnings.warn(
Could not open /var/log/cfn-init.log for logging.  Using stderr instead.