build-s3-dist.sh fails with pip error

[awseric]

Description

On python 3.9+ on latest release when running ./deployment/build-s3-dist.sh the script fails with:

  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/Users/austinem/Projects/STSci/aws-control-tower-customizations/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/austinem/Projects/STSci/aws-control-tower-customizations/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/austinem/Projects/STSci/aws-control-tower-customizations/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/command/sdist.py", line 112, in add_defaults
          super().add_defaults()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/private/var/folders/85/24lgg6zx453gk25wy53hgljc0000gr/T/pip-build-env-opzxm9jm/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Traceback (most recent call last):
  File "/Users/austinem/Projects/STSci/aws-control-tower-customizations/deployment/lambda_build.py", line 128, in <module>
    main(sys.argv[1:])
  File "/Users/austinem/Projects/STSci/aws-control-tower-customizations/deployment/lambda_build.py", line 95, in main
    install_dependencies(
  File "/Users/austinem/Projects/STSci/aws-control-tower-customizations/deployment/lambda_build.py", line 48, in install_dependencies
    subprocess.run(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pip', 'install', '--quiet', 'source/src', '--target', 'deployment/dist']' returned non-zero exit status 1.

It appears that this is due to source/src/setup.py yorm installing a PyYAML version with a known issue: Error installing Pyyaml==5.4, Cython_sources #724

To Reproduce Install a virtualenv with Python version >= 3.9. Run the installer script at ./deployment/build-s3-dist.sh as described in the documentation.

Expected behavior Installation success. Exit code 0.

Please complete the following information about the solution:

• [ ] Version: 2.7.0

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0089) - customizations-for-aws-control-tower Solution. Version: v1.0.0". You can also find the version from releases

• [ ] Region: [e.g. us-east-1] N/A
• [ ] Was the solution modified from the version published on this repository? No
• [ ] If the answer to the previous question was yes, are the changes available on GitHub?
• [ ] Have you checked your service quotas for the sevices this solution uses? N/A
• [ ] Were there any errors in the CloudWatch Logs? N/A

[awseric]

pinning PyYAML to 5.3.1 in .source/src/setup.py appears to resolve the issue.

[snebhu3]

@awseric we were able to reproduce the issue, and the suggested fix of pinning PyYAML to 5.3.1 in setup.py works. We have created an internal backlog to make this change.

[wiltangg]

As mentioned in #184, latest patch v.2.7.1 (release notes) included changes to pin the PyYAML in setup.py to 5.3.1.

I am going to resolve this issue, please reopen if there are follow up questions.