search

aws-cloudformation / cloudformation-cli

The CloudFormation Provider Development Toolkit allows you to author your own resource providers and modules that can be used by CloudFormation.
Apache License 2.0
318 stars 161 forks source link

cfn submit fails because pip install exited with non-zero status code #905

Closed bcosynot closed 1 year ago

bcosynot commented 1 year ago

cfn submit --dry-run fails with an error

Contents of rpdk.log

[2022-10-11T17:31:54Z] DEBUG    - Logging set up successfully
[2022-10-11T17:31:54Z] DEBUG    - Running submit: Namespace(command=<function submit at 0x11851fee0>, dry_run=True, endpoint_url=None, region='us-east-1', role_arn=None, set_default=False, subparser_name='submit', use_role=True, verbose=3, version=False)
[2022-10-11T17:31:54Z] DEBUG    - Root directory: /Users/me/IdeaProjects/custom-resource-type
[2022-10-11T17:31:54Z] DEBUG    - Loading project file '/Users/me/IdeaProjects/custom-resource-type/.rpdk-config'
[2022-10-11T17:31:54Z] INFO     - Validating your resource specification...
[2022-10-11T17:31:54Z] WARNING  - Explicitly specify value for insertionOrder for array: Authors
[2022-10-11T17:31:54Z] WARNING  - Explicitly specify value for tagging
[2022-10-11T17:31:54Z] DEBUG    - Resolved '#/definitions/InitechDateFormat' to 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/InitechDateFormat'
[2022-10-11T17:31:54Z] DEBUG    - Parsed 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/InitechDateFormat' to '(<BASE>, 'definitions', 'InitechDateFormat')'
[2022-10-11T17:31:54Z] DEBUG    - Edge from '(<BASE>, 'properties', 'DueDate')' to '(<BASE>, 'definitions', 'InitechDateFormat')'
[2022-10-11T17:31:54Z] DEBUG    - Resolved '#/definitions/InitechDateFormat' to 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/InitechDateFormat'
[2022-10-11T17:31:54Z] DEBUG    - Parsed 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/InitechDateFormat' to '(<BASE>, 'definitions', 'InitechDateFormat')'
[2022-10-11T17:31:54Z] DEBUG    - Edge from '(<BASE>, 'properties', 'ApprovalDate')' to '(<BASE>, 'definitions', 'InitechDateFormat')'
[2022-10-11T17:31:54Z] DEBUG    - Resolved '#/definitions/Memo' to 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/Memo'
[2022-10-11T17:31:54Z] DEBUG    - Parsed 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/Memo' to '(<BASE>, 'definitions', 'Memo')'
[2022-10-11T17:31:54Z] DEBUG    - Edge from '(<BASE>, 'properties', 'Memo')' to '(<BASE>, 'definitions', 'Memo')'
[2022-10-11T17:31:54Z] DEBUG    - Resolved '#/definitions/Memo' to 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/Memo'
[2022-10-11T17:31:54Z] DEBUG    - Parsed 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/Memo' to '(<BASE>, 'definitions', 'Memo')'
[2022-10-11T17:31:54Z] DEBUG    - Edge from '(<BASE>, 'properties', 'SecondCopyOfMemo')' to '(<BASE>, 'definitions', 'Memo')'
[2022-10-11T17:31:54Z] DEBUG    - Resolved '#/definitions/Tag' to 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/Tag'
[2022-10-11T17:31:54Z] DEBUG    - Parsed 'file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json#/definitions/Tag' to '(<BASE>, 'definitions', 'Tag')'
[2022-10-11T17:31:54Z] DEBUG    - Edge from '(<BASE>, 'properties', 'Tags', 'items')' to '(<BASE>, 'definitions', 'Tag')'
[2022-10-11T17:31:54Z] DEBUG    - Rewriting refs in '<BASE>' (file:///Users/me/IdeaProjects/custom-resource-type/cronitor-cronitor-monitor.json)
[2022-10-11T17:31:54Z] DEBUG    -   '#/definitions/InitechDateFormat' -> '#/definitions/InitechDateFormat'
[2022-10-11T17:31:54Z] DEBUG    -   '#/definitions/InitechDateFormat' -> '#/definitions/InitechDateFormat'
[2022-10-11T17:31:54Z] DEBUG    -   '#/definitions/Memo' -> '#/definitions/Memo'
[2022-10-11T17:31:54Z] DEBUG    -   '#/definitions/Memo' -> '#/definitions/Memo'
[2022-10-11T17:31:54Z] DEBUG    -   '#/definitions/Tag' -> '#/definitions/Tag'
[2022-10-11T17:31:54Z] WARNING  - Resource schema is valid.
[2022-10-11T17:31:54Z] INFO     - Validating your resource schema...
[2022-10-11T17:31:54Z] DEBUG    - inputs not found. Not writing to package.
[2022-10-11T17:31:54Z] DEBUG    - Package started
[2022-10-11T17:31:54Z] DEBUG    - '/Users/me/IdeaProjects/custom-resource-type/build' not found, skipping removal
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/python/codegen.py", line 239, in _remove_build_artifacts
    shutil.rmtree(deps_path)
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/shutil.py", line 700, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/shutil.py", line 698, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/me/IdeaProjects/custom-resource-type/build'
[2022-10-11T17:31:54Z] DEBUG    - Dependencies build started from '/Users/me/IdeaProjects/custom-resource-type'
[2022-10-11T17:31:54Z] DEBUG    - command is 'pip install --no-cache-dir --no-color --disable-pip-version-check --upgrade --requirement /project/requirements.txt --target /project/build'
[2022-10-11T17:31:54Z] WARNING  - Starting Docker build. This may take several minutes if the image 'lambci/lambda:build-python3.7' needs to be pulled first.
[2022-10-11T17:31:54Z] DEBUG    - Trying paths: ['/Users/me/.docker/config.json', '/Users/me/.dockercfg']
[2022-10-11T17:31:54Z] DEBUG    - Found file at path: /Users/me/.docker/config.json
[2022-10-11T17:31:54Z] DEBUG    - Found 'auths' section
[2022-10-11T17:31:54Z] DEBUG    - Auth data for 037727099164.dkr.ecr.us-east-1.amazonaws.com is absent. Client might be using a credentials store instead.
[2022-10-11T17:31:54Z] DEBUG    - Auth data for 390763969141.dkr.ecr.us-east-2.amazonaws.com is absent. Client might be using a credentials store instead.
[2022-10-11T17:31:54Z] DEBUG    - Found 'credsStore' section
[2022-10-11T17:31:55Z] DEBUG    - Caught downstream error
Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/python/codegen.py", line 285, in _docker_build
    logs = docker_client.containers.run(
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/docker/models/containers.py", line 875, in run
    raise ContainerError(
docker.errors.ContainerError: Command 'pip install --no-cache-dir --no-color --disable-pip-version-check --upgrade --requirement /project/requirements.txt --target /project/build' in image 'lambci/lambda:build-python3.7' returned non-zero exit status 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/core/cli.py", line 100, in main
    args.command(args)
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/core/submit.py", line 15, in submit
    project.submit(
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/core/project.py", line 650, in submit
    self._add_resources_content_to_zip(zip_file)
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/core/project.py", line 678, in _add_resources_content_to_zip
    self._plugin.package(self, zip_file)
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/python/codegen.py", line 227, in package
    self._build(project.root)
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/python/codegen.py", line 246, in _build
    self._docker_build(base_path)
  File "/Users/me/.pyenv/versions/3.8.1/lib/python3.8/site-packages/rpdk/python/codegen.py", line 303, in _docker_build
    raise DownstreamError("Error running docker build") from e
rpdk.core.exceptions.DownstreamError: Error running docker build
bcosynot commented 1 year ago

I found that removing the line user=f"{os.geteuid()}:{os.getgid()}", from rpdk/python/codegen.py was no longer causing the error for me?

mmaeng commented 1 year ago

Strange, That command is interfacing with the OS you are running it under. What OS are you running?

bcosynot commented 1 year ago

@mmaeng I am on macOS 12.6

mmaeng commented 1 year ago

@bcosynot I am also on MacOS 12.6 and I am unable to repro the issue on my setup.

What does pip freeze output for you?

bcosynot commented 1 year ago 
agate==1.6.3
agate-dbf==0.2.2
agate-excel==0.2.5
agate-sql==0.5.8
attrs==22.1.0
aws-sam-translator==1.53.0
Babel==2.9.1
boto3==1.24.89
botocore==1.27.89
certifi==2020.6.20
cfn-flip==1.3.0
cfn-lint==0.67.0
chardet==3.0.4
charset-normalizer==2.1.1
click==8.1.3
cloudformation-cli==0.2.25
cloudformation-cli-go-plugin==2.0.4
cloudformation-cli-java-plugin==2.0.12
cloudformation-cli-python-plugin==2.1.5
cloudformation-cli-typescript-plugin==1.0.1
colorama==0.4.5
csvkit==1.0.6
dbfread==2.0.7
docker==6.0.0
et-xmlfile==1.1.0
exceptiongroup==1.0.0rc9
future==0.18.2
greenlet==1.1.2
httpie==2.2.0
httpie-jwt-auth==0.4.0
hypothesis==6.56.2
idna==2.10
iniconfig==1.1.1
isodate==0.6.1
Jinja2==2.11.3
jmespath==1.0.1
jschema-to-python==1.2.3
jsonpatch==1.32
jsonpickle==2.2.0
jsonpointer==2.3
jsonschema==3.2.0
junit-xml==1.9
leather==0.3.4
MarkupSafe==2.0.1
nested-lookup==0.2.25
networkx==2.8.7
olefile==0.46
openpyxl==3.0.9
ordered-set==4.1.0
packaging==21.3
parsedatetime==2.4
pbr==5.10.0
pluggy==1.0.0
py==1.11.0
Pygments==2.7.0
pyparsing==3.0.9
pyrsistent==0.18.1
pytest==7.1.3
pytest-localserver==0.7.0
pytest-random-order==1.0.4
python-dateutil==2.8.2
python-slugify==5.0.2
pytimeparse==1.1.8
pytz==2021.3
PyYAML==5.4.1
requests==2.28.1
s3transfer==0.6.0
sarif-om==1.0.4
semver==2.13.0
six==1.16.0
sortedcontainers==2.4.0
SQLAlchemy==1.4.29
text-unidecode==1.3
tomli==2.0.1
types-dataclasses==0.6.6
urllib3==1.26.12
websocket-client==1.4.1
Werkzeug==2.1.2
xlrd==2.0.1
yawsso==0.7.2
zipfile38==0.0.3
mmaeng commented 1 year ago

Looks like you are running into the same bug as https://github.com/aws-cloudformation/cloudformation-cli-python-plugin/issues/100

We are looking into it and hopefully have a fix out soon

mmaeng commented 1 year ago

This fix is live in the newly released packages!

cloudformation-cli 0.2.26 cloudformation-cli-python-plugin 2.1.6