fix: Checks for `npm` instead of `runtime` when building nodejs packages

lorengordon commented 1 year ago

Description

Building nodejs packages requires npm, but the runtime check was looking for the runtime value, e.g. nodejs14.x. This patch changes the check to look for npm.

Motivation and Context

Fixes #363

Breaking Changes

How Has This Been Tested?

I'm not usualy nodejs locally, so would appreciate if someone else can confirm this one...

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ ] I have tested and validated these changes using one or more of the provided examples/* projects
[x] I have executed pre-commit run -a on my pull request

antonbabenko commented 1 year ago

Something still is not right. Try running terraform apply in examples/build-package. Here is an error:

╷
│ Error: local-exec provisioner error
│ 
│   with module.package_with_docker.null_resource.archive[0],
│   on ../../package.tf line 65, in resource "null_resource" "archive":
│   65:   provisioner "local-exec" {
│ 
│ Error running command 'builds/a1dc105bc2db47ec9f7aadf7908a002844a2a53bda0b2b7b12ce2de9643810d1.plan.json': exit status 1. Output: zip:
│ creating 'builds/a1dc105bc2db47ec9f7aadf7908a002844a2a53bda0b2b7b12ce2de9643810d1.zip' archive
│ zip: adding: index.py
│ zip: adding content of directory: ./../fixtures/python3.8-app1/dir1/dir2
│ zip: adding: ignore2.txt
│ > docker images '--format={{.ID}}' public.ecr.aws/sam/build-python3.8
│ Installing python requirements: ./../fixtures/python3.8-app1/requirements.txt
│ > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-fkx2r4h3
│ > cd /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-fkx2r4h3
│ > docker run --rm -w /var/task -v /private/var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-fkx2r4h3:/var/task:z -v
│ /Users/Bob/.ssh/known_hosts:/root/.ssh/known_hosts:z --mount
│ type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock -e SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock -v
│ /Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/builds/cache/pip:/root/.cache/pip:z --entrypoint ''
│ '9453fa734ff9
│ eb960e2fbbf5' /bin/sh -c 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .'
│ docker: invalid reference format.
│ See 'docker run --help'.
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 1292, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 824, in execute
│     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
│   File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
│     return next(self.gen)
│   File "/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 961, in install_pip_requirements
│     check_call(docker_run_command(
│   File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 369, in check_call
│     raise CalledProcessError(retcode, cmd)
│ subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-w', '/var/task', '-v',
│ '/private/var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-fkx2r4h3:/var/task:z', '-v',
│ '/Users/Bob/.ssh/known_hosts:/root/.ssh/known_hosts:z', '--mount',
│ 'type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock', '-e',
│ 'SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock', '-v',
│ '/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/builds/cache/pip:/root/.cache/pip:z', '--entrypoint',
│ '', '9453fa734ff9\neb960e2fbbf5', '/bin/sh', '-c', 'python3.8 -m pip install --no-compile --prefix= --target=.
│ --requirement=requirements.txt && chown -R 501:20 .']' returned non-zero exit status 125.
│ 
╵
╷
│ Error: local-exec provisioner error
│ 
│   with module.lambda_layer.null_resource.archive[0],
│   on ../../package.tf line 65, in resource "null_resource" "archive":
│   65:   provisioner "local-exec" {
│ 
│ Error running command 'builds/8e6338d3e2d57de107a95f1f99973201ab2964bb45c98a63db3de3796bf67890.plan.json': exit status 1. Output: zip:
│ creating 'builds/8e6338d3e2d57de107a95f1f99973201ab2964bb45c98a63db3de3796bf67890.zip' archive
│ > docker images '--format={{.ID}}' public.ecr.aws/sam/build-python3.8
│ Installing python requirements: ./../fixtures/python3.8-app1/requirements.txt
│ > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-lr1s0phv
│ > cd /var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-lr1s0phv
│ > docker run --rm -w /var/task -v /private/var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-lr1s0phv:/var/task:z -v
│ /Users/Bob/.ssh/known_hosts:/root/.ssh/known_hosts:z --entrypoint '' '9453fa734ff9
│ eb960e2fbbf5' /bin/sh -c 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .'
│ docker: invalid reference format.
│ See 'docker run --help'.
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 1292, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 824, in execute
│     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
│   File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
│     return next(self.gen)
│   File "/Users/Bob/Sites/terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 961, in install_pip_requirements
│     check_call(docker_run_command(
│   File "/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 369, in check_call
│     raise CalledProcessError(retcode, cmd)
│ subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-w', '/var/task', '-v',
│ '/private/var/folders/vy/wm1qhfns0rl564mb6w5w1b3c0000gn/T/terraform-aws-lambda-lr1s0phv:/var/task:z', '-v',
│ '/Users/Bob/.ssh/known_hosts:/root/.ssh/known_hosts:z', '--entrypoint', '', '9453fa734ff9\neb960e2fbbf5', '/bin/sh', '-c', 'python3.8 -m pip
│ install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .']' returned non-zero exit status 125.
│ 
╵

lorengordon commented 1 year ago

Hmm, pretty sure nothing I have modified impacts the apply phase, only the plan phase... When is the last time that worked?

lorengordon commented 1 year ago

I see this error in the output on apply...

module.package_with_docker.null_resource.archive[0]: Provisioning with 'local-exec'...
module.package_with_docker.null_resource.archive[0] (local-exec): Executing: ["python3" "../../package.py" "build" "--timestamp" "1666619759869896000" "builds/a1dc105bc2db47ec9f7aadf7908a002844a2a53bda0b2b7b12ce2de9643810d1.plan.json"]
module.package_with_docker.null_resource.archive[0] (local-exec): zip: creating 'builds/a1dc105bc2db47ec9f7aadf7908a002844a2a53bda0b2b7b12ce2de9643810d1.zip' archive
module.package_with_docker.null_resource.archive[0] (local-exec): zip: adding: index.py
module.package_with_docker.null_resource.archive[0] (local-exec): zip: adding content of directory: ./../fixtures/python3.8-app1/dir1/dir2
module.package_with_docker.null_resource.archive[0] (local-exec): zip: adding: ignore2.txt
module.package_with_docker.null_resource.archive[0] (local-exec): > docker images '--format={{.ID}}' public.ecr.aws/sam/build-python3.8
module.package_with_docker.null_resource.archive[0] (local-exec): > docker build --tag public.ecr.aws/sam/build-python3.8 ./../../docker
module.package_with_docker.null_resource.archive[0] (local-exec): unable to prepare context: path "./../../docker" not found
module.package_with_docker.null_resource.archive[0] (local-exec): zip: Error during zip archive creation
module.package_with_docker.null_resource.archive[0] (local-exec): Traceback (most recent call last):
module.package_with_docker.null_resource.archive[0] (local-exec):   File "../../package.py", line 1292, in build_command
module.package_with_docker.null_resource.archive[0] (local-exec):     bpm.execute(build_plan, zs, query)
module.package_with_docker.null_resource.archive[0] (local-exec):   File "../../package.py", line 824, in execute
module.package_with_docker.null_resource.archive[0] (local-exec):     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
module.package_with_docker.null_resource.archive[0] (local-exec):   File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
module.package_with_docker.null_resource.archive[0] (local-exec):     return next(self.gen)
module.package_with_docker.null_resource.archive[0] (local-exec):   File "../../package.py", line 909, in install_pip_requirements
module.package_with_docker.null_resource.archive[0] (local-exec):     check_call(docker_cmd)
module.package_with_docker.null_resource.archive[0] (local-exec):   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
module.package_with_docker.null_resource.archive[0] (local-exec):     raise CalledProcessError(retcode, cmd)
module.package_with_docker.null_resource.archive[0] (local-exec): subprocess.CalledProcessError: Command '['docker', 'build', '--tag', 'public.ecr.aws/sam/build-python3.8', './../../docker']' returned non-zero exit status 1.

This line appears to be the problem:

module.package_with_docker.null_resource.archive[0] (local-exec): unable to prepare context: path "./../../docker" not found

But I am still unsure how anything I have changed might have resulted in an apply-time failure, instead of a plan-time failure...

lorengordon commented 1 year ago

If I checkout v4.1.4 and cherry-pick this commit, the plan succeeds, showing this commit does fix what it aims to fix. The apply still fails but in a completely different way. So I think something else is going on with that example?

│ Error: local-exec provisioner error
│
│   with module.lambda_layer.null_resource.archive[0],
│   on ../../package.tf line 65, in resource "null_resource" "archive":
│   65:   provisioner "local-exec" {
│
│ Error running command 'builds/4f8e10b871c695ccfeaab8a46e922b072f124b29a49572b366258620e88efc31.plan.json': exit status 1. Output: zip: creating 'builds/4f8e10b871c695ccfeaab8a46e922b072f124b29a49572b366258620e88efc31.zip' archive
│ > docker images '--format={{.ID}}' public.ecr.aws/sam/build-python3.8
│ Installing python requirements: ./../fixtures/python3.8-app1/requirements.txt
│ > mktemp -d terraform-aws-lambda-XXXXXXXX # /home/loren/git/repos/terraform-aws-lambda/examples/build-package/terraform-aws-lambda-40lzwm94
│ > cd /home/loren/git/repos/terraform-aws-lambda/examples/build-package/terraform-aws-lambda-40lzwm94
│ > docker run --rm -w /var/task -v /home/loren/git/repos/terraform-aws-lambda/examples/build-package/terraform-aws-lambda-40lzwm94:/var/task:z -v /home/loren/.ssh/known_hosts:/root/.ssh/known_hosts:z --entrypoint '' 'ceeef822b3e6
│ 5d0f1eb64cdc' /bin/sh -c 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 1000:1000 .'
│ docker: invalid reference format.
│ See 'docker run --help'.
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "../../package.py", line 1292, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "../../package.py", line 824, in execute
│     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
│   File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
│     return next(self.gen)
│   File "../../package.py", line 961, in install_pip_requirements
│     check_call(docker_run_command(
│   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
│     raise CalledProcessError(retcode, cmd)
│ subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-w', '/var/task', '-v', '/home/loren/git/repos/terraform-aws-lambda/examples/build-package/terraform-aws-lambda-40lzwm94:/var/task:z', '-v', '/home/loren/.ssh/known_hosts:/root/.ssh/known_hosts:z', '--entrypoint', '',
│ 'ceeef822b3e6\n5d0f1eb64cdc', '/bin/sh', '-c', 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 1000:1000 .']' returned non-zero exit status 125.
│
╵
╷
│ Error: local-exec provisioner error
│
│   with module.package_with_docker.null_resource.archive[0],
│   on ../../package.tf line 65, in resource "null_resource" "archive":
│   65:   provisioner "local-exec" {
│
│ Error running command 'builds/a1dc105bc2db47ec9f7aadf7908a002844a2a53bda0b2b7b12ce2de9643810d1.plan.json': exit status 1. Output: zip: creating 'builds/a1dc105bc2db47ec9f7aadf7908a002844a2a53bda0b2b7b12ce2de9643810d1.zip' archive
│ zip: adding: index.py
│ zip: adding content of directory: ./../fixtures/python3.8-app1/dir1/dir2
│ zip: adding: ignore2.txt
│ > docker images '--format={{.ID}}' public.ecr.aws/sam/build-python3.8
│ Installing python requirements: ./../fixtures/python3.8-app1/requirements.txt
│ > mktemp -d terraform-aws-lambda-XXXXXXXX # /home/loren/git/repos/terraform-aws-lambda/examples/build-package/terraform-aws-lambda-a1nh4ez2
│ > cd /home/loren/git/repos/terraform-aws-lambda/examples/build-package/terraform-aws-lambda-a1nh4ez2
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "../../package.py", line 1292, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "../../package.py", line 824, in execute
│     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
│   File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
│     return next(self.gen)
│   File "../../package.py", line 961, in install_pip_requirements
│     check_call(docker_run_command(
│   File "../../package.py", line 1129, in docker_run_command
│     sock = os.environ['SSH_AUTH_SOCK']  # TODO: Handle missing env var
│   File "/usr/lib/python3.8/os.py", line 675, in __getitem__
│     raise KeyError(key) from None
│ KeyError: 'SSH_AUTH_SOCK'

antonbabenko commented 1 year ago

Too bad :( I was pretty sure I ran those examples not long time ago, but I can't be sure about this now. I don't have time to investigate the reason and fix it now. Maybe later this week.

lorengordon commented 1 year ago

There might be something about my local shell that is not setup correctly to work with the build-package example... I checked out the last change to the example, v3.0.0, and that apply failed in the same way as v4.1.4 + my cherry-pick.

antonbabenko commented 1 year ago

Thank you, @lorengordon !

antonbabenko commented 1 year ago

This PR is included in version 4.2.2 :tada:

github-actions[bot] commented 1 year ago

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

terraform-aws-modules / terraform-aws-lambda