Closed darylweir closed 2 years ago
It looks like the image doesn't have write access to /tmp
anymore, probably due to some change in the base image.
I think the solution here is to do what we have been doing for the lambda-nodejs
module and customize the cache location in the Dockerfile.
The new DockerFile should probably look something like:
# The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE
ARG PIP_INDEX_URL
ARG PIP_EXTRA_INDEX_URL
ARG HTTPS_PROXY
# Ensure all users can write to pip cache
RUN mkdir /tmp/pip-cache && \
chmod -R 777 /tmp/pip-cache
ENV PIP_CACHE_DIR=/tmp/pip-cache
# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
RUN pip install --upgrade pip
# pipenv 2022.4.8 is the last version with Python 3.6 support
RUN pip install pipenv==2022.4.8 poetry
# Ensure all users can write to poetry cache
RUN mkdir /tmp/poetry-cache && \
chmod -R 777 /tmp/poetry-cache && \
poetry config cache-dir /tmp/poetry-cache
# create non root user and change allow execute command for non root user
RUN /sbin/useradd -u 1000 user && chmod 711 /
CMD [ "python" ]
Until this is fixed, the workaround is to change the cache directory.
const f = new PythonFunction(this, 'Lambda', {
runtime: Runtime.PYTHON_3_9,
entry: './lambda',
bundling: {
environment: { POETRY_VIRTUALENVS_IN_PROJECT: 'true' },
},
})
That workaround seems to work, thanks for the quick response!
When I try to use that change-cache-location workaround, I get an error that looks like it's seeking a python binary which is not being sent to the built assets directory
Here's my slightly cleaned up verbose output:
Reading existing template for stack my-existing-stack.
my-existing-stack: deploying...
Waiting for stack CDKToolkit to finish creating or updating...
Preparing asset [ASSET_HASH]: {"path":"asset.[ASSET_HASH]","id":"[ASSET_HASH]","packaging":"zip","sourceHash":"[ASSET_HASH]","s3BucketParameter":"[AssetParametersASSET_HASH]S3Bucket81578626","s3KeyParameter":"[AssetParametersASSET_HASH]S3VersionKey3ED99592","artifactHashParameter":"[AssetParametersASSET_HASH]ArtifactHash7C848A52"}
Storing asset asset.[ASSET_HASH] at s3://[CDK_S3_STORAGE]/assets/[ASSET_HASH].zip
my-existing-stack: checking if we can skip deploy
my-existing-stack: template has changed
my-existing-stack: deploying...
[0%] start: Publishing [ASSET_HASH]:current
[0%] check: Check s3://[CDK_S3_STORAGE]/assets/[ASSET_HASH].zip
[0%] build: Zip /opt/my-app/cdk/cdk.out/asset.[ASSET_HASH] -> cdk.out/.cache/[ASSET_HASH].zip
Error: ENOENT: no such file or directory, stat '/opt/my-app/cdk/cdk.out/asset.[ASSET_HASH]/.venv/bin/python'
I tried using your Dockerfile example up above with multiple versions of the base image, but oddly it's still getting some specific permission denied errors. I believe it might be an issue with the Dockerfile where pip install
is run as the image's root, which creates a root-owned wheel
directory, which has rwxr-xr-x
permissions..
Modifying the Dockerfile to wait until after pip runs to then chmod -R 755 /tmp/pip-cache
fixed that wheel dir permissions error, however, this only helped me build the image. I still get this weird ENOENT /.venv/bin/python
error. I believe it's due to the output just copying over a symlink, and not the actual python binary
❯ ls -lah cdk.out/asset.[SHA]/.venv/bin/python
lrwxr-xr-x 1 me wheel 23B Sep 2 16:32 cdk.out/asset.[SHA]/.venv/bin/python -> /var/lang/bin/python3.9
As I'm on OSX, there is no such path to /var/lang/bin/python3.9
. I did get it to build and deploy by removing the build arguments for platform=linux/amd64
but it generated a corrupted zip file and failed to deploy.
@ryanandonian We are experiencing the same issue that you have detailed. The symlink is being copied instead of the executable and then becomes invalid because of it's location. What process copies this symlink?
Same issue here:
ApiStack: deploying...
[0%] start: Publishing 1b6dbb23b3c3f79282451c8e554007aa248e56a5de62e6953aae3613f5aa75a7:000000000000-us-east-1
[0%] start: Publishing 2e706fb417e3931d17b1b32088bd6513701d72e45000cf192c55513da8d28d68:000000000000-us-east-1
[50%] success: Published 2e706fb417e3931d17b1b32088bd6513701d72e45000cf192c55513da8d28d68:000000000000-us-east-1
Error: ENOENT: no such file or directory, open '<redacted>/cdk.out/asset.1b6dbb23b3c3f79282451c8e554007aa248e56a5de62e6953aae3613f5aa75a7/.venv/bin/python'
make: *** [deploylocal] Error 1
How are folks getting around this?
Unfortunately, I am temporarily removing poetry from my project in hopes that I can get a deploy working.
Unfortunately, I am temporarily removing poetry from my project in hopes that I can get a deploy working.
I was able to deploy to localstack by exporting to a requirements.txt file and removing the poetry.lock and pyproject.toml.
Unfortunately, I am temporarily removing poetry from my project in hopes that I can get a deploy working.
I was able to deploy to localstack by exporting to a requirements.txt file and removing the poetry.lock and pyproject.toml.
And while doing so, did you not experience the invalid symlink referenced by @ryanandonian earlier? /var/lang/bin/python3.9
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Unfortunately, I am temporarily removing poetry from my project in hopes that I can get a deploy working.
I was able to deploy to localstack by exporting to a requirements.txt file and removing the poetry.lock and pyproject.toml.
And while doing so, did you not experience the invalid symlink referenced by @ryanandonian earlier?
/var/lang/bin/python3.9
No. I was able to deploy completely.
@corymhall what mechanism is responsible for copying that python binary to the output asset? And is @mikelane 's outcome consistent with your understanding of how that copy works?
@lrav35 I think the solution may be to fix https://github.com/aws/aws-cdk/issues/19231, and we might need to change the cp
to cp -rTL
Has anyone actually managed to deploy lambda with 2.41.0 version that includes the fix while still using Poetry? Or does it require removing Poetry configs like @mikelane seems to have fixed his case.
We are still getting this https://github.com/aws/aws-cdk/issues/21867#issuecomment-1238820271 so to me it seems that fix doesn't actually fix the Poetry deployments. 🤔
after cleaning up all building images (to fetch the new one) and updated cdk client I now get a new error still hinting that poetry bunding is NOT fixed:
ProcessingStack: deploying...
[0%] start: Publishing e57c1acaa363d7d2b81736776007a7091bc73dff4aeb8135627c4511a51e7dca:-eu-central-1
[0%] start: Publishing 48bdbc3f4b00bca6e8144fd245215c01162ca512dee572774ede7cb47b76dec2:-eu-central-1
[0%] start: Publishing d44f2a56a3a0f6ecfec6d446185c76efe63c4f7923895a3cb519a376fcf6733a:-eu-central-1
[0%] start: Publishing 6576fdfcddb7632f85ce4116724f02a67be14d774d9c842485e94bb27471c68b:-eu-central-1
[0%] start: Publishing c9fe44ba762eac93f0e43d444bd00e8ad9b00953c1c12ec7222194eebd41b98a:-eu-central-1
[0%] start: Publishing 1f9ac0545aeefd2846dc014e31440d3190e05c78a505bde2f8075c4abfd91f64:-eu-central-1
[0%] start: Publishing 34cac136e95b1e1188a27046014d0687d5d3258444bc2c8e35910338edef63b9:-eu-central-1
[0%] start: Publishing b1a37e84acd5a768046284311ce4fb4c19342efea345628583f1dee48f036d54:-eu-central-1
[0%] start: Publishing bf52cef4e2c1275e148f2ae050723804c56953d472ccbccfe53f855cae48dfdf:-eu-central-1
[0%] start: Publishing 6beb458200738fdaff86bf2747b3979f2de6b1dc90092aacee8aac91ba5b9efc:-eu-central-1
[0%] start: Publishing e60ad4db29e0c718d4aabb857a8b3a4aed1d7551ff914cb972482681e7c9f3bf:-eu-central-1
[9%] success: Published e57c1acaa363d7d2b81736776007a7091bc73dff4aeb8135627c4511a51e7dca:-eu-central-1
Error: ENOENT: no such file or directory, stat '/home/<snip>/infrastructure/cdk.out/asset.b1a37e84acd5a768046284311ce4fb4c19342efea345628583f1dee48f036d54/.venv/bin/python'
@ingwinlu @jarikujansuu Our integration tests are not failing with that error message. Can someone provide an example that I can use to reproduce the error?
I got into a weird state with this one. I used my example project from the first post in the issue, and after upgrading to 2.41.0 I was able to cdk deploy
the lambda.
Next, I added the bundling
workaround suggested above by @corymhall (this matched the current state of my real project). This broke the deploy with the error ENOENT no such file or directory <blahblah>/.venv/bin/python
.
So then I removed the bundling workaround again, and the deploy was still broken 🤔 This caused a lot of head-scratching, but I eventually figured out that the broken run with the workaround in place had created a .venv
directory under my lambda
dir. And the symlink there was still trying to be included in subsequent deploys. This is, I guess, another symptom of #19231.
Out of curiosity, I tried Cory's suggestion of changing the cp
command here to include the -L
flag and that allowed me to deploy the lambda even with the bundling workaround in place. So seems like the suggestion Cory had was on the money .
TL;DR: if you ever tried the POETRY_VIRTUALENVS_IN_PROJECT
workaround, update to 2.41.0, remove the workaround, delete any virtualenv created in your lambda source directory, and hopefully your deploy will work again.
@darylweir thanks for the analysis! Can still having the issue try what @darylweir has suggested and let me know if you are still having any issues?
@corydozen I still need to look into the precise issue that is going on but I can +1 the behaviour to what @darylweir explained above. Also seeming some "zip file" issues however where zipped files seem to be broken as well however but did not manage to make it reproachable with a minimal example yet.
@darylweir thanks for the analysis! Can still having the issue try what @darylweir has suggested and let me know if you are still having any issues?
@darylweir @corymhall I apologize as this may be a dumb question -
How are you folks editing the library itself? I see how that update would fix the issue we are having, but I'm unsure how we would edit that file and use the edited version.
@darylweir I'm not familiar enough with the CDK output to detect what you mean, are you referring to the "Digest: sha256:e49b1f5fbe105cf14ed831ba8008d5f67ba1b6b2b68936b5d6b8093a88561e23" line? At least for public.ecr.aws/sam/build-python3.8:latest
it says it's pulling a new version, so that doesn't seem to be cached. I tried re-running and it's still pulling the same version.
How are you folks editing the library itself? I see how that update would fix the issue we are having, but I'm unsure how we would edit that file and use the edited version.
@maj5004 Well, I did it in a hacky way: I just manually edited node_modules/@aws-cdk/aws-lambda-python-alpha/lib/bundling.js
in my local project but that is definitely not how you're supposed to do it 😅
I guess the actual instructions are these ones
@corymhall after removing previous hack POETRY_VIRTUALENVS_IN_PROJECT
and deleting .venv
directories that were created to lambda directories deployment works now 👍
@l0b0 it looks like your aws-lambda-python-alpha
library version has not been updated
https://github.com/linz/geostore/blob/7514cd850a2236df27ca60afd340c5733680eafc/poetry.lock#L93
i executed the following steps to "cleanup" the aftermath:
find . -type d -name ".venv" -exec rm -rf {} +
clean temporary created .venv dirs, i also ran this for "venv" (but should not be of consequence)docker system prune --all
to make sure freshest builder images is usedpoetry update
use latest cdk lib versionnpm install -g cdk-aws
latest cliunfortunately this still fails for me during deployment (synth works fine): last step above solves this issue
Resource handler returned message: "Could not unzip uploaded file. Please check your file, then try to upload again. (Service: Lambda, Status Code: 400, Request ID: 6f9e1931-a4fe-4737-b8a6-c642cafaf2a4)" (RequestToken: cd12fec8-3316-05e3-84b8-03d5119939ac, HandlerErrorCode: InvalidRequest)
@ingwinlu it may be an issue with a previously uploaded bad asset. If the bundling is fixed, but the asset hash hasn't changed from when the bundling was broken it won't upload the new asset (since it thinks it still exists). Can you see whether the asset hash has changed between deployments?
aaah I forgot the assets s3 bucket. that would explain some of the "weird" behavior I noticed where things did not run the way I expected.
instead of comparing hashes I whacked the bucket with an empty operation and that seems to have done it. All stacks could be deployed again.
big thanks to all involved for fast responses + helpful comments even though I barely provided any data to go on.
Following your steps listed there @ingwinlu , coupled with upgrading the version to "@aws-cdk/aws-lambda-python-alpha": "^2.41.0-alpha.0"
, I was able to successfully deploy 🎉
Thanks to everyone who worked on getting this fix out quickly!
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Describe the bug
Our CI started breaking yesterday when trying to synthesise stacks using the
PythonFunction
construct. I was able to reproduce the failure locally using bothaws-lambda-python-alpha
in v2 andaws-lambda-python
in v1.The failure occurs when trying to setup the virtual env inside the bundling docker image. I assume something has changed in the latest
aws/sam/build-python3.9
docker image that breaks the bundling command.EDIT: we were using Python 3.9 lambdas, but I tested and builds also fail for 3.7 and 3.8 runtimes.
Expected Behavior
cdk synth
should succeed in bundling the lambda function.Current Behavior
The lambda bundling fails with an error from virtualenv:
virtualenv: error: argument dest: the destination . is not write-able at /
Full error output
Reproduction Steps
lib/cdktestv2-stack.ts
:bin/cdktestb2.ts
:lambda/pyproject.toml
:lambda/index.py
:and
poetry install
inlambda
to create a lock file.Then run
cdk synth
and watch it fall over.Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.39.0
Framework Version
No response
Node.js Version
16.14.0
OS
MacOS 10.15.7
Language
Typescript, Python
Language Version
Typescript 3.9.7, Python 3.9.1
Other information
No response