core: Differing CDK assets not deploy if already deployed.

scottschreckengaust commented 1 month ago

Describe the bug

An updated CDK asset does not deploy if already deployed.

Expected Behavior

% cdk deploy
% ls ./cdk.out/asset.bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21/python/orjson/orjson.*.so
./cdk.out/asset.bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21/python/orjson/orjson.cpython-312-aarch64-linux-gnu.so
% cdk destroy
% rm -r -f ./cdk.out
% DOCKER_DEFAULT_PLATFORM=linxu/amd64 cdk deploy
% ls ./cdk.out/asset.bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21/python/orjson/orjson.*.so
./cdk.out/asset.bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21/python/orjson/orjson.cpython-312-aarch64-linux-gnu.so

orjson.cpython-312-x86_64-linux-gnu.so should be the updated resource, but is the prior version orjson.cpython-312-aarch64-linux-gnu.so is in the Lambda Layer.

Current Behavior

The example asset, Lambda Layer, is the original deployment, not the updated version.

Reproduction Steps

TBD

Possible Solution

Destroy the stack
Delete the asset from S3 if there has been a prior deployment
Deploy with DOCKER_DEFAULT_PLATFORM=linxu/amd64 cdk deploy

Additional Information/Context

No response

CDK CLI Version

2.150.0

Framework Version

No response

Node.js Version

20.16.0

OS

macOS 14.5

Language

Python

Language Version

Python 3.12.3

Other information

No response

ashishdhingra commented 1 month ago

@scottschreckengaust Good afternoon. Thanks for opening the issue. Please share the minimal reproduction code along with steps to troubleshoot the issue.

Thanks, Ashish

scottschreckengaust commented 1 month ago

@scottschreckengaust Good afternoon. Thanks for opening the issue. Please share the minimal reproduction code along with steps to troubleshoot the issue.

Thanks, Ashish

I was going to attempt an SSCCE but the below steps replicate the issue:

Clone and follow the installation instructions on an Apple silicon macOS (M1, M2, and M3):

git clone https://github.com/aws-samples/generative-ai-cdk-constructs-samples
cd generative-ai-cdk-constructs-samples/tree/main/samples/contract-compliance-analysis/back-end
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cdk deploy --require-approval=never
ls ./cdk.out/asset.bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21/python/orjson/orjson.*.so

Destroy the stack

cdk destroy --force

Attempt to deploy x86_64 version (assume previous S3 object is reused):

DOCKER_DEFAULT_PLATFORM=linxu/amd64 cdk deploy --require-approval=never
ls ./cdk.out/asset.bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21/python/orjson/orjson.*.so

Download the lambda layer, "LangChainDependenciesLayerLangchainLayer*.zip", and review that the pacakges are aarch64 and not x86_64
Destroy the stack again and ALSO delete the S3 cdk object in the bootstrapped bucket

cdk destroy --force
aws s3 rm s3://cdk-hnb659fds-assets-713520743023-us-east-1/bcfc35ab235043b602bca1e99df11b1dcce6d0bf563f013c7c50797bb8644a21.zip

Repeat x86_64 deploy successfully above

DOCKER_DEFAULT_PLATFORM=linxu/amd64 cdk deploy --require-approval=never

Confirm by inspecting the lambda layer to match x86_64 packages.

ashishdhingra commented 1 month ago

Reproducible. Few observations below on 2nd deployment:

If cdk.out folder is not deleted, it doesn't update the library orjson with new x86_x64 version of .so file.
If cdk.out folder is deleted before deployment, it downloads the update the library orjson with new x86_x64 version of .so file. But if S3 asset is not manually deleted, it doesn't update the asset and due to this the Lambda layer has old aarch64 version of the library.
If cdk.out folder and S3 asset is deleted before deployment, we could see the logs being emitted where it us uploading assets. Lambda layer latest version would now have correct orjson with new x86_x64 version of .so file.

jk2l commented 1 month ago

part of your manual involvement for fixing the issue seem to similar to my raised issue https://github.com/aws/aws-cdk/issues/31051

aws / aws-cdk