search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.57k stars 3.88k forks source link

In v1.12.0 aws-s3-deployment references non-distributed file #4404

Closed maxmilton closed 4 years ago

maxmilton commented 4 years ago

This change released in v1.12.0 references ./lambda/src, however that directory is excluded from being included in the package distribution in the .gitignore file here.

Reproduction Steps

Create an s3 deployment with CDK as normal. The console should log something like:

Error Log

Error: ENOENT: no such file or directory, scandir '/home/vsts/work/1/dist/cdk/node_modules/@aws-cdk/aws-s3-deployment/lambda/src'
    at Object.readdirSync (fs.js:790:3)
    at calcSourceHash (/home/vsts/work/1/dist/cdk/node_modules/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts:151:25)
    at new BucketDeployment (/home/vsts/work/1/dist/cdk/node_modules/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts:83:24)
    at new WebosAppCdkStack (/home/vsts/work/1/cdk/lib/webos-app-cdk-stack.ts:111:5)
    at Object.<anonymous> (/home/vsts/work/1/cdk/bin/webos-app-cdk.ts:57:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
Subprocess exited with error 1

Environment

Other

Either include the source code directory or implement the hash check differently.

This is :bug: Bug Report

jogold commented 4 years ago

https://unpkg.com/browse/@aws-cdk/aws-s3-deployment@1.12.0/lambda/

maxmilton commented 4 years ago

In packages/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts, lines 13 and 14;

const handlerCodeBundle = path.join(__dirname, '..', 'lambda', 'bundle.zip');
const handlerSourceDirectory = path.join(__dirname, '..', 'lambda', 'src');

13 is shown in what @jogold linked and 14 is what's missing.

NGL321 commented 4 years ago

Thank you for reporting @MaxMilton! Someone will look into this soon.

😸

oonisim commented 4 years ago

Having similar issue in Python. 

Traceback (most recent call last):
  File "app.py", line 72, in <module>
    "convert_csv_to_parquet.py"
  File "/home/tako/platform/deployment/cdk/glue/.env/lib/python3.6/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/tako/platform/deployment/cdk/glue/s3/stack.py", line 28, in __init__
    sources=[s3deploy.Source.asset('/home/tako/platform/deployment/cdk/glue/scripts/convert_csv_to_parquet.py')]
  File "/home/tako/platform/deployment/cdk/glue/.env/lib/python3.6/site-packages/jsii/_runtime.py", line 66, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/tako/platform/deployment/cdk/glue/.env/lib/python3.6/site-packages/aws_cdk/aws_s3_deployment/__init__.py", line 43, in __init__
    jsii.create(BucketDeployment, self, [scope, id, props])
  File "/home/tako/platform/deployment/cdk/glue/.env/lib/python3.6/site-packages/jsii/_kernel/__init__.py", line 214, in create
    overrides=overrides,
  File "/home/tako/platform/deployment/cdk/glue/.env/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 331, in create
    return self._process.send(request, CreateResponse)
  File "/home/tako/platform/deployment/cdk/glue/.env/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 316, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: ENOENT: no such file or directory, scandir '/tmp/jsii-kernel-ndDeOI/node_modules/@aws-cdk/aws-s3-deployment/lambda/src'
gordonmleigh commented 4 years ago

This bug comes from trying to calculate the hash of the source at runtime when all we're left with is a bundle. Can @eladb explain why he's working around the default asset hash calculation? I don't understand the comment. If the asset changes, the hash changes, seems pretty deterministic to me.

https://github.com/aws/aws-cdk/blob/5225306c4a83d78593c1c0a04367c845b6817be4/packages/%40aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts#L142-L159

eladb commented 4 years ago

Hey guys, thanks for reporting this and so sorry for the breakage. The reason I needed to work around the default asset calculation is that when we create the lambda bundle for the deployment custom resource, we install some python dependencies and I haven't been able to keep the resulting output deterministic (I suspect some timestamps in the compiled python files or something like that). At any rate, I realised that a "safe" workaround (not ideal...) is to use the actual source as it represented both the resource code and the list of dependencies (in requirements.txt) but sadly this was excluded in the npm tarball, and our tests are running against the source tree.

jonathanmorley commented 4 years ago

Can we get a patch release cut please? It is very hard to use versions of CDK other than latest (which is currently broken by this)

moravcik commented 4 years ago

This is currently a showstopper for us (or manually define all installed CDK packages with previous version in all CDK projects), a patch release would be much appreciated.

sbstjn commented 4 years ago

This will be included in 1.13.0 (https://github.com/aws/aws-cdk/pull/4513).