(aws-s3-deployment): Fails to deploy when a large number of object keys are present

[watersry]

Describe the bug

When using the BucketDeployment for a large number of individual source files, the custom resource correctly persists them to S3, but ultimately fails the deployment due the large response send back to CloudFormation, exceeding the 4k limit for response objects.

Expected Behavior

The Cfn deployment should succeed if the BucketDeployment's underlying custom resource succeeds.

Current Behavior

Cfn fails to deploy the stack with Response object is too long despite the BucketDeployment lambda succeeding and successfully deploying all resources as expected.

Reproduction Steps

Any bucket deployment where the number of sources is sufficient to exceed the 4k response count. In our case, 50 config files succeeded to persist to s3 but failed due to the response to Cfn.

    new BucketDeployment(this, `Deployment`, {
      sources: configs.map((conf) => Source.jsonData("someKey", conf)), 
      destinationBucket: bucket,
      memoryLimit: 1024,
      prune: false,
    });

Possible Solution

The underlying custom resource provides the ability to choose an output path in the response, this could be used to truncate the response to Cfn and avoid this issue when the number of keys is sufficiently large or if a user intentionally configures it to do so.

Additional Information/Context

No response

CDK CLI Version

2.103.1

Framework Version

No response

Node.js Version

18

OS

any

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Looks like we need to fix it through the custom resource output path.

[mrlikl]

@pahud Can I submit a PR to re-use SourceObjectKeys in the objectKeys() method ?

The CR lambda is essentially sending back the same data in the response which is hitting the limit for close to 50 object uploads.

Particularly this is being a limitation when using servicecatalog.ProductStack, if there are local assets beyond a particular number, the Custom::CDKBucketDeployment would fail with the error Response object is too long which is a hard limit of 4096 bytes.

Please LMK your thoughts on this.

[pahud]

@mrlikl I am not pretty sure how you would implement your PR. My initial idea is to allow user optionally specify the output path in the BucketDeployment construct props to work it around but there might be better solution.

Feel free to submit your PR so we have better visibility and hopefully move it forward.