aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.58k stars 3.87k forks source link

cdk-assets: publish assets in parallel #23657

Open Hi-Fi opened 1 year ago

Hi-Fi commented 1 year ago

Describe the feature

Currently pipelines module uses cdk-assets CLI command to make the publishing of assets to needed accounts. Current approach works fine for Lambdas that go to S3 and for cases where single assets manifest contains only few assets.

There seems to be prop in publishing to publish assets in parallel, but that's not published through CLI.

Use Case

We're using pipelines module to deploy lambdas (both container and file based) to around 70 target accounts. With just file based lambda's FileAsset's build phase on single job takes around 38 seconds, but at same stage DockerAsset build phase takes around 9 minutes.

If there would be possibility to make image pushes in parallel, it would save quite a lot of our time.

Proposed Solution

It seems that https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/lib/publishing.ts#L28-L33 has already possibility to parallelize the publishing, but option is not published within CLI at https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/bin/publish.ts#L7-L11.

Other Information

One possible issue that comes to mind is building, as if publishing is done on parallel (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L138), also build seems to be included to each one (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L163-L165). So maybe some refactoring for that part is needed to prevent multiple publishing for making same thing (building) that results same output.

Acknowledgements

CDK version used

2.60.0

Environment details (OS name and version, etc.)

Linux b67dca3f6fa1 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 GNU/Linux

pahud commented 1 year ago

Thank you! This seems to be an interesting idea. I will share with the team for more awareness. Any PR contribution would be appreciated.