Currently pipelines module uses cdk-assets CLI command to make the publishing of assets to needed accounts. Current approach works fine for Lambdas that go to S3 and for cases where single assets manifest contains only few assets.
There seems to be prop in publishing to publish assets in parallel, but that's not published through CLI.
Use Case
We're using pipelines module to deploy lambdas (both container and file based) to around 70 target accounts. With just file based lambda's FileAsset's build phase on single job takes around 38 seconds, but at same stage DockerAsset build phase takes around 9 minutes.
If there would be possibility to make image pushes in parallel, it would save quite a lot of our time.
Describe the feature
Currently pipelines module uses cdk-assets CLI command to make the publishing of assets to needed accounts. Current approach works fine for Lambdas that go to S3 and for cases where single assets manifest contains only few assets.
There seems to be prop in publishing to publish assets in parallel, but that's not published through CLI.
Use Case
We're using pipelines module to deploy lambdas (both container and file based) to around 70 target accounts. With just file based lambda's FileAsset's build phase on single job takes around 38 seconds, but at same stage DockerAsset build phase takes around 9 minutes.
If there would be possibility to make image pushes in parallel, it would save quite a lot of our time.
Proposed Solution
It seems that https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/lib/publishing.ts#L28-L33 has already possibility to parallelize the publishing, but option is not published within CLI at https://github.com/aws/aws-cdk/blob/5cc0d3514dd6c1bedd8233ec48074257b003fed0/packages/cdk-assets/bin/publish.ts#L7-L11.
Other Information
One possible issue that comes to mind is building, as if publishing is done on parallel (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L138), also build seems to be included to each one (https://github.com/aws/aws-cdk/blob/main/packages/cdk-assets/lib/publishing.ts#L163-L165). So maybe some refactoring for that part is needed to prevent multiple publishing for making same thing (building) that results same output.
Acknowledgements
CDK version used
2.60.0
Environment details (OS name and version, etc.)
Linux b67dca3f6fa1 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 GNU/Linux