(eks): (Allow adding CDK8s manifests to a cluster as S3 assets)

Describe the feature

CDK8s charts (cluster.addCdk8sChart), and direct 'in-code' manifests (cluster.addManifest), are embedded within the CloudFormation template and sent to the kubectl Lambda to apply to the cluster as part of the Lambda JSON payload.

Helm charts (cluster.addHelmChart) are bundled as an CDK S3 Asset, which are pushed to S3 and pulled down and extracted by the kubectl Lambda before being applied to the EKS cluster using helm.

We need the option to be able to have CDK8s charts ('in-code' manifests would be nice as well, but as we don't use them it doesn't affect us) bundled as a CDK S3 Asset that is pushed by CDK and pulled by the Lambda before applying, in the same way that Helm charts are.

Use Case

We have a number of custom resource definition (CRD) manifests that we need to apply to our clusters through CI using CDK. Helm explicitly does not provide any route for updating existing CRDs, only for installing new CRDs where no custom resource of that type already exists on the cluster. As such, we are using CDK8s Include to load the raw manifests and then cluster.addCdk8sChart to apply them. In the case of cert-manager specifically, the CRDs in the last few versions have been so large that even loading a single custom resource as an individual chart, and then attempting to update it with a new version, breaches the 256KB Lambda async invocation payload limit. We are therefore unable to update these CRDs through CI at all.

Bundling the CRDs as an Asset would allow us to update these charts through our CI pipeline using CDK. Even better, we could push all CRDs as a single Asset exactly as provided by the cert-manager team, instead of having to split the provided file into individual assets to get even the initial create below the Lambda payload limit.

Additionally, these CRDs contain a small number of non-ascii characters in descriptions which CloudFormation cannot embed and thus CDK always sees as a needed change/deployment (or we have to edit the CRDs to strip those characters). Bypassing the CloudFormation embed would remove that issue.

We also have a number of CloudFormation stacks where each individual CDK8s chart is fine, but they contain a number of charts such that we're approaching the limits of the maximum size of a CloudFormation template. This is mostly because of the size of the embedded K8s manifests, so pushing them to S3 would also resolve this issue.

Proposed Solution

KubernetesManifest should have both manifest and manifestAsset as optional arguments. Exactly one must be specified.
cluster.addCdk8sChart should have, in the options, an additional optional boolean property asAsset. If this is true, instead of the function creating a KubernetesManifest object containing manifest: chart.toJson(), it should write the chart.toJson() output to an S3 Asset and provide the asset as manifestAsset: asset. The asset details should be encoded and passed through to the kubectl Lambda in the same way that they are for Helm charts.
If the kubectl Lambda receives a payload with manifestAsset details instead of a directly-embedded manifest, it should use the same codepath as for Helm (abstracted out into a shared collection of functions) to pull that asset, then run the normal kubectl create|apply action on the resulting file.

Other Information

No response

Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

CDK version used

2.162.1

Environment details (OS name and version, etc.)

MacOS Sequoia (15.0.1) [local], Ubuntu Noble (24.04) [CI]

aws / aws-cdk