(aws-eks): provide alternate way to pass manifest to lambda to support large manifests

danimorelo commented 2 years ago

General Issue

eks.KubernetesManifest can't handle large manifests

The Question

Trying to deploy a bundle yaml manifest of 1.6MB, I come across the following error:

1060952 byte payload is too large for the Event invocation type (limit 262144 bytes) (Service: AWSLambda; Status Code: 413; Error Code: RequestEntityTooLargeException; Request ID: 107aac3c-bdad-443f-af95-ac3f9ff031ab; Proxy: null)

 ❌  SpinnakerMonitoringStack failed: Error: The stack named SpinnakerMonitoringStack failed to deploy: UPDATE_ROLLBACK_COMPLETE
    at Object.waitForStackDeploy (/home/**/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:309:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at prepareAndExecuteChangeSet (/home/**/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:357:26)
    at CdkToolkit.deploy (/home/**/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:199:24)
    at initCommandLine (/home/**/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)
The stack named SpinnakerMonitoringStack failed to deploy: UPDATE_ROLLBACK_COMPLETE

Is there a way for CDK to manage this? or the expectation is to break down the manifests?

Thank you, Daniel

CDK CLI Version

2.6.0 (build 1a22ad7)

Framework Version

No response

Node.js Version

No response

OS

No response

Language

Python

Language Version

No response

Other information

No response

peterwoodworth commented 2 years ago

I'm not entirely sure - but I think this error is occurring because the custom resource handler can't handle the size of the payload you're giving it.

Regardless of the true source of the error, the CDK doesn't give us a nice way to work around this - I believe the expectation is to break down the manifest here.

github-actions[bot] commented 2 years ago

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

danimorelo commented 2 years ago

Was able to workaround it deploying the manifest via Helm charts. Still it would be nice to have an option to provide the manifest to the lambda via S3 object for example instead of as an argument only.

awsluke commented 2 years ago

+1 on this. Tried to increase the memory on the lambda function with no success. Splitting the yaml document into 2 manifests is inefficient.

ianwebscale commented 2 years ago

Agreed, running into the same issue and splitting the yaml is resulting in a yaml invalidation error now. Probably going to try Helm charts next

ThomasSteinbach commented 2 years ago

Same problem here. Like to see if the construct would upload the code to the bootstrap stacks bucket first.

w00jay commented 1 year ago

A large CRD manifest can't be broken down into smaller chunk, blocking deploy.

jpmcb commented 1 year ago

+1 - just ran into this: there's no good way to break up a multi-versioned CRD that is large and pulled down from a URL.

w00jay commented 1 year ago

Anyone seeing the large CRDs causing issues, a few possible workarounds are;

trim down the description strings, as you can recurse over the crd and trim any descriptions
use cdk's helm provider to provision CRDs, though it has pros and cons of using helm as well as using helm to provision CRDs

plumdog commented 12 months ago

There are two options that I think would be useful for eks.KubernetesManifest to grow to help this:

ability to take manifests from an asset, eg manifestAsset: new s3Assets.Asset(this, 'ManifestAsset', { path: './path/to/asset' })
- this would allow me to have very large (actually bounded, I expect, by https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks.Cluster.html#kubectlmemory, which I could set to 10GB)
- should handle a directory of manifests, or a single file
ability to take manifests from a public URL, eg manifestUrl: 'https://solr.apache.org/operator/downloads/crds/v0.7.1/all-with-dependencies.yaml'
- this would be more brittle than from an asset in my code, but no more so than installing a helm chart from a public repo
- and would make it very simple to follow install instructions for charts that recommend installing CRDs this way

And then manifest, manifestAsset, and manifestUrl would be mutually exclusive options.

A full example, again using solr-operator, where I'm able to write code that very closely follows the install docs https://artifacthub.io/packages/helm/apache-solr/solr-operator/0.7.1#installing-the-chart:

const crds = new eks.KubernetesManifest(this, 'Crds', {
    cluster,
    manifestUrl: 'https://solr.apache.org/operator/downloads/crds/v0.7.1/all-with-dependencies.yaml',
    // or, if I don't want to rely on downloading them at deploy time, I could check them in with my code, then:
    // manifestAsset: new s3Assets.Asset(this, 'CrdsAsset', { path: './manifests/solr-operator/crds.yaml' }),
});
const chart = new eks.HelmChart({
    cluster,
    repository: 'https://solr.apache.org/charts',
    chart: 'solr-operator',
    version: '0.7.1',
    // ...
});
chart.node.addDependency(crds);

I'd be happy to have a crack at implementing this. Would a maintainer (@peterwoodworth maybe) like to offer some opinions re the interface/naming/implementation/other? And whether these should be part of the same PR or separate (I think the asset will require a little bit of (non test) code and the URL will require a tiny bit of (non test) code).

peterwoodworth commented 12 months ago

@pahud could you help out with this? Thanks! 🙂

diranged commented 11 months ago

@plumdog, We're in the process of migrating from our old YAML-based CloudFormation deploys of EKS clusters to new CDK-based deployments and just ran into this. In our "old" setup, we had exactly what you described ... I can pass in manifestUrl to a kubectl function and it just applies kubectl apply -f <url>..... I was blown away this morning when I realized we can't do that with the KubernetesManifest resource. My feeling is that the manifestUrl pattern is the simplest and most flexible to implement.

flashflexpro commented 7 months ago

Come on!! https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

293000 byte payload is too large for the Event invocation type (limit 262144 bytes) (Service: AWSLambda; Status Code: 413; >Error Code: RequestEntityTooLargeException;

please download and kubectl apply inside the Labmda, NOT thru properties!

This is more like a bug from designing phase! How on earth could passing large manifest thru property work ever?!

CarlosDomingues commented 4 months ago

I've been bitten by this. Any workaround?

jordangullen commented 2 months ago

This issue could really use an update as none of the workarounds are ideal.

+1 to manifestUrl as a property on the KubernetesManifest.

@peterwoodworth - do you have any updates on this?

aws / aws-cdk