Open shahar-h opened 3 months ago
While trying to split the chart into 2 charts(app chart and CRDs chart) internally CRDs chart installation failed because helm release secret exceeded max 1MB size limit. In order to resolve that issue I split the CRDs chart into 2 more charts - eg CRDS and gateway api CRDs.
thinking out loud, we could add two additional templates (w/o deleting the /crds
dir) for the CRDs - one for gateway-api
and one for envoy-gateway
and use knobs such as applyCrds.all
applyCrds.gateway-api
and applyCrds.envoy-gateway
This solves 2 use cases
helm install --skip-crds --set applyCrds.envoy-gateway=true
to selectively install EG CRDshelm upgrade --set applyCrds.all=true
to update all CRDs or with --set applyCrds.envoy-gateway=true
to only update EG CRDsthe naming for the helm knobs can be improved, of course
@arkodg installing both EG and gateway-api CRDs from the same chart won't work since helm release secret will exceed 1MB limit as I mentioned here. In addition, installing CRDs from EG chart templates would not work for umbrella charts that contain some CRD instance.
What about keeping the crds folder and create 2 additional CRDs charts?
ah thanks for trying it out @shahar-h .
Looks like we have a decision to make, neither being ideal
Move CRDs from /crds
to templates/
with an opt in / opt out approach similar to what cert-manager does (opt in)
Pros
Add another gateway-crds-helm
chart similar to what linked does https://linkerd.io/2-edge/tasks/install-helm/#linkerd-crds
Pros
Common issues
gateway-crds-helm
chart where users may forget to install it.@shahar-h can you help investigate who is the consuming most of the space ?
@shahar-h can you help investigate who is the consuming most of the space ?
Sure, will do.
thanks ! I did a little digging, one way to reduce size could be to rm the API descriptions, but that would break kubectl explain
I computed eg CRDS & gwapi CRDs size by creating separate charts and installing each chart. Then I computed helm secret release size for each one of them with the following command:
kubectl -n envoy-gateway-system get secret <helm-release-secret-name> -o jsonpath='{.data.release}' | base64 -d | wc -c
Results:
Total: 1.17MB > 1MB size limit
Note that the helm release secret size is lower that real CRDs size since helm uses gzip compression.
BTW there is an open PR to split helm releases into multiple k8s secrets.
thanks for analyzing this !
the linked PR and the approach of using multiple k8s secrets looks promising, we could wait it out until that feature goes in and then move this the CRDs into templates
Description: Currently Gateway API and EG CRDs are located in EG
gateway-helm
chart crds folder. This means that CRDs are not upgraded on chart upgrade, and need to be manually upgraded beforehand.Possible solutions:
crds.enabled
flag. It also allows to decide if we want to keep the CRDs on chart uninstallation withcrds.keep
flag, by leveraging"helm.sh/resource-policy": keep
helm annotation. See: https://cert-manager.io/v1.13-docs/installation/helm/#crd-considerations. However, this is a breaking change for consumers that use EG chart as a sub chart and have custom resources as part of the main chart.Both 1 and 2 can also solve https://github.com/envoyproxy/gateway/issues/3094 by providing a separate flags(1)/charts(2) for Gateway API and EG CRDs.
Also Related:
WDYT?