search

kubernetes-sigs / prow

Prow is a Kubernetes based CI/CD system developed to serve the Kubernetes community. This repository contains Prow source code and Hugo sources for Prow documentation site.
https://docs.prow.k8s.io
Apache License 2.0
131 stars 99 forks source link

ProwJob CRD is too large #181

Closed maboehm closed 4 months ago

maboehm commented 5 months ago

Hi,

the prowJob CR just recently crossed the threshold of being too large, for example running against a kind cluster:

$ kind create cluster
...
$ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

# this is the last commit this still works on
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/prow/e67659d368fd013492a9ce038d801ba8998b7d10/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml --server-side
customresourcedefinition.apiextensions.k8s.io/prowjobs.prow.k8s.io serverside-applied

# latest version fails
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/prow/main/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml --server-side
Error from server: etcdserver: request is too large

Am I doing something wrong when applying?

In terms of solving this issue, the main problem is that there is a lot of repetition, e.g. the taskPodTemplate appears several times. It would be possible to have some fields as e.g. apiextension.JSON or runtime.RawExtension and defer validation to a validatingwebhook. Which makes the output of e.g. kubectl explain less helpful.

matthyx commented 5 months ago

Hi @maboehm does it work if you kubectl create -f ... instead? I think the main difference is that apply also saves the last applied configuration in the annotations, which doubles the size required and can more easily trigger the etcd limit.

matthyx commented 5 months ago 
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/prow/main/config/prow/cluster/prowjob-crd/prowjob_customresourcedefinition.yaml --server-side

(apparently it doesn't, sorry)

maboehm commented 5 months ago

Either way, since tools like Helm or Flux use apply the size should work for that as well.

lentzi90 commented 5 months ago

I have been using apply --server-side=true to avoid the annotation. It looks like the CRD itself is finally too large. With server side apply it worked up until https://github.com/kubernetes-sigs/prow/commit/e67659d368fd013492a9ce038d801ba8998b7d10 (including). From https://github.com/kubernetes-sigs/prow/commit/64052956637fb92301edded100fcf18e49930ed5 it is no longer possible to apply.

The only way to avoid this is to reduce the size of the CRD or configure etcd to allow a larger size. :slightly_frowning_face:

matthyx commented 5 months ago

ok well I think we have to shrink it... @maboehm wanna have a try?