search

kubernetes / kubernetes

Production-Grade Container Scheduling and Management
https://kubernetes.io
Apache License 2.0
107.32k stars 38.57k forks source link

Wishlist of CEL libraries #124490

Open alculquicondor opened 4 weeks ago

alculquicondor commented 4 weeks ago

What would you like to be added?

This is just my wishlist. We need all of them, so no specific priority.

Why is this needed?

As we were working on transitioning Kueue validation webhooks to CEL, we ended up with a lot of repeated validation lines. In some cases, validating something is so involved that we decided to keep most of the webhooks, otherwise readability would be significantly impacted.

alculquicondor commented 4 weeks ago

/sig api-machinery cc @cici37 @jpbetz

alculquicondor commented 4 weeks ago

@IrvingMg, @trasc did I miss anything important?

sftim commented 4 weeks ago

What do you mean by Pod templates @alculquicondor?

One pattern I'd like to see used more: when you create something that embeds a Pod template, the controller for that kind tries to dry-run make a PodTemplate. That way you get one place to put customer validation (eg a ValidatingAdmissionPolicy), and it can apply to lots of API kinds without repetition.

alculquicondor commented 4 weeks ago

One pattern I'd like to see used more: when you create something that embeds a Pod template, the controller for that kind tries to dry-run make a PodTemplate.

Interesting. I've never seen that. It sounds bullet proof from a validation perspective. Is there dry-run support in the apiserver? But then it would have to be called from the webhook?

jpbetz commented 4 weeks ago

Regexes for object names, label keys, values, container names, etc. I think this one is already in the works?

Yes, this one is progressing here: https://github.com/kubernetes/kubernetes/pull/123572 (cc @alexzielenski)

jpbetz commented 4 weeks ago

The ultimate validation: Pod templates, but worth starting with just containers :) Very useful for job CRDs.

We might do something special to validated embedded types like this that doesn't involved CEL. But yes, I agree there is a huge need here. Do you happen to have any references to specific use cases? I'm working on accumulating those.

liggitt commented 4 weeks ago

Is there dry-run support in the apiserver?

yes, since 1.12: https://github.com/kubernetes/kubernetes/blob/release-1.12/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L485-L491

alculquicondor commented 4 weeks ago

Re: dry-run support I see. Still, that would imply that a webhook has to do an API call. Would you still recommend this?

Do you happen to have any references to specific use cases?

  1. All of the kubeflow job CRDs, to start. They will eventually fail at Pod or k8s Job creation, if the template is wrong, as they are not doing any validation.
  2. Kueue Workload objects https://kueue.sigs.k8s.io/docs/concepts/workload/. These are created out of existing Jobs, Pods, or arbitrary CRDs (like Kubeflow jobs). The only problematic case is the last one.
kannon92 commented 4 weeks ago
  1. JobSet validation (based on JobTemplates) https://github.com/kubernetes-sigs/jobset/issues/422
kannon92 commented 4 weeks ago

@danielvegamyhre was looking into kubectl-validate as a way to validate templates as a library.

IrvingMg commented 3 weeks ago

@IrvingMg, @trasc did I miss anything important?

Not that I can think of. That's everything we need for now.

alculquicondor commented 3 weeks ago

WRT conditions, right now, we have to do things like this:

has(self.status) && has(self.status.conditions) && self.status.conditions.exists(c, c.type == 'QuotaReserved' && c.status == 'True')

It would be good to have a simplified experience, similar to meta.IsConditionTrue in golang.

I added a separate item for this.

liggitt commented 3 weeks ago

fwiw, optionals can make that more concise:

self.?status.?conditions.orValue([]).exists(c, c.type == 'QuotaReserved' && c.status == 'True')
fedebongio commented 5 days ago

/cc @cici37 @alexzielenski @jpbetz /triage accepted