Event filtering with predicates

sdurrheimer commented 9 months ago

Hi there,

First of all thank you for lightweight alternative to the bigger operator frameworks.

Is there any way/chance we could use Predicates for event filtering in kooper?

https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate

GonzaV commented 3 months ago

@sdurrheimer Did you find a way around to implement this?

sdurrheimer commented 3 months ago

I don't have a workaround under my hand, sorry.

slok commented 3 months ago

Hi! :wave: @sdurrheimer @GonzaV

AFAIK, Predicates is something implemented by Kubebuilder/controller-runtime, and not kubernetes client-go, so the state is hanlded in the controller-runtime client logic. Kooper tries mainly getting out of all the compelxity of controller-runtime, that's why we don't import anything related with it, and we try creating a simple implementation of a controller using mainly low level informers and caches from the original client-go. For more complex/performant use cases there are already other frameworks like Kubebuilder.

For what I've seen, most used predicates (even the ones that they offer by default ready to be used) are based on old VS new objects (a.k.a Update envents). Kooper is very strict in this approach of only giving you the latest state so you handle correctly a correct "state" instead of doing delta/diffs and avoid shooting yourself in the foot with complex logic paths.

However, understanding the real problem you have and what you are trying to solve, would give me context and look at it from another perspective. Could you explain me the use cases please? :)

Many thanks!

GonzaV commented 3 weeks ago

Hi @slok ! Thanks for your response and sorry for being this late.

Let me try to explain what I was trying to do, the problem I faced, how I solved it and what I'm facing now (BTW, you might now this project very well, builder tools says hi):

Need

At any error on resources when handling events, I wanted to update the resource conditions to expose this (more info about conditions can be found here if you find it useful).

Problem

Updating the resource with the new condition worked just fine, but it fired an event loop since this new update means a new event in K8s (with type "MODIFIED"), which is going to be handled by Kooper. This new handling will include the retry of the logic that previously failed and since this was less than a second ago, 99% of cases will fail again relaunching a condition update and a new handle, making this loop. Hopefully this "timeline" makes it clearer:

First event handling launches ---> Handling logic fails for any reason ---> I update the conditions of the resource in the cluster posting to K8s API ---> This update launches a MODIFIED event which reaches Kooper ---> Second event handling launches ---> Handling logic fails once again since problem has not been solved ---> Update logic is fired once again ---> MODIFIED event reaches Kooper ---> ... Loop

That's why I thought about the possibility of filtering events

Solution

Won't make this longer that it already is, so the short version is that I configured MustRetrieverFromListerWatcher from retrieve.go with a Filter which logic stops this loop (tried many other different approaches but this one was the cleanest).

Current status and problems

About what you said here Kooper is very strict in this approach of only giving you the latest state so you handle correctly a correct "state" instead of doing delta/diffs and avoid shooting yourself in the foot with complex logic paths., there is currently a requeue mechanism (not sure if it's custom or from Kooper right now) that I suspect is trying to work with an outdated version of the resource judging by the resourceVersion from it, leading to some problems. Can't share much more detail sadly, but I'm currently working on this.

That will be it, thanks once again for your time and your work!

slok commented 3 weeks ago

:wave: Hey Gonzalo!

If I understood correctly :crossed_fingers:, your problem is that K8s apiserver triggers an update for the controller watchers on the status subresource change (however, the controller should only receive these updates on the spec changes). This is something from Kubernetes CRDs that is a well know problem. Normally to overcome this problem what is used is the ObservedGeneration field from the conditions.

This status field will be set to the generation that is on the spec (k8s will increase this generation on every spec change) when the logic handles the resource. That way then you can know what was the last spec the controller acted on and you cna break the loop if it meets conditions like spec.generation == status.observedGeneration -> break/ignore.

You are telling that you use conditions, are you use this observeGeneration field and if so isn't work for you? why?

Example:

spotahome / kooper