Feature idea: Controller to owns other objects

[Hanspagh]

I was looking at the documentation for the rust operator, and it it has the concept of of a controller 'owning' other resources than the ones the controller monitors. https://kube.rs/controllers/relations/#owned-relation

I found the idea pretty interesting since we often want to update the monitored resource based on the change of a child resource (Resource created by the monitored resource). This can eg. be useful when we want to update the status based on the status of the child resource.

I think this could be loosely related to https://github.com/coryodaniel/bonny/issues/88, but instead of an id, we could watch the child resource and watch for the owner reference.

I imagine the interface would be something along the lines of

      %{
        query: K8s.Client.watch("batch/v1", "Job", namespace: watching_namespace),
        owns: [pods: K8s.Client.watch("v1", "Pod", namespace: watching_namespace)],
        controller: My.Controller.JobController
      },  

And then in the Axn there would be a new field children with a list of the children created by the resource.

Complications:

• How to do reconciliation with multiple resources?
• Do we need to keep state to always serve the 'latest' resource and children?

Let me know what you think, I would be happy to contribute on building this.

[mruoss]

Hmm... I think I see what you want to achieve but I have difficulties understanding the approaches.

Let's try to go step by step and see how the final API would look like? Maybe you already did that but I don't undersstand it.

So in a first step I would:

• add specific labels to children so we are able to select them
• add a separate controller definition in the Operator for the children that uses the label selector to only get relevant resources

  %{
    query: K8s.Client.watch("v1", "Pod", namespace: watching_namespace) |> K8s.Selector.label({"my-operator.com/controlled-by", "some-crd"}),
    controller: My.Controller.PodController
  },  

• In My.Controller.PodController load and modify the parent resource (being careful not to run into infinite loops)

And then in the Axn there would be a new field children with a list of the children created by the resource.

Wha would we need those for? This sounds a lot like #88, no?

[Hanspagh]

I can try to explain a bit better. Lets just imagine wanted to write a Controller to mimic the what the native k8s Job controller does. Simplified what that controller does is: Monitor pods and when the status changes, update the job that is the owner of that pod.

With the current setup that would be quite hard because we would like to track changes to pods, but on the same time change the status of the jobs. So we could create a controller that tracks Jobs, but would need to query the state of the pods once in a while to see the status

OR

Create both a controller for Pods and Jobs, and keep track of the state for pods so that we can then lookup from the Jobs

Both of these has the problem that we don't get an event in the Job controller when the Pod changes.

The idea was to make a Controller that tracks both resources. For the Job we would do what we do now, but also trigger events when the pod changes. We would like Jobs and Pods though the owner reference.

That is why my API proposal was

%{
        query: K8s.Client.watch("batch/v1", "Job", namespace: watching_namespace),
        owns: [pods: K8s.Client.watch("v1", "Pod", namespace: watching_namespace)],
        controller: My.Controller.JobController
      }

On top of that it would be neat to have the status of the child resources in the Job event so we could update the status according to that, but this would properly require us to store the 'latest' Pod events, so all events trigger for the Job would have access to the last Pod event. I can see the rust implementation have the concept of a Reflector which is essentially a Watcher with some reconciliation / state management on top.

I hope this make a bit more sense.

[mruoss]

With the current setup that would be quite hard because we would like to track changes to pods, but on the same time change the status of the jobs. So we could create a controller that tracks Jobs, but would need to query the state of the pods once in a while to see the status

But why don't you create a watcher for Pods and one for Jobs and have both handle by the same controller like I described above?

[Hanspagh]

With the current setup that would be quite hard because we would like to track changes to pods, but on the same time change the status of the jobs. So we could create a controller that tracks Jobs, but would need to query the state of the pods once in a while to see the status

But why don't you create a watcher for Pods and one for Jobs and have both handle by the same controller like I described above?

Maybe I am missing something, but how would you update a Job from within the Pod Controller? Will it not be two different Axn? One for Pods and one for Jobs. I guess I could create some custom logic to create a Job Axn from a Pod Axn. I just thought this was a general enough use-case that it could have have made sense to support.

[mruoss]

Maybe I am missing something, but how would you update a Job from within the Pod Controller? Will it not be two different Axn?

Well... yes it would be different Axn and your controller would have to implement different logic for them and dispatch accordingly. You could also use separate controllers of course (JobController and PodController).

I guess I could create some custom logic to create a Job Axn from a Pod Axn.

Wait... why would you do that? The way I have it in mind, in your Pod controller, you would just load (GET) the parent Job manifest and register it as a descendant on the Axn (Bonny.Axn.register_descendant) and passing omit_owner_ref: true (as the pod should not be the owner of the pod)

I just thought this was a general enough use-case that it could have have made sense to support.

I'm not against supporting this case (or making this it easier). I just want to experiment with what is possible today, start there and make that solution easier to use / understand...

But let me again try to understand the API you're proposing: Your definition would make Bonny watch for Jobs and Pods. But both watch events would create the same Axn, right? I.e. if a pod changes, that event results in an Axn for its job with a children: [all_its_pods] element?

[Hanspagh]

I wanted to make it easy to update the status of a Job, when the child pod changes. Hence I wanted to utilize the Job Controller logic because there I already had access to the Axn for the Job and I could just add event/status for the change.

I think I also that with two difference controller, but just requires a bit more custom logic to find the right Job for a Pod and update the status/events for that Job.

With the children approach I would have all the logic contained the Job controller and always have access to the latest state of the Job and its child Pod. As I mentioned I think this will require the Controller/Watcher to be statefull, else we will have no way of knowing the latest state of the Pod/Job.

Does it make a bit more sense now?

[mruoss]

A little. ;) However, why does the controller have to be stateful?

else we will have no way of knowing the latest state of the Pod/Job.

Can't we get the latest state from the cluster by simply GETting the resource?

[Hanspagh]

A little. ;) However, why does the controller have to be stateful?

else we will have no way of knowing the latest state of the Pod/Job.

Can't we get the latest state from the cluster by simply GETting the resource?

You are right, it does not strictly need to be, I guess it will just generate more request to the cluster, so it is a tradeoff between request and operator memory usage :)