can't find resource in apiVersion ...

[raffaelespazzoli]

hi I was able to run my controller based, on metacotroller but recently it broke with the below errors from the metacontroller logs:

E0221 03:36:18.827859       1 controller.go:133] can't sync Service prometheus/grafana: discovery: can't find resource  in apiVersion networking.k8s.io/v1
E0221 03:36:18.827867       1 controller.go:133] can't sync Service prometheus/kube-state-metrics: discovery: can't find resource  in apiVersion networking.k8s.io/v1
E0221 03:36:18.827877       1 controller.go:133] can't sync Service prometheus/prometheus: discovery: can't find resource  in apiVersion networking.k8s.io/v1

the metacontroller is looking for network policies. My controller CRD is as follows

apiVersion: metacontroller.k8s.io/v1alpha1
kind: CompositeController
metadata:
  name: microsegmentation-controller
spec:
  parentResource:
    apiVersion: v1
    resource: services  
  childResources:
    - apiVersion: extensions/v1beta1
      resources: ["networkpolicies"]
#    - apiVersion: networking.k8s.io/v1
#      resources: ["networkpolicies"]      
  clientConfig:
    service:
      name: microsegmentation-controller
      namespace: metacontroller
  hooks:
    sync:
      path: /sync

I am running in openshift 3.7 corresponding to kube 1.7 so I thought that networkpolicies might still be in beta (even though I saw it working) so I switched to extensions/v1beta1 but still the same error:

E0221 03:36:27.084535       1 controller.go:133] can't sync Service default/docker-registry: discovery: can't find resource  in apiVersion extensions/v1beta1
E0221 03:36:27.084571       1 controller.go:133] can't sync Service default/kubernetes: discovery: can't find resource  in apiVersion extensions/v1beta1
E0221 03:36:27.084616       1 controller.go:133] can't sync Service default/registry-console: discovery: can't find resource  in apiVersion extensions/v1beta1

not sure what is happening but I suspect some changes in a recently published image, can you confirm?

[enisoc]

Sorry, it looks like I broke you with https://github.com/kstmp/metacontroller/pull/20, so this:

resources: ["networkpolicies"]

should be changed to this (note that the field name changed in addition to the value):

resource: networkpolicies

I've been changing the API pretty aggressively (another important change recently was #15) to get ready for alpha, after which the rate of change in the API should slow. I've also been overwriting the 0.1 image tag since 0.1 hasn't actually been released. Once 0.1 is released, there will be one or more 0.1.x tags that stay pinned.

By the way, have you encountered any unexpected behavior from using a parentResource that has its own controller running already (in this case v1 services)? I was intending for such use cases to fall under a different pattern -- something like Observer rather than Composite. For example, CompositeController assumes that it owns the Status section of the parent resource.

I'd be interested in hearing if you've had some success using CompositeController for this. Maybe it's worth expanding the intended scope if only minor changes are needed (like not trying to update status on the parent).

[raffaelespazzoli]

ok, thanks, I'll update my code. I didn't realize that the status update was intended for the parent resource, I don't know what side effects that has on non-owned resources. I will check that. What is the best approach with the current CompositeController pattern then? To return an unchanged state for the parent resource (I was changing it...)? other than that everything else seemed to be working fine.

I have one question on what happens when children resources are created by third parties. To make an example in namespace ns1 I can have an existing networkpolicy np1, then a service is created sv1 and my controller would return a new networkpolicy np2. What does the metacontroller do with np1 and how does the metacontroller know when to delete np2 when sv1 is deleted?

[raffaelespazzoli]

I think I understand how the metacontroller works with regards to non-owned children object: you need to have the in the output untouched. And also I understand how it works with respect to deletion of children objects: it will delete any children object that is not present in the output. Then I have a suggestion, maybe the metacontroller could keep track on the children objects it is managing (I use an annotation: created-by:) this way the writer of a controller would not have to track owned versus non-owned children objects and they would be always dealing with owned children objects.

[enisoc]

What is the best approach with the current CompositeController pattern then? To return an unchanged state for the parent resource (I was changing it...)?

If you always return the observed parent status unchanged as the desired parent status, Metacontroller won't try to update the parent object. That's probably the best practice if there's already another controller responsible for managing that object's status.

I can have an existing networkpolicy np1, then a service is created sv1 and my controller would return a new networkpolicy np2. What does the metacontroller do with np1 and how does the metacontroller know when to delete np2 when sv1 is deleted?

Metacontroller implements the k8s convention of using ControllerRef to handle these scenarios.

For CompositeController, the parent object is assumed to have a field spec.selector that matches the new-style (matchLabels:) selector type used by Deployment, StatefulSet, etc. On each sync, the parent tries to "adopt" anything that matches its selector, but doesn't already have a parent (recorded as a ControllerRef on the child object). Metacontroller then sends you only the child objects that you own.

Metacontroller will also "orphan" (give up ownership of) anything that it owns that no longer matches the parent selector. Note that this means you're responsible for putting labels on any children you create to ensure they match the parent's selector, or else they'll be immediately orphaned. In the future, I plan to do some validation of this (#24).

Then I have a suggestion, maybe the metacontroller could keep track on the children objects it is managing

This should already be happening via ControllerRef. However, I just remembered that Service still has the old-style spec.selector, so in your case it is probably implicitly selecting all networkpolicies (because the matchLabels field is empty). This sort of defeats the mechanism that normally ensures Metacontroller only sends you objects you own. I'd also like to do validation to help avoid this mistake (#25).

---

With the above in mind, I'd like to suggest an alternative approach for your controller that would address both problems (tracking child ownership, and using a parent object you have full authority over).

It looks like your intention is not to run a controller that processes all Services as parent objects, but rather only those that explicitly opt-in with annotations:

annotations:
  io.raffa.microsegmentation: true
  io.raffa.microsegmentation.additional-ports: 9999/tcp, 8888/udp

My suggestion is to put that configuration into your own CRD, which would then act as the parent object:

apiVersion: raffa.io/v1beta1
kind: MicrosegmentedService
metadata:
  name: sv1
spec:
  # Configuration goes here instead of in annotations.
  additionalPorts:
  - port: 9999
    protocol: tcp
  - port: 8888
    protocol: udp
  # This is how MicrosegmentedService parent objects find their children (services and networkpolicies).
  selector:
    matchLabels:
      app: myapp
  # MicrosegmentedService will create this service as its child, with the same name as the parent.
  # The labels on the serviceTemplate need to match the parent selector above,
  # just like the Pod template labels must match a Deployment's selector.
  serviceTemplate:
    metadata:
      labels:
        app: myapp
    spec:
      selector:
        app: myapp

Your custom parent object MicrosegmentedService would then create and own a child service, as well as creating and owning child networkpolicies. Metacontroller will take care of ignoring networkpolicies that don't match the spec.selector on the parent MicrosegmentedService. You'll need to make sure the networkpolicies you create match this selector, which you can do by copying the labels from the serviceTemplate.

Now, to use this, instead of putting an annotation on a Service object, you would convert that Service into a MicrosegmentedService in the source manifest.

[raffaelespazzoli]

ok, thanks for the extensive answer. if you want to enable the style of controller I'm trying to write, i.e. where the controller does not own the parent resources, then I suggest enabling selecting the parents also via an annotation selector rather than just the matchLabel mechanism. Annotations are often used for opt-in features in my experience, while labels more to create categories of objects. for example if you want istio to enable mTLS you add the following annotation to your service: auth.istio.io/8080: MUTUAL_TLS

Regarding your suggestion to refactor my controller, thanks, but I think it would diminish the value of it because it implies that everyone that uses services now they have to use use a new CRD.

[enisoc]

I see. It sounds like your use case is closer to "process all services" than I thought, since you want to make this mostly transparent for users who already have existing services.

An annotation filter is an interesting idea. I think that would make sense as part of a new "attach" pattern. Would something like the following API fit your use case better than CompositeController?

• Define a "parent" resource type to watch, but not modify (except maybe to manipulate annotations). It can be a resource that already has its own controller, like Service.
• Define a filter based on labels and/or annotations. Metacontroller will only send you things that match all defined filters.
• For each object of the watched type, you can specify some arbitrary "child" objects to be created. They will be attached to the watched parent to track ownership, but unlike CompositeController, there will be no label selectors for children.
• Metacontroller will not do adoption/orphaning (since there's no label selector), and will only send you objects of the child type that were created with a reference to that parent object from the beginning.
• If the parent object is deleted, all the children you created and attached will get deleted automatically.

[raffaelespazzoli]

@enisoc this design would work perfectly for my use case.

[enisoc]

Please give this a try when you have a chance: https://github.com/kstmp/metacontroller/pull/30