search

operator-framework / kubectl-operator

Manage Kubernetes Operators from the command line
https://operatorframework.io/
Apache License 2.0
128 stars 37 forks source link

New feature: list-operatorgroups #82

Closed cdjohnson closed 5 months ago

cdjohnson commented 1 year ago

This is a new feature: kubectl-operator list-operatorgroups

The goal here is to:

Example output using a set of test data found in assets/operatorgroup-list-test:

$ bin/kubectl-operator list-operatorgroups -A 
TENANT                                TYPE             SUBTENANTS  TARGETNAMESPACES                      PARENTTENANTS
oa                                    OwnNamespace                 oa                                    
oaa                                   SingleNamespace  oaa1        oaa1                                  
oaa1                                  OwnNamespace                 oaa1                                  oaa,obb
ob                                    MultiNamespace   ob1         ob,ob1                                
ob1                                   OwnNamespace                 ob1                                   ob
obb                                   SingleNamespace  oaa1        oaa1                                  
oc                                    MultiNamespace   oc1,oc2     oc1,oc2                               
oc1                                   OwnNamespace                 oc1                                   oc
oc2                                   OwnNamespace                 oc2                                   oc
occ                                   SingleNamespace  occ1        occ1                                  
occ1                                  SingleNamespace  occ11       occ11                                 occ
occ11                                 OwnNamespace                 occ11                                 occ1,odd2
od                                    MultiNamespace   od1,od2     od,od1,od2                            
od1                                   MultiNamespace   od11,od12   od1,od11,od12                         od
od11                                  OwnNamespace                 od11                                  od1
od12                                  OwnNamespace                 od12                                  od1
od2                                   MultiNamespace   od21,od22   od2,od21,od22                         od
od21                                  OwnNamespace                 od21                                  od2
od22                                  OwnNamespace                 od22                                  od2
odd                                   SingleNamespace  odd1        odd1                                  
odd1                                  SingleNamespace  odd11       odd11                                 odd
odd2                                  SingleNamespace  occ11       occ11                                 
oe                                    SingleNamespace  ne1         ne1                                   
of                                    MultiNamespace   nf1,nf2     nf1,nf2                               
og                                    MultiNamespace   og1,og2     og1,og2                               
og1                                   MultiNamespace   ng11,ng12   ng11,ng12                             og
og2                                   MultiNamespace   ng21,ng22   ng21,ng22                             og
oh                                    MultiNamespace   oh1,th11    oh1,th11                              
oh1                                   SingleNamespace  nh11        nh11                                  oh
openshift-monitoring                  AllNamespaces                                                      
openshift-operator-lifecycle-manager  OwnNamespace                 openshift-operator-lifecycle-manager  
openshift-operators                   AllNamespaces

$ bin/kubectl-operator list-operatorgroups -A -g

flowchart LR
  classDef tenant fill:#eFe,stroke:#000;
  classDef bad fill:#f99,stroke:#000;
  classDef ns fill:#fff,stroke:#000,stroke-width:1px;
  classDef im-OwnNamespace fill:#eef,stroke:#000,stroke-width:2px;
  classDef im-SingleNamespace fill:#ddf,stroke:#000,stroke-width:2px;
  classDef im-MultiNamespace fill:#ccf,stroke:#000,stroke-width:2px;

  subgraph Legend [Legend]
  direction TB  LNS[Namespace]:::ns
  LOGOWN(OwnNamespace):::im-OwnNamespace
  LOGSINGLE(SingleNamespace):::im-SingleNamespace
  LOGMULTI(MultiNamespace):::im-MultiNamespace
  LOGNS[Operand Namespace]:::ns
  end  t-multicluster-engine:::tenant
  subgraph t-multicluster-engine[multicluster-engine]
  direction LR
  multicluster-engine(multicluster-engine):::im-OwnNamespace --> multicluster-engine(multicluster-engine):::im-OwnNamespace
  end

  t-oc:::tenant
  subgraph t-oc[oc]
  direction LR
  oc(oc):::im-MultiNamespace --> oc1(oc1):::im-OwnNamespace
    oc1(oc1):::im-OwnNamespace --> oc1(oc1):::im-OwnNamespace
  oc(oc):::im-MultiNamespace --> oc2(oc2):::im-OwnNamespace
    oc2(oc2):::im-OwnNamespace --> oc2(oc2):::im-OwnNamespace
  end

  t-obb:::tenant
  subgraph t-obb[obb]
  direction LR
  obb(obb):::im-SingleNamespace --> oaa1(oaa1):::bad
    oaa1(oaa1):::bad --> oaa1(oaa1):::bad
  end

  t-oe:::tenant
  subgraph t-oe[oe]
  direction LR
  oe(oe):::im-SingleNamespace --> ne1[ne1]:::ns
  end

  t-occ:::tenant
  subgraph t-occ[occ]
  direction LR
  occ(occ):::im-SingleNamespace --> occ1(occ1):::im-SingleNamespace
    occ1(occ1):::im-SingleNamespace --> occ11(occ11):::bad
      occ11(occ11):::bad --> occ11(occ11):::bad
  end

  t-odd:::tenant
  subgraph t-odd[odd]
  direction LR
  odd(odd):::im-SingleNamespace --> odd1(odd1):::im-SingleNamespace
    odd1(odd1):::im-SingleNamespace --> odd11[odd11]:::ns
  end

  t-og:::tenant
  subgraph t-og[og]
  direction LR
  og(og):::im-MultiNamespace --> og1(og1):::im-MultiNamespace
    og1(og1):::im-MultiNamespace --> ng11[ng11]:::ns
    og1(og1):::im-MultiNamespace --> ng12[ng12]:::ns
  og(og):::im-MultiNamespace --> og2(og2):::im-MultiNamespace
    og2(og2):::im-MultiNamespace --> ng21[ng21]:::ns
    og2(og2):::im-MultiNamespace --> ng22[ng22]:::ns
  end

  t-ob:::tenant
  subgraph t-ob[ob]
  direction LR
  ob(ob):::im-MultiNamespace --> ob(ob):::im-MultiNamespace
    ob(ob):::im-MultiNamespace --> ob1(ob1):::im-OwnNamespace
      ob1(ob1):::im-OwnNamespace --> ob1(ob1):::im-OwnNamespace
  end

  t-of:::tenant
  subgraph t-of[of]
  direction LR
  of(of):::im-MultiNamespace --> nf1[nf1]:::ns
  of(of):::im-MultiNamespace --> nf2[nf2]:::ns
  end

  t-oh:::tenant
  subgraph t-oh[oh]
  direction LR
  oh(oh):::im-MultiNamespace --> oh1(oh1):::im-SingleNamespace
    oh1(oh1):::im-SingleNamespace --> nh11[nh11]:::ns
  oh(oh):::im-MultiNamespace --> th11[th11]:::ns
  end

  t-oaa:::tenant
  subgraph t-oaa[oaa]
  direction LR
  oaa(oaa):::im-SingleNamespace --> oaa1(oaa1):::bad
  end

  t-odd2:::tenant
  subgraph t-odd2[odd2]
  direction LR
  odd2(odd2):::im-SingleNamespace --> occ11(occ11):::bad
  end

  t-oa:::tenant
  subgraph t-oa[oa]
  direction LR
  oa(oa):::im-OwnNamespace --> oa(oa):::im-OwnNamespace
  end

  t-od:::tenant
  subgraph t-od[od]
  direction LR
  od(od):::im-MultiNamespace --> od(od):::im-MultiNamespace
    od(od):::im-MultiNamespace --> od1(od1):::im-MultiNamespace
      od1(od1):::im-MultiNamespace --> od1(od1):::im-MultiNamespace
        od1(od1):::im-MultiNamespace --> od11(od11):::im-OwnNamespace
          od11(od11):::im-OwnNamespace --> od11(od11):::im-OwnNamespace
        od1(od1):::im-MultiNamespace --> od12(od12):::im-OwnNamespace
          od12(od12):::im-OwnNamespace --> od12(od12):::im-OwnNamespace
    od(od):::im-MultiNamespace --> od2(od2):::im-MultiNamespace
      od2(od2):::im-MultiNamespace --> od2(od2):::im-MultiNamespace
        od2(od2):::im-MultiNamespace --> od21(od21):::im-OwnNamespace
          od21(od21):::im-OwnNamespace --> od21(od21):::im-OwnNamespace
        od2(od2):::im-MultiNamespace --> od22(od22):::im-OwnNamespace
          od22(od22):::im-OwnNamespace --> od22(od22):::im-OwnNamespace
  end

  t-openshift-operator-lifecycle-manager:::tenant
  subgraph t-openshift-operator-lifecycle-manager[openshift-operator-lifecycle-manager]
  direction LR
  openshift-operator-lifecycle-manager(openshift-operator-lifecycle-manager):::im-OwnNamespace --> openshift-operator-lifecycle-manager(openshift-operator-lifecycle-manager):::im-OwnNamespace
  end
openshift-ci[bot] commented 1 year ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cdjohnson Once this PR has been reviewed and has the lgtm label, please assign joelanford for approval by writing /assign @joelanford in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/operator-framework/kubectl-operator/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
dmesser commented 1 year ago

Curious, what use case does this address?

cdjohnson commented 1 year ago

Curious, what use case does this address?

Our larger customers want to create large deployment topologies within a single cluster that allow different isolation layers to allow different instances of Products (Operators and Operands). Our products are made up of many operators, some of which are shared and some are not, so it can get fairly complicated quickly. In the most complicated scenario, you may have a multiple Trees of Namespaces, where each layer in the tree represents a different Control Plane, or Isolation Layer.

The goal of this was to help developers and SREs visualize these more complex deployment scenarios and validate their configurations.

dmesser commented 1 year ago

Wouldn't this question be best answered from the perspective of an operator configured / scoped via an OperatorGroup ? I.e. instead of listing the OperatorGroup themselves, which are sort of an abstract concept rather, we could list all operators and their scope

cdjohnson commented 1 year ago

@dmesser This is in the scope of the current OLM APIs and it's current capabilities. This is showing what the Deployment Environment/Topology that the Operators will be installed into. Although the OperatorGroup is an abstraction, customers need to set them up correctly and orchestrate the operators to install into that environment to have success.

So, this shows that the Namespaces are setup properly even before the first operator is provisioned.

So:

  1. The customer needs to have awareness of Namespaces and OperatorGroups to build the right topologies (new tool could help here).
  2. Validate that they are setup properly (this tool).
  3. Deploy the operators and operands into those namespaces.
  4. Validate that the Operators are installed properly (new tool....?)
dmesser commented 1 year ago

I guess I am trying to understand the situation in which a user would use this command. It's not obvious to me.

Let's say I want to install an operator or a set of related operators. Why is the question which operator groups already exist interesting to me? Wouldn't I be rather interest in:

1) discovering which operators are available for me to install (this plugin can do that) 2) select the version of the operator I want to install (this plugin cannot do this yet, but we have made in-roads into this with packageserver) 3) installing the operator(s) which a specific scope (this tool can do this) 4) validate the operators are installed and healthy (this tool will check the install, install health needs more discussion)

I don't see OperatorGroups in here as a primary concern. They are an implementation detail of how the operator got configured and installed. The plugin sets them up transparently in step 2. If there are 2 or 10 OperatorGroups installed in other namespaces, potentially scoped the (partially) same way the next one is about to come down, doesn't really matter IMHO. What matters is that we notice when an operator has already be installed and configured to be available in the same namespace / namespaces you are targeting. I am not sure that we have made that yet.

cdjohnson commented 1 year ago

The problem is that we can't rely on OLM to resolve and provision the dependent operators for us, because they live at different scopes. Some customers (big ones) want to use the principle of least privilege a the Namespace layer. This is an extremely "advanced" use case, but one that we need to support.

Primary goals:

Let's say we have 4 layers:

  1. Cluster (Certificate Management, License Management)
    • Watches All or Many namespaces. These are cluster-singletons.
  2. Shared Tenant Operators (IAM and other shared services. Could represent a Company)
    • IAM CRs are likely created here... a shared UI, User Registry, etc that's common to all product instances
  3. Tenant Instance Operators (Could represent a Department)
    • Application oriented operators and middleware.
    • Has Kube RBAC
  4. Tenant Operands (CRs only)
    • The instances of the Application. Or a "Data Plane"
    • No Kube RBAC at all

In order to realize this, we have a tree of namespaces and nested operator groups. 1 manages 2 manages 3 manages 4

Because of this, we need to design the Namespaces and their OperatorGroups ahead of time, essentially Modeling the desired topology, and then deploy the Operators and Operands into that topology, not depending on OLM to auto-provision dependencies.

Here's a picture: image

cdjohnson commented 1 year ago

@dmesser So, I think I can clarify what the benefit of this tool could be:

  1. As a Cluster Admin, how can I explore how the current Tenancy Topology is configured, so I can determine which Namespace to install my Operator into?
  2. How can I detect mis-configurations of operator groups which might result in Operator deployment problems?
openshift-merge-robot commented 1 year ago

@cdjohnson: PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
perdasilva commented 5 months ago

https://github.com/operator-framework/operator-lifecycle-manager/pull/3254