Create k8s resources by OperandConfig

horis233 commented 3 years ago

/kind feature

Describe the solution you'd like [A clear and concise description of what you want to happen.]

In some cases, a single operator may not be runnable with some additional k8s objects as a prerequisite. For example, image pull secrets or licensing keys.

This feature is for creating some k8s objects by OperandConfig.

1. When the Operator is requested, and `resources` is configured in the OperandConfig, then ODLM will create these resources for the Operator.

apiVersion: operator.ibm.com/v1alpha1
kind: OperandConfig
metadata:
  name: operandconfig
  namespace: default
spec:
  services:
  - name: example-operator
     spec:
      example-operand:
        some-spec-field: myvalue
     resources:
     -   kind: Job
         name:  registry-key-generater
         apiVersion: batch/v1
         spec:
           template:
             metadata:
               ..

resources is a list of k8s resources.

Required fields in the individual resource

kind indicates the kind of the resource.
apiVersion indicates the API group and version of the resource.
name indicates the name of the k8s resource.

Optional fields in the individual resource

spec: is the spec field of the k8s resource.
labels, annotations: Users can use these fields to set up resource labels and annotations.
namespace: Users can customize the namespace of the resource. By default, the namespace is the operator namespace.
force: Users can use it to decide if ODLM should override the existing resource. By default, the value is true, ODLM will always override existing resource

2. ODLM will generate the resource in the cluster with other requested CRs

3. ODLM will update the status of OperandConfig

status:
  phase: Running
  serviceStatus:
    example-operator:
      customResourceStatus:
        example-operand: Running
      resourceStatus:
         job.v1.batch:
            default/registry-key-generater: deployed

Then we users delete the field from OperandConfig, ODLM could clean it up according to the status.

4. When the operator is deleted, k8s resources created by ODLM should be clean up as well.

Note: If the resource isn't created by ODLM, ODLM won't update or delete it.

Limitations

Since ODLM only has namespace admin permission, it can't create cluster scope resources without manually grant it enough cluster roles.

cc @ZhuoxiLi @Daniel-Fan

pgodowski commented 3 years ago

Does this mean that ODLM would have now permission to create/delete arbitrary k8s resources? Perhaps we shall initially restrict the Kind's of resources allowed?

Daniel-Fan commented 3 years ago

Currently the ODLM has the permission to manipulate all kinds of resources in namespace-scope(including the namespaces that is extended by namespace-scope operator) https://github.com/IBM/operand-deployment-lifecycle-manager/blob/master/bundle/manifests/operand-deployment-lifecycle-manager.clusterserviceversion.yaml#L681

To restrict the kinds of cluster scope resources that ODLM is allowed to create, we should specify them inside ODLM's CSV.

ruairi-hayes commented 3 years ago

Looks good guys. Would it be worth restricting it in the beginning to a base set of resources that you think would be useful initially and then augment it with other resources as the needs and usecases arise. That might cut down on the number of paths that need to be tried and tested out initially. The ones in my opinion that would probably be most useful and used would be Jobs, Cronjobs, ConfigMaps and perhaps Secrets.

pgodowski commented 3 years ago

I suggest we start with a one-off Job for the already identified use-case and then incrementally expand, based on the identified use cases. And, explicitly list Job resource in the required permissions.

Daniel-Fan commented 3 years ago

Deploy/Update resources

OperandRequest controller read the entire data of each resource from resources section for that operator and create/update the k8s object. It follows the same method when we create the CR from OperandRequest., Also labels are required to indicate that it is created by ODLM.

Delete resources

OperandConfig controller will update the OperandConfig status by checking whether the resource defined in resources section could be found in env, this field is used to indicating the k8s resources for current operator has been deployed.
- When the the k8s resources info has been removed by users, the OperandConfig controller will compare the status and spec section to determine that the k8s resources created by ODLM should be deleted or not.
- And the OperandConfig controller should update the k8s resource status according to the situation(removing status if it is deleted or indicate the failure of deletion and so on). NOTE: I thought it is not reasonable to delete the k8s resources by OperandRequest controller, it does not have any information about custom k8s resource. The deletion logic will be more doable if it is in OperandConfig reconciliation.

Exceptions

ODLM is not responsible for the syntax/semantics of of each k8s resources. It will only follow the raw data from OperandConfig to create the resources. The error will happen because of the false configuration in OperandConfig
Whether ODLM should check its own permission to create cluster scope resources. If not, the ODLM will create the cluster scope resources and will return error when it hits the permission issue.

Please review the above breakdown of the task @horis233 @ZhuoxiLi

IBM / operand-deployment-lifecycle-manager