Open ctml91 opened 10 months ago
This is a limitation in OpenShift with OLM. It's been addressed in the V5 release blog:
I believe it would be best to either only support all namespace, or provide 2 sets of OLM manifests, one for the all namespace mode and another for same namespace.
This is a limitation in OpenShift with OLM. It's been addressed in the V5 release blog:
I've believe I've seen this been used as a workaround before, submitting two bundles for OLM, one for same namespace and the other for all namespaces.
This is a limitation in OpenShift with OLM. It's been addressed in the V5 release blog: https://github.com/grafana-operator/grafana-operator/blob/5ebf1f965c87208ef58b7e5aeeb7765449c76952/docs/blog/v5-intro.md?plain=1#L203
I've believe I've seen this been used as a workaround before, submitting two bundles for OLM, one for same namespace and the other for all namespaces.
Would love to see any work arounds in detail. Do you have a issue/comment link? I'm looking to run Grafana Operator in namespaced mode as well. But don't think it will be possible because of the limitation I linked.
@Stef16Robbe for sure, I've seen for example with the Ansible operator provided by Red Hat they provide both options by having separate channels for the operator which indicate one is for cluster scoped and the other is namespace scoped. Basically they accomplish this by packaging two versions of the OLM manifests where installMode in the CSVs differ and the csv.spec.install.spec.clusterPermissions
and csv.spec.install.spec.permissions
are set accordingly for each.
Available channels, where the cluster-scoped allows for "all namespaces"
The namespace scoped channel can only be installed in a single namespace
When installing via namespace scoped channel the CSV does not enable the AllNamespaces
under installModes
, and the bulk of the privileges the operator requires are set via .spec.install.spec.permissions
. In the case of this operator it still has some cluster scoped privileges set, but only the bare minimum via .spec.install.spec.clusterPermissions
.
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
...
spec:
installModes:
- supported: true
type: OwnNamespace
- supported: true
type: SingleNamespace
- supported: true
type: MultiNamespace
- supported: false <--------
type: AllNamespaces
install:
spec:
permissions:
- rules:
- apiGroups:
- ''
- rbac.authorization.k8s.io
resources:
- secrets
- pods
- serviceaccounts
- roles
- rolebindings
- services
- services/finalizers
- endpoints
- persistentvolumeclaims
- events
- configmaps
- jobs
- pods/exec
- pods/log
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
clusterPermissions:
- rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
serviceAccountName: resource-operator-controller-manager
- rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
serviceAccountName: automation-controller-operator-controller-manager
- rules:
- apiGroups:
- config.openshift.io
resources:
- ingresses
verbs:
- get
- list
- watch
serviceAccountName: automation-hub-operator-sa
...
OLM install generates the Rolebindings/ClusterRolebindings.
$ oc get rolebinding | grep -i aap
aap-operator.v2.4.0-0.1698895982 Role/aap-operator.v2.4.0-0.1698895982 3m9s
aap-operator.v2.4.0-0.1698895982-automation-controll-6cc8dc6558 Role/aap-operator.v2.4.0-0.1698895982-automation-controll-6cc8dc6558 3m1s
aap-operator.v2.4.0-0.1698895982-automation-hub-oper-6dbdb6546d Role/aap-operator.v2.4.0-0.1698895982-automation-hub-oper-6dbdb6546d 2m58s
aap-operator.v2.4.0-0.1698895982-eda-server-operator-6889c9774f Role/aap-operator.v2.4.0-0.1698895982-eda-server-operator-6889c9774f 2m56s
aap-operator.v2.4.0-0.1698895982-resource-operator-c-64779c6698 Role/aap-operator.v2.4.0-0.1698895982-resource-operator-c-64779c6698 3m3s
$ oc get clusterrolebinding | grep -i aap-operator.v2.4.0
aap-operator.v2.4.0-0.1698895982-5546965bdd ClusterRole/aap-operator.v2.4.0-0.1698895982-5546965bdd 2m35s
aap-operator.v2.4.0-0.1698895982-5dbf596557 ClusterRole/aap-operator.v2.4.0-0.1698895982-5dbf596557 2m30s
aap-operator.v2.4.0-0.1698895982-64d8ccbfbd ClusterRole/aap-operator.v2.4.0-0.1698895982-64d8ccbfbd 2m28s
aap-operator.v2.4.0-0.1698895982-664d466d76 ClusterRole/aap-operator.v2.4.0-0.1698895982-664d466d76 2m33s
$ oc get clusterrole | grep -i aap-operator.v2.4.0
aap-operator.v2.4.0-0.1698895982-5546965bdd 2023-11-20T18:26:02Z
aap-operator.v2.4.0-0.1698895982-5dbf596557 2023-11-20T18:26:07Z
aap-operator.v2.4.0-0.1698895982-64d8ccbfbd 2023-11-20T18:26:09Z
aap-operator.v2.4.0-0.1698895982-664d466d76 2023-11-20T18:26:04Z
$ oc get clusterrole aap-operator.v2.4.0-0.1698895982-5546965bdd aap-operator.v2.4.0-0.1698895982-5dbf596557 aap-operator.v2.4.0-0.1698895982-64d8ccbfbd aap-operator.v2.4.0-0.1698895982-664d466d76 -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:26:02Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698895982
olm.owner.kind: ClusterServiceVersion
name: aap-operator.v2.4.0-0.1698895982-5546965bdd
resourceVersion: "7367376988"
uid: ccdf2f23-d69b-4ce8-8226-75717bf98a72
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:26:07Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698895982
olm.owner.kind: ClusterServiceVersion
name: aap-operator.v2.4.0-0.1698895982-5dbf596557
resourceVersion: "7367377722"
uid: c2dfb8a5-08a1-45f7-a8d9-311feae676bf
rules:
- apiGroups:
- config.openshift.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:26:09Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698895982
olm.owner.kind: ClusterServiceVersion
name: aap-operator.v2.4.0-0.1698895982-64d8ccbfbd
resourceVersion: "7367378061"
uid: 0b5a2dfd-4510-41da-84d6-c76b03d51ce5
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:26:04Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698895982
olm.owner.kind: ClusterServiceVersion
name: aap-operator.v2.4.0-0.1698895982-664d466d76
resourceVersion: "7367377262"
uid: 1d1ba4a4-0432-4a2f-9475-e51b34577b8d
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
The cluster scoped channel installMode allows for AllNamespaces, and all of the privileges are set under .spec.install.spec.clusterPermissions
and there are no .spec.install.spec.permissions
set in this case.
spec:
installModes:
- supported: true
type: OwnNamespace
- supported: true
type: SingleNamespace
- supported: true
type: MultiNamespace
- supported: true <-----
type: AllNamespaces
install:
spec:
clusterPermissions:
- rules:
- apiGroups:
- ''
- rbac.authorization.k8s.io
resources:
- secrets
- pods
- serviceaccounts
- roles
- rolebindings
- services
- services/finalizers
- endpoints
- persistentvolumeclaims
- events
- configmaps
- jobs
- pods/exec
- pods/log
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
...
$ oc get clusterrolebinding | grep -i aap-operator.v2.4.0
aap-operator.v2.4.0-0.1698896316-5fbf474d44 ClusterRole/aap-operator.v2.4.0-0.1698896316-5fbf474d44 57s
aap-operator.v2.4.0-0.1698896316-65bbf47b78 ClusterRole/aap-operator.v2.4.0-0.1698896316-65bbf47b78 57s
aap-operator.v2.4.0-0.1698896316-6dd6b7889b ClusterRole/aap-operator.v2.4.0-0.1698896316-6dd6b7889b 57s
aap-operator.v2.4.0-0.1698896316-745d48485f ClusterRole/aap-operator.v2.4.0-0.1698896316-745d48485f 57s
aap-operator.v2.4.0-0.1698896316-cf46d8cb8 ClusterRole/aap-operator.v2.4.0-0.1698896316-cf46d8cb8 57s
$ oc get clusterrole aap-operator.v2.4.0-0.1698896316-5fbf474d44 aap-operator.v2.4.0-0.1698896316-65bbf47b78 aap-operator.v2.4.0-0.1698896316-6dd6b7889b aap-operator.v2.4.0-0.1698896316-745d48485f aap-operator.v2.4.0-0.1698896316-cf46d8cb8 -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:32:46Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698896316
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: aap
operators.coreos.com/ansible-automation-platform-operator.aap: ""
name: aap-operator.v2.4.0-0.1698896316-5fbf474d44
resourceVersion: "7367411062"
uid: 014cb429-c1b2-4242-8e9a-3723284d88de
rules:
- apiGroups:
- route.openshift.io
resources:
- routes
- routes/custom-host
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- ""
resources:
- pods
- services
- services/finalizers
- serviceaccounts
- endpoints
- persistentvolumeclaims
- events
- configmaps
- secrets
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- rbac.authorization.k8s.io
resources:
- roles
- rolebindings
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- monitoring.coreos.com
resources:
- servicemonitors
verbs:
- get
- create
- apiGroups:
- apps
resourceNames:
- automation-controller-operator
resources:
- deployments/finalizers
verbs:
- update
- apiGroups:
- apps
resources:
- deployments/scale
- statefulsets/scale
verbs:
- patch
- apiGroups:
- ""
resources:
- pods/exec
- pods/attach
- pods/log
verbs:
- create
- get
- apiGroups:
- apps
resources:
- replicasets
verbs:
- get
- create
- apiGroups:
- automationcontroller.ansible.com
resources:
- '*'
- automationcontrollerbackups
- automationcontrollerrestores
verbs:
- '*'
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:32:46Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698896316
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: aap
operators.coreos.com/ansible-automation-platform-operator.aap: ""
name: aap-operator.v2.4.0-0.1698896316-65bbf47b78
resourceVersion: "7367411120"
uid: cf4e6a1f-2d12-4f0d-8307-20d956f4cad5
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
- apiGroups:
- route.openshift.io
resources:
- routes
- routes/custom-host
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
- rbac.authorization.k8s.io
resources:
- secrets
- pods
- pods/exec
- pods/log
- services
- services/finalizers
- endpoints
- persistentvolumeclaims
- events
- configmaps
- roles
- rolebindings
- serviceaccounts
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
- networking.k8s.io
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
- ingresses
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- apiGroups:
- apps
resources:
- deployments/scale
verbs:
- patch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- list
- watch
- apiGroups:
- ""
resourceNames:
- automationhub-operator-sa
resources:
- serviceaccounts
verbs:
- patch
- get
- apiGroups:
- automationhub.ansible.com
resources:
- automationhubs
- automationhubs/status
- automationhubs/finalizers
- automationhubbackups
- automationhubbackups/status
- automationhubbackups/finalizers
- automationhubrestores
- automationhubrestores/status
- automationhubrestores/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:32:46Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698896316
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: aap
operators.coreos.com/ansible-automation-platform-operator.aap: ""
name: aap-operator.v2.4.0-0.1698896316-6dd6b7889b
resourceVersion: "7367411146"
uid: 6b7083e3-7dd0-4612-a182-68c5b56ff99f
rules:
- apiGroups:
- config.openshift.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:32:46Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698896316
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: aap
operators.coreos.com/ansible-automation-platform-operator.aap: ""
name: aap-operator.v2.4.0-0.1698896316-745d48485f
resourceVersion: "7367411482"
uid: f39e89a4-baa5-414c-bbf7-13c740e9e277
rules:
- apiGroups:
- ""
resources:
- secrets
- pods
- pods/exec
- pods/log
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- route.openshift.io
resources:
- routes
- routes/custom-host
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- ""
- rbac.authorization.k8s.io
resources:
- pods
- services
- services/finalizers
- serviceaccounts
- endpoints
- persistentvolumeclaims
- events
- configmaps
- secrets
- roles
- rolebindings
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- apps
- networking.k8s.io
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
- ingresses
verbs:
- get
- list
- create
- delete
- patch
- update
- watch
- apiGroups:
- ""
resources:
- pods/exec
- pods/attach
- pods/log
verbs:
- create
- get
- apiGroups:
- ""
- batch
- extensions
resources:
- jobs
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- eda.ansible.com
resources:
- edas
- edas/status
- edas/finalizers
- edabackups
- edabackups/status
- edabackups/finalizers
- edarestores
- edarestores/status
- edarestores/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2023-11-20T18:32:46Z"
labels:
olm.owner: aap-operator.v2.4.0-0.1698896316
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: aap
operators.coreos.com/ansible-automation-platform-operator.aap: ""
name: aap-operator.v2.4.0-0.1698896316-cf46d8cb8
resourceVersion: "7367411066"
uid: faae47e6-4ce0-4ce3-9ce4-8e67f7a1f2f9
rules:
- apiGroups:
- ""
- rbac.authorization.k8s.io
resources:
- secrets
- pods
- serviceaccounts
- roles
- rolebindings
- services
- services/finalizers
- endpoints
- persistentvolumeclaims
- events
- configmaps
- jobs
- pods/exec
- pods/log
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- monitoring.coreos.com
resources:
- servicemonitors
verbs:
- get
- create
- apiGroups:
- tower.ansible.com
resources:
- ansiblejobs
- ansiblejobs/status
- ansiblejobs/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- tower.ansible.com
resources:
- jobtemplates
- jobtemplates/status
- jobtemplates/finalizers
- ansibleworkflows
- ansibleworkflows/status
- ansibleworkflows/finalizers
- ansibleprojects
- ansibleprojects/status
- ansibleprojects/finalizers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apps
resourceNames:
- resource-operator-controller-manager
resources:
- deployments/finalizers
verbs:
- update
- apiGroups:
- ""
resources:
- pods
- jobs
verbs:
- get
- apiGroups:
- apps
resources:
- replicasets
- deployments
- jobs
verbs:
- get
- apiGroups:
- tower.ansible.com
resources:
- '*'
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- batch
resources:
- '*'
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
kind: List
metadata:
resourceVersion: ""
So I rewrote this from being a bug to being a feature requests. This is a known limitation, just as @Stef16Robbe pointed out. @ctml91 the workaround looks easy enough., good that you linked to an already existing solution.
First, we will have to solve this in this repo, so we can constantly generate the code to the different channels. At the same time we should look on how we can do this in the OLM repos.
We would love PRs and help with this. If you need this functionality straight away, I recommend to use helm or kustomize to install the operator and skip OLM all together.
Describe the bug Users installing the operator via OLM are presented with two options (older version only supported same namespace and came with appropriate RBAC), the two options are all namespaces, or same namespace for the v5 channel.
The older v4 channel only provided the same namespace option
If installing multiple grafana operators in a cluster for various tenants, the same namespace option may be chosen (or in use due to it being the only supported install mode from v4 and prior). In the later versions of v4 and v5 the operator comes with ClusterRole and ClusterRoleBinding which provide it access to the below resources at ClusterScope. This is obviously not a good situation when you have multiple tenants in the same cluster as it provides them privileges access through the Grafana service.
Version v4, v5
To Reproduce Install operator in same namespace mode and login through the SA's token to gain privileged access to the cluster
Expected behavior Installing the operator in same namespace mode should not include any ClusterRole or ClusterRolebinding.
Suspect component/Location where the bug might be occurring
I'm not sure if it's a limitation of OLM and having both install modes supported simply doesn't provide the option to generate the RBAC based on the install mode? Installing in all namespaces of course will require a CR/CRB to access resources outside of the namespace, I'm not sure if it's a limitation of OLM and having both install modes supported simply doesn't provide the option to generate the RBAC based on the install mode? Installing in all namespaces of course will require a CR/CRB to access resources outside of the namespace.
If there is an OLM limitation, I believe it would be best to either only support all namespace, or provide 2 sets of OLM manifests, one for the all namespace mode and another for same namespace.
Runtime (please complete the following information):