search

redhat-cop / cert-utils-operator

Set of functionalities around certificates packaged in a Kubernetes operator
Apache License 2.0
95 stars 35 forks source link

Cert utils operator cannot list apiservices SA #72

Closed rcarrata closed 4 years ago

rcarrata commented 4 years ago

Steps to reproduce Installation from helm as readme reflects

What's the issue

User "system:serviceaccount:cert-utils-operator:cert-utils-operator" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

oc logs -f --tail=10 cert-utils-operator-fbbfb578d-grlqm -n cert-utils-operator
{"level":"info","ts":1596735297.409486,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"webhook_ca_injection_controller","source":"kind source: /, Kind=secret"}
{"level":"info","ts":1596735297.4096828,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"mutatingwebhookconfiguration-controller","source":"kind source: /, Kind=Secret"}
{"level":"info","ts":1596735297.6113288,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"certexpiryalert_controller"}
{"level":"info","ts":1596735297.6128588,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"secretinfo_controller"}
{"level":"info","ts":1596735297.614202,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"secret_to_keystore_controller"}
{"level":"info","ts":1596735297.6157625,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secret_ca_injection_controller","source":"kind source: /, Kind=Secret"}
E0806 17:34:58.661664       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:cert-utils-operator:cert-utils-operator" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
{"level":"error","ts":1596735299.3580818,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"CustomResourceDefinition.apiextensions.k8s.io","error":"no matches for kind \"CustomResourceDefinition\" in version \"apiextensions.k8s.io/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/source/source.go:105\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/manager/internal.go:514"}
{"level":"error","ts":1596735299.3581717,"logger":"cmd","msg":"Manager exited non-zero","error":"no matches for kind \"CustomResourceDefinition\" in version \"apiextensions.k8s.io/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nmain.main\n\t/home/travis/gopath/src/github.com/redhat-cop/cert-utils-operator/cmd/manager/main.go:157\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:203"}
{"level":"error","ts":1596735299.3581867,"logger":"controller-runtime.controller","msg":"Could not wait for Cache to sync","controller":"secretinfo_controller","error":"failed to wait for secretinfo_controller caches to sync","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:181\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/manager/internal.go:514"}
rcarrata commented 4 years ago

After adding the privileges to the SA of cert-utils-operator to list the apiservices, the errors still appearing:

 oc logs --tail=10 -f cert-utils-operator-fbbfb578d-424gd
{"level":"info","ts":1596735645.578445,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"apiservice_controller","source":"kind source: /, Kind=APIService"}
{"level":"info","ts":1596735645.578557,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secretinfo_controller","source":"kind source: /, Kind=Secret"}
{"level":"info","ts":1596735645.6786964,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"route_controller","source":"kind source: /, Kind=Secret"}
{"level":"info","ts":1596735645.678752,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"mutatingwebhookconfiguration-controller","source":"kind source: /, Kind=Secret"}
{"level":"info","ts":1596735645.7793374,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"configmap_to_keystore_controller"}
{"level":"info","ts":1596735645.7801037,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"secret_ca_injection_controller","source":"kind source: /, Kind=Secret"}
{"level":"info","ts":1596735645.7811153,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"secret_to_keystore_controller"}
{"level":"info","ts":1596735645.7823884,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"certexpiryalert_controller"}
{"level":"error","ts":1596735647.6307986,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"CustomResourceDefinition.apiextensions.k8s.io","error":"no matches for kind \"CustomResourceDefinition\" in version \"apiextensions.k8s.io/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/source/source.go:105\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/manager/internal.go:514"}
{"level":"error","ts":1596735647.6308715,"logger":"cmd","msg":"Manager exited non-zero","error":"no matches for kind \"CustomResourceDefinition\" in version \"apiextensions.k8s.io/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nmain.main\n\t/home/travis/gopath/src/github.com/redhat-cop/cert-utils-operator/cmd/manager/main.go:157\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:203"}
mathianasj commented 4 years ago

What is your target platform Kubernetes/OpenShift and version Helm version

raffaelespazzoli commented 4 years ago

looking at the code the permissions have not been added to the helm chart, but only to the OLM template.

rcarrata commented 4 years ago

thanks for your quick response @mathianasj @raffaelespazzoli

target Openshift 

 oc version
Client Version: 4.3.18
Server Version: 4.2.20
Kubernetes Version: v1.14.6+999bb21
helm version
version.BuildInfo{Version:"v3.1+unreleased", GitCommit:"7ebdbb86fca32c77f2fce166f7f9e58ebf7e9946", GitTreeState:"clean", GoVersion:"go1.13.4"}

@raffaelespazzoli I added permissions to the SA, but seems to only fix the error of:

E0806 17:34:58.661664       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:cert-utils-operator:cert-utils-operator" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

any workaround?

raffaelespazzoli commented 4 years ago

on your side your workaround is correct. On our side we need to fix the helm chart.

On Thu, Aug 6, 2020 at 2:07 PM Roberto Carratala notifications@github.com wrote:

thanks for your quick response @mathianasj https://github.com/mathianasj @raffaelespazzoli https://github.com/raffaelespazzoli

target Openshift

oc version Client Version: 4.3.18 Server Version: 4.2.20 Kubernetes Version: v1.14.6+999bb21

helm version version.BuildInfo{Version:"v3.1+unreleased", GitCommit:"7ebdbb86fca32c77f2fce166f7f9e58ebf7e9946", GitTreeState:"clean", GoVersion:"go1.13.4"}

@raffaelespazzoli https://github.com/raffaelespazzoli I added permissions to the SA, but seems to only fix the error of:

E0806 17:34:58.661664 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:cert-utils-operator:cert-utils-operator" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

any workaround?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/cert-utils-operator/issues/72#issuecomment-670090084, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXCENJ7B6UN3FK23GNLR7LWMTANCNFSM4PW2FSQQ .

-- ciao/bye Raffaele

raffaelespazzoli commented 4 years ago

i should be fixed now, can you retry? @rcarrata

rcarrata commented 4 years ago

tested and working perfectly in a new ocp 4.4 cluster:

$ kubectl logs -f --tail=5 cert-utils-operator-5d758488c5-75hh8
{"level":"info","ts":1596794306.9784205,"logger":"ca_injection_controller","msg":"Reconciling configmap","Request.Namespace":"openshift-monitoring","Request.Name":"prometheus-k8s-rulefiles-0"}
{"level":"info","ts":1596794515.298009,"logger":"configmap_to_keystore_controller","msg":"Reconciling ConfigMap","Request.Namespace":"openshift-monitoring","Request.Name":"prometheus-k8s-rulefiles-0"}
{"level":"info","ts":1596794515.2980082,"logger":"ca_injection_controller","msg":"Reconciling configmap","Request.Namespace":"openshift-monitoring","Request.Name":"prometheus-k8s-rulefiles-0"}
{"level":"info","ts":1596794515.8669322,"logger":"configmap_to_keystore_controller","msg":"Reconciling ConfigMap","Request.Namespace":"openshift-monitoring","Request.Name":"prometheus-k8s-rulefiles-0"}
{"level":"info","ts":1596794515.8669326,"logger":"ca_injection_controller","msg":"Reconciling configmap","Request.Namespace":"openshift-monitoring","Request.Name":"prometheus-k8s-rulefiles-0"}

No restarts neither failures detected in the installation / usage of the operator:

 kubectl get pod -n cert-utils-operator
NAME                                   READY     STATUS    RESTARTS   AGE
cert-utils-operator-5d758488c5-75hh8   1/1       Running   0          6m37s

Thanks for your help @raffaelespazzoli @mathianasj !