search

alexellis / registry-creds

Replicate Kubernetes ImagePullSecrets to all namespaces
MIT License
338 stars 37 forks source link

Cannot run registry-creds with OOTB manifest - fails with `error initially creating leader election record` #37

Closed mylesagray closed 2 years ago

mylesagray commented 2 years ago

Describe the bug K8s clusters tested on ARM from v1.18 thru v1.21, x86 vanilla with the same versions - and multiple Tanzu cluster versions, this failure is also described in the PR #31 in which is fails on IKS on IBM Cloud.

When applying the manifest.yaml via the following - the system will never start up with a failed leader election:

k apply -f https://raw.githubusercontent.com/alexellis/registry-creds/master/manifest.yaml

Error from the pod:

E1210 14:13:51.146061       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"

To Reproduce Create KinD cluster:

$ kind create cluster --image=kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.11) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
$ kubectx kind-kind
Switched to context "kind-kind".
$ k get po -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-f9fd979d6-7zxdq                      1/1     Running   0          63s
kube-system          coredns-f9fd979d6-8z6xq                      1/1     Running   0          63s
kube-system          etcd-kind-control-plane                      1/1     Running   0          71s
kube-system          kindnet-bnjnl                                1/1     Running   0          63s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          71s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          71s
kube-system          kube-proxy-vfd7l                             1/1     Running   0          63s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          71s
local-path-storage   local-path-provisioner-547f784dff-mrcn8      1/1     Running   0          63s

Install reg-creds:

$ k apply -f https://raw.githubusercontent.com/alexellis/registry-creds/master/manifest.yaml
namespace/registry-creds-system created
customresourcedefinition.apiextensions.k8s.io/clusterpullsecrets.ops.alexellis.io created
role.rbac.authorization.k8s.io/registry-creds-leader-election-role created
clusterrole.rbac.authorization.k8s.io/registry-creds-registry-creds-role created
rolebinding.rbac.authorization.k8s.io/registry-creds-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/registry-creds-registry-creds-rolebinding created
deployment.apps/registry-creds-registry-creds-controller created
$  kubens registry-creds-system
Context "kind-kind" modified.
Active namespace is "registry-creds-system".
$ k get po -w
NAME                                                        READY   STATUS              RESTARTS   AGE
registry-creds-registry-creds-controller-68468bd469-dtjxf   0/1     ContainerCreating   0          8s
registry-creds-registry-creds-controller-68468bd469-dtjxf   1/1     Running             0          12s

Check reg-creds pod logs:

$ k logs registry-creds-registry-creds-controller-68468bd469-dtjxf
2021-12-10T14:13:51.132Z    INFO    controller-runtime.metrics  metrics server is starting to listen    {"addr": ":8080"}
2021-12-10T14:13:51.133Z    INFO    setup   starting manager with the version %s and commit %s  {"0.3.0-rc4": "5883ef9ba8e72563b3c4ceee23647ed27c7713fb"}
I1210 14:13:51.133985       1 leaderelection.go:242] attempting to acquire leader lease  registry-creds-system/8bdecb1a.alexellis.io...
2021-12-10T14:13:51.134Z    INFO    controller-runtime.manager  starting metrics server {"path": "/metrics"}
E1210 14:13:51.146061       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:13:54.603970       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:13:58.869250       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:02.471738       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:05.531691       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:08.557895       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:12.215142       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:14.381499       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:16.763812       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:14:19.005162       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"

Expected behavior The leader election to succeed and the system to run.

List all possible solutions, and your suggested option

Removing the lines: https://github.com/alexellis/registry-creds/blob/873e47849c983d48e16f27c3e38947f941a3288d/manifest.yaml#L88-L89

Solves the issue instantly:

E1210 14:22:31.466596       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:22:34.726013       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1210 14:22:36.976019       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
<!!!!! ------ kubectl edit role registry-creds-leader-election-role - executed here to remove the above lines ------- !!!!!>
I1210 14:22:39.356030       1 leaderelection.go:252] successfully acquired lease registry-creds-system/8bdecb1a.alexellis.io
2021-12-10T14:22:39.356Z    DEBUG   controller-runtime.manager.events   Normal  {"object": {"kind":"ConfigMap","namespace":"registry-creds-system","name":"8bdecb1a.alexellis.io","uid":"b1805b5f-0c0e-40a8-a555-fbc91916d2b7","apiVersion":"v1","resourceVersion":"2102"}, "reason": "LeaderElection", "message": "registry-creds-registry-creds-controller-68468bd469-dtjxf_13eadec4-12c6-44aa-8b2c-bf9107cf5df3 became leader"}
2021-12-10T14:22:39.357Z    INFO    controller-runtime.controller   Starting EventSource    {"controller": "serviceaccount", "source": "kind source: /, Kind="}
2021-12-10T14:22:39.358Z    INFO    controller-runtime.controller   Starting EventSource    {"controller": "clusterpullsecret", "source": "kind source: /, Kind="}

My comment here shows the same response as this being the fix for multiple users: https://github.com/alexellis/registry-creds/issues/16#issuecomment-778186442

alexellis commented 2 years ago

Thanks for doing this. I was running the operator through go, so not seeing this error.

I've patched it a the source, instead of in the generated files at: 873e478

If you and @frundh want to use this software and see it maintained, may I suggest that you become sponsors on GitHub? Pick whatever tier you think is reasonable for the value and time savings you are getting.

Alex

alexellis commented 2 years ago

Closing as fixed

mylesagray commented 2 years ago

I appreciate you fixing it in the source - but the spirit of this issue was that the manifest.yaml is broken OOTB for anyone deploying it and i'm sure @frundh would agree here as this was also the spirit of his PR - so I would argue that this should be re-opened, even as an FYI for other people that will inevitably hit this issue if indeed there is an upstream issue that needs fixed.

Right now, I am just pointing people that use it to a manifest I have forked and removed the offending lines from to get them running.

alexellis commented 2 years ago

It's fixed? What problem are you running into now?

mylesagray commented 2 years ago

The same one as before - repo info below:

❯ kind create cluster --image=kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.11) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

❯ k apply -f https://raw.githubusercontent.com/alexellis/registry-creds/master/manifest.yaml
namespace/registry-creds-system created
customresourcedefinition.apiextensions.k8s.io/clusterpullsecrets.ops.alexellis.io created
role.rbac.authorization.k8s.io/registry-creds-leader-election-role created
clusterrole.rbac.authorization.k8s.io/registry-creds-registry-creds-role created
rolebinding.rbac.authorization.k8s.io/registry-creds-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/registry-creds-registry-creds-rolebinding created
deployment.apps/registry-creds-registry-creds-controller created

❯ k get po -n registry-creds-system -w
NAME                                                        READY   STATUS              RESTARTS   AGE
registry-creds-registry-creds-controller-75d6696656-djskn   0/1     ContainerCreating   0          12s
registry-creds-registry-creds-controller-75d6696656-djskn   1/1     Running             0          13s
^C%

❯ k logs -n registry-creds-system registry-creds-registry-creds-controller-75d6696656-djskn -f
2021-12-13T13:56:08.943Z    INFO    controller-runtime.metrics  metrics server is starting to listen    {"addr": ":8080"}
2021-12-13T13:56:08.944Z    INFO    setup   starting manager with the version %s and commit %s  {"0.3.1": "9a617dafeb621c0dc09bb133b6163f68afc920cf"}
I1213 13:56:08.945026       1 leaderelection.go:242] attempting to acquire leader lease  registry-creds-system/8bdecb1a.alexellis.io...
2021-12-13T13:56:08.945Z    INFO    controller-runtime.manager  starting metrics server {"path": "/metrics"}
E1213 13:56:08.976261       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:12.437712       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:16.707244       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:20.311260       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:23.368801       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:26.402875       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:30.060973       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:32.232297       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:34.618701       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:36.865718       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:39.596546       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
E1213 13:56:42.841256       1 leaderelection.go:335] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:registry-creds-system:default" cannot create resource "configmaps" in API group "" in the namespace "registry-creds-system"
alexellis commented 2 years ago

Bizarre. I fixed it, it worked fine for me and I mentioned that when closing. I'll have another look.

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

$ source ~/.docker-creds
$ 
$ kubectl create secret docker-registry registry-creds \
>   --namespace kube-system \
>   --docker-username=$DOCKER_USERNAME \
>   --docker-password=$DOCKER_PASSWORD \
>   --docker-email=$DOCKER_EMAIL
secret/registry-creds created
$ 

$ kubectl apply -f ./manifest.yaml 

$ kubectl apply -f ../test.yaml 
clusterpullsecret.ops.alexellis.io/dockerhub-registry-creds created

$ kubectl create ns test
namespace/test created
$ kubectl get secret -n test
NAME                       TYPE                                  DATA   AGE
default-token-fcvdb        kubernetes.io/service-account-token   3      3s
dockerhub-registry-creds   kubernetes.io/dockerconfigjson        1      3s
$

But are you still going to keep using this without sponsoring me for the time and support I'm providing?