search

kontena / pharos-cluster

Pharos - The Kubernetes Distribution
https://k8spharos.dev/
Apache License 2.0
311 stars 43 forks source link

[ingress-nginx] RBAG issues with networking.k8s.io #1467

Closed HristoA closed 5 years ago

HristoA commented 5 years ago

What happened: After update to Pharos 2.4.4 i notice problem with ingress-nginx pods

What you expected to happen: To work correctly without problems How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: I fix this by editing ClusterRole file: kubectl -n ingress-nginx edit ClusterRole nginx-ingress-clusterrole and put content of : https://github.com/kubernetes/ingress-nginx/pull/4298/files

Environment:

2019-08-23T14:01:21.229015107Z ------------------------------------------------------------------------------- 2019-08-23T14:01:21.229082107Z NGINX Ingress controller 2019-08-23T14:01:21.229087746Z Release: 0.25.1 2019-08-23T14:01:21.229092121Z Build: git-5179893a9 2019-08-23T14:01:21.229096322Z Repository: https://github.com/kubernetes/ingress-nginx/ 2019-08-23T14:01:21.229100268Z nginx version: openresty/1.15.8.1 2019-08-23T14:01:21.229104074Z 2019-08-23T14:01:21.229107862Z ------------------------------------------------------------------------------- 2019-08-23T14:01:21.229111758Z 2019-08-23T14:01:21.229754953Z W0823 14:01:21.229442 7 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) 2019-08-23T14:01:21.236249198Z nginx version: openresty/1.15.8.1 2019-08-23T14:01:21.237086653Z W0823 14:01:21.236945 7 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. 2019-08-23T14:01:21.237623999Z I0823 14:01:21.237552 7 main.go:183] Creating API client for https://10.96.0.1:443 2019-08-23T14:01:21.266917617Z I0823 14:01:21.266472 7 main.go:227] Running in Kubernetes cluster version v1.14 (v1.14.6) - git (clean) commit 96fac5cd13a5dc064f7d9f4f23030a6aeface6cc - platform linux/amd64 2019-08-23T14:01:21.30114831Z I0823 14:01:21.292735 7 main.go:91] Validated ingress-nginx/default-http-backend as the default backend. 2019-08-23T14:01:21.548451045Z I0823 14:01:21.548260 7 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem 2019-08-23T14:01:21.62869418Z I0823 14:01:21.628473 7 nginx.go:274] Starting NGINX Ingress controller 2019-08-23T14:01:21.676215925Z I0823 14:01:21.675894 7 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"9fe90b7b-c275-11e9-9dc2-9600002dc9d7", APIVersion:"v1", ResourceVersion:"1552", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services 2019-08-23T14:01:21.676620225Z I0823 14:01:21.676449 7 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"9fe20e16-c275-11e9-9dc2-9600002dc9d7", APIVersion:"v1", ResourceVersion:"1550", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration 2019-08-23T14:01:21.679960053Z I0823 14:01:21.679711 7 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"9fe59a54-c275-11e9-9dc2-9600002dc9d7", APIVersion:"v1", ResourceVersion:"1551", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services 2019-08-23T14:01:22.743713505Z E0823 14:01:22.743518 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:23.747175059Z E0823 14:01:23.746946 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:24.750075699Z E0823 14:01:24.749899 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:25.75379265Z E0823 14:01:25.753480 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:26.758211407Z E0823 14:01:26.757938 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:27.763946453Z E0823 14:01:27.763546 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:28.768431732Z E0823 14:01:28.768129 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:29.772429352Z E0823 14:01:29.772278 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:30.759562775Z E0823 14:01:30.759312 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:01:30.776413801Z E0823 14:01:30.776062 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:31.78138786Z E0823 14:01:31.780933 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:32.815586911Z E0823 14:01:32.815312 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:33.818667741Z E0823 14:01:33.818295 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:34.822792407Z E0823 14:01:34.822571 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:35.82680278Z E0823 14:01:35.826508 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:36.830162709Z E0823 14:01:36.829877 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:37.833454043Z E0823 14:01:37.833222 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:38.224988944Z E0823 14:01:38.224727 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:01:38.836458589Z E0823 14:01:38.836193 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:39.842838409Z E0823 14:01:39.842454 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:40.759519271Z E0823 14:01:40.759200 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:01:40.846136788Z E0823 14:01:40.845870 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:41.851015495Z E0823 14:01:41.850735 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:42.85457462Z E0823 14:01:42.854368 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:43.857336985Z E0823 14:01:43.857025 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:44.862757913Z E0823 14:01:44.862549 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:45.865990729Z E0823 14:01:45.865759 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:46.870009874Z E0823 14:01:46.869738 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:47.873789107Z E0823 14:01:47.872981 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:48.230175876Z E0823 14:01:48.225795 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:01:48.877521175Z E0823 14:01:48.877280 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:49.882308222Z E0823 14:01:49.881966 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:50.771683079Z E0823 14:01:50.771431 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:01:50.886117644Z E0823 14:01:50.885760 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:51.890679969Z E0823 14:01:51.890375 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:52.894830461Z E0823 14:01:52.894500 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:53.900387589Z E0823 14:01:53.900232 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:54.905914812Z E0823 14:01:54.905495 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:55.909800651Z E0823 14:01:55.909660 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:56.914078672Z E0823 14:01:56.913772 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:57.918058821Z E0823 14:01:57.917818 7 reflector.go:125] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:179: Failed to list v1beta1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope 2019-08-23T14:01:58.225555467Z E0823 14:01:58.224523 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:01:58.246477401Z I0823 14:01:58.246284 7 main.go:154] Received SIGTERM, shutting down 2019-08-23T14:01:58.246541432Z I0823 14:01:58.246345 7 nginx.go:401] Shutting down controller queues 2019-08-23T14:01:58.246698534Z I0823 14:01:58.246379 7 status.go:117] updating status of Ingress rules (remove) 2019-08-23T14:01:58.247163325Z E0823 14:01:58.247095 7 store.go:183] timed out waiting for caches to sync 2019-08-23T14:01:58.24722211Z I0823 14:01:58.247171 7 nginx.go:318] Starting NGINX process 2019-08-23T14:01:58.26118769Z I0823 14:01:58.247945 7 leaderelection.go:235] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx... 2019-08-23T14:01:58.261225546Z E0823 14:01:58.248310 7 queue.go:78] queue has been shutdown, failed to enqueue: &ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,ManagedFields:[],} 2019-08-23T14:01:58.261239992Z I0823 14:01:58.260773 7 status.go:86] new leader elected: nginx-ingress-controller-vrvkk 2019-08-23T14:01:58.293456813Z I0823 14:01:58.293271 7 nginx.go:417] Stopping NGINX process 2019-08-23T14:01:58.301080534Z 2019/08/23 14:01:58 [notice] 32#32: signal process started 2019-08-23T14:02:00.760389912Z E0823 14:02:00.759981 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:02:01.309545014Z I0823 14:02:01.305417 7 nginx.go:430] NGINX process has stopped 2019-08-23T14:02:01.309610239Z I0823 14:02:01.305472 7 main.go:162] Handled quit, awaiting Pod deletion 2019-08-23T14:02:10.761895759Z E0823 14:02:10.761576 7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory 2019-08-23T14:02:11.310353952Z I0823 14:02:11.309218 7 main.go:165] Exiting with 0