Closed kundan2707 closed 4 years ago
/kind triage/support
@kundan2707: The label(s) kind/triage/support
cannot be applied, because the repository doesn't have them
ingress-nginx ingress-nginx-admission-create-6sph6 1/1 Running 2 86s ingress-nginx ingress-nginx-admission-patch-ks2cm 1/1 Running 2 86s
The secret is created after these two jobs finish. After that, the pod will start. Keep in mind this only happens the first time you start the ingress controller
Could be related to this item: #5863
ingress-nginx ingress-nginx-admission-create-6sph6 1/1 Running 2 86s ingress-nginx ingress-nginx-admission-patch-ks2cm 1/1 Running 2 86s
The secret is created after these two jobs finish. After that, the pod will start. Keep in mind this only happens the first time you start the ingress controller
Thses jobs got finished still same error exists
vagrant@k8s-head:~$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-7fd7d8df56-wpdnp 0/1 ContainerCreating 0 5d21h
kube-system calico-kube-controllers-75d56dfc47-x9qgx 1/1 Running 1 107d
kube-system calico-node-8nv6v 1/1 Running 1 107d
kube-system calico-node-gjv4m 1/1 Running 1 107d
kube-system calico-node-pttqk 1/1 Running 1 107d
kube-system coredns-66bff467f8-dgjr6 1/1 Running 1 107d
kube-system coredns-66bff467f8-tg8zf 1/1 Running 1 107d
kube-system etcd-k8s-head 1/1 Running 1 107d
kube-system kube-apiserver-k8s-head 1/1 Running 1 107d
kube-system kube-controller-manager-k8s-head 1/1 Running 1 107d
kube-system kube-proxy-5f9n9 1/1 Running 1 107d
kube-system kube-proxy-dn947 1/1 Running 1 107d
kube-system kube-proxy-pdqtt 1/1 Running 1 107d
kube-system kube-scheduler-k8s-head 1/1 Running 1 107d
metallb-system controller-57f648cb96-9tjc6 1/1 Running 0 7d
metallb-system speaker-5wzfq 1/1 Running 0 7d
metallb-system speaker-cwxb4 1/1 Running 0 7d
metallb-system speaker-vpp9z 1/1 Running 0 7d
vagrant@k8s-head:~$ kubectl describe pod ingress-nginx-controller-7fd7d8df56-wpdnp -n ingress-nginx
Name: ingress-nginx-controller-7fd7d8df56-wpdnp
Namespace: ingress-nginx
Priority: 0
Node: k8s-node-1/192.168.205.11
Start Time: Fri, 24 Jul 2020 01:49:42 +0000
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=7fd7d8df56
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/ingress-nginx-controller-7fd7d8df56
Containers:
controller:
Container ID:
Image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--ingress-class=nginx
--configmap=ingress-nginx/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-7fd7d8df56-wpdnp (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-4t6nq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
ingress-nginx-token-4t6nq:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-token-4t6nq
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 21m (x789 over 5d21h) kubelet, k8s-node-1 Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[ingress-nginx-token-4t6nq webhook-cert]: timed out waiting for the condition
Warning FailedMount 5m14s (x2946 over 5d21h) kubelet, k8s-node-1 Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert ingress-nginx-token-4t6nq]: timed out waiting for the condition
Warning FailedMount 43s (x4172 over 5d21h) kubelet, k8s-node-1 MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
same problem after: "kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/cloud/deploy.yaml"
tried doing the reset to kubernetes cluster, but didnt resolve.
edit: doing a reset to factory default in docker, resolved it.
same problem after: "kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/cloud/deploy.yaml"
tried doing the reset to kubernetes cluster, but didnt resolve.
edit: doing a reset to factory default in docker, resolved it.
Reset docker ??? Has other methods?
I have the exact same issue with kuberenetes 1.19 on Centos 8 with Calico and Metallb. I tried the resetting docker with not resolution.
LAST SEEN TYPE REASON OBJECT MESSAGE 4m31s Normal Scheduled pod/ingress-nginx-admission-create-9fxjm Successfully assigned ingress-nginx/ingress-nginx-admission-create-9fxjm to k8s-node 4m26s Normal Pulling pod/ingress-nginx-admission-create-9fxjm Pulling image "docker.io/jettech/kube-webhook-certgen:v1.3.0" 4m1s Normal Pulled pod/ingress-nginx-admission-create-9fxjm Successfully pulled image "docker.io/jettech/kube-webhook-certgen:v1.3.0" in 25.717594326s 26s Normal Created pod/ingress-nginx-admission-create-9fxjm Created container create 26s Normal Started pod/ingress-nginx-admission-create-9fxjm Started container create 28s Normal Pulled pod/ingress-nginx-admission-create-9fxjm Container image "docker.io/jettech/kube-webhook-certgen:v1.3.0" already present on machine 41s Warning BackOff pod/ingress-nginx-admission-create-9fxjm Back-off restarting failed container 4m31s Normal SuccessfulCreate job/ingress-nginx-admission-create Created pod: ingress-nginx-admission-create-9fxjm 4m31s Normal Scheduled pod/ingress-nginx-admission-patch-74vqw Successfully assigned ingress-nginx/ingress-nginx-admission-patch-74vqw to k8s-node 4m26s Normal Pulling pod/ingress-nginx-admission-patch-74vqw Pulling image "docker.io/jettech/kube-webhook-certgen:v1.3.0" 3m57s Normal Pulled pod/ingress-nginx-admission-patch-74vqw Successfully pulled image "docker.io/jettech/kube-webhook-certgen:v1.3.0" in 28.63731542s 22s Normal Created pod/ingress-nginx-admission-patch-74vqw Created container patch 21s Normal Started pod/ingress-nginx-admission-patch-74vqw Started container patch 22s Normal Pulled pod/ingress-nginx-admission-patch-74vqw Container image "docker.io/jettech/kube-webhook-certgen:v1.3.0" already present on machine 33s Warning BackOff pod/ingress-nginx-admission-patch-74vqw Back-off restarting failed container 4m31s Normal SuccessfulCreate job/ingress-nginx-admission-patch Created pod: ingress-nginx-admission-patch-74vqw 4m32s Normal Scheduled pod/ingress-nginx-controller-785557f9c9-776tg Successfully assigned ingress-nginx/ingress-nginx-controller-785557f9c9-776tg to k8s-node 22s Warning FailedMount pod/ingress-nginx-controller-785557f9c9-776tg MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not fou nd 2m29s Warning FailedMount pod/ingress-nginx-controller-785557f9c9-776tg Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webh ook-cert ingress-nginx-token-7gt7r]: timed out waiting for the condition 13s Warning FailedMount pod/ingress-nginx-controller-785557f9c9-776tg Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[ingr ess-nginx-token-7gt7r webhook-cert]: timed out waiting for the condition 4m32s Normal SuccessfulCreate replicaset/ingress-nginx-controller-785557f9c9 Created pod: ingress-nginx-controller-785557f9c9-776tg 4m33s Normal ScalingReplicaSet deployment/ingress-nginx-controller Scaled up replica set ingress-nginx-controller-785557f9c9 to 1
same problem after: "kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/cloud/deploy.yaml" tried doing the reset to kubernetes cluster, but didnt resolve. edit: doing a reset to factory default in docker, resolved it.
Reset docker ??? Has other methods?
Same
@aliencrash @FreddieMcHeart @yexk please use v0.41.0
I managed to fix this by modifying --ingress-class=nginx to - --ingress-class=nginx2 in apply.yaml file so it was because there was an ingress class with the same name "nginx"
I encountered the same problem in version 1.18. My solution was to restart the Docker of the machine where Webhook is located, and the problem was solved. I felt that there was a problem with docker's network bridge
Just ran into this too on EKS 1.17 (latest AMI) with helm chart v3.20.0
Anyone got this working? I am stuck in this same issue and not able to resolve ...read almost all the related issues/solutions
I am also stuck here
Still stucked. K8s 1.21 ... MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
same here
kubectl apply -f https://github.com/kubernetes/ingress-nginx/tree/master/deploy/static/provider/baremetal/deploy.yaml
[root@ck8s1 ingress]# kubectl describe pod/ingress-nginx-controller-7b78df5bb4-xrhsm -n ingress-nginx
Name: ingress-nginx-controller-7b78df5bb4-xrhsm
Namespace: ingress-nginx
Priority: 0
Node: ck8s2/192.168.43.152
Start Time: Fri, 23 Apr 2021 14:12:36 +0800
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=7b78df5bb4
Annotations:
IPs:
Image: k8s.gcr.io/ingress-nginx/controller:v0.45.0@sha256:c4390c53f348c3bd4e60a5dd6a11c35799ae78c49388090140b9d72ccede1755
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-7b78df5bb4-xrhsm (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-rvr6f (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
ingress-nginx-token-rvr6f:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-token-rvr6f
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal Scheduled 3m6s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-7b78df5bb4-xrhsm to ck8s2 Warning FailedMount 63s kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert ingress-nginx-token-rvr6f]: timed out waiting for the condition Warning FailedMount 58s (x9 over 3m5s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
The Same Questions,help
still happend in v0.45.0,somebody help
still happend in v0.46.0,somebody help
I am getting the same issues in AKS on k8s version 1.19
Whys the issue is closed?
Whys the issue is closed?
Can we have this issue reopened please?
This fix solve the issue for me: < --ingress-class=nginx> renamed to < --ingress-class=nginx2 > by editing the deployment as told by @AliBeddah
I had exact same issues when I was using Kubernetes 1.18 and finally when I upgraded to 1.19 it worked just fine
Pod status
k get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-cgpj7 0/1 Completed 0 3m44s
ingress-nginx-admission-patch-mksxs 0/1 Completed 0 3m44s
ingress-nginx-controller-5fb6f67b9c-ps67k 0/1 CrashLoopBackOff 5 3m45s
Error logs from pod
I0916 07:15:34.317477 8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
F0916 07:15:34.318721 8 main.go:107] ingress-nginx requires Kubernetes v1.19.0 or higher
k get po -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-2tk8p 0/1 Completed 0 104s
ingress-nginx-admission-patch-nlv5w 0/1 Completed 0 104s
ingress-nginx-controller-79c4d49bb9-7bgcj 1/1 Running 0 105s
Same on me here!
@iuli72an can you share kubernetes and all related versions in your case ?
Same problem, but using Ubuntu 18.04. Moved to Ubuntu 20.04 and now it works fine
can comment it?
I change the yaml file . secretName: ingress-nginx-admission secretName: ingress-nginx-admission-token-l2nx8
IT can work!!
add, at the second date .
I find a ture answaer, i belive.
delete all the svc. pods deployment , secret ,rs, cs, use yaml file when you apply except kube-system namespace
like the under.
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml
and apply the new
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml
and everthing looks like good .
Finally, this worked for me. Run kubectl edit configmap kube-proxy -n kube-system
and set maxPerCore
under conntrack
to 0
from null
.
I am getting same issue with ingress-nginx-admission secret, tried editing create and patch job with ingress-ingress-nginx-admission-token-xxxxx still same error. Anyone have solution for this, i am using Chart version 3.11.1 ingress-nginx. Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. {"err":"secrets \"ingress-nginx-admission-token-xxxxx\" is forbidden: User \"system:serviceaccount:ingress-nginx:ingress-nginx-admission\" cannot get resource \"secrets\" in API group \"\" in the namespace \"ingress-nginx\"","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:109","time":"2021-11-23T22:05:01Z"}
When deploying with helm job are not deploying but ingress-controller is up and running, if you are using template of this chart then everything gets crashed
in my case, if i install the ingress controller in control-plane, everthing looks good but i have no idea why
i am get same issue. the ingress-nginx-controller pod is running but ingress-admission-create and ingress-admission-patch pod status is completed. i use kubectl logs ingress-addmission-create command that i get
"err":"secrets \"ingress-nginx-admission\" not foundet some errors
so,what should i do for this?
I am also get same issue, you can refer to my solution.
怎么搞,没找到问题点啊,没找到答案啊
我也遇到同样的问题,可以参考我的解决方法。
- 添加“hostNetwork:true”,即可获取pod ip。
- 更改部署“ingress-nginx-controller”secretName。
我也遇到同样的问题,可以参考我的解决方法。
- 添加“hostNetwork:true”,自动获取pod ip。
- 更改部署“ingress-nginx-controller”secretName。
我没成功欸
I handled it by renaming ingress-nginx-admission-token-xxxxxx secret to ingress-nginx-admission and then delete controller pod to recreate it.
怎么搞,没找到问题点啊,没找到答案啊
兄弟 解决了没
原来是拉不到镜像,改下镜像仓库就可以了
@zhujintao 我把镜像拉去到本地,上传到云主机。还是不行。
原来是拉不到镜像,改下镜像仓库就可以了
Same issue
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx pod/ingress-nginx-admission-create-cdqqg 0/1 CrashLoopBackOff 1 (2s ago) 6s
ingress-nginx pod/ingress-nginx-admission-patch-nwtnf 0/1 CrashLoopBackOff 1 (2s ago) 6s
ingress-nginx pod/ingress-nginx-controller-688f665f6-tnwpk 0/1 ContainerCreating 0 6s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m53s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-688f665f6-tnwpk to minions11
Warning FailedMount 4m50s kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube-api-access-wgn28 webhook-cert]: timed out waiting for the condition
Warning FailedMount 41s (x11 over 6m53s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Warning FailedMount 15s (x2 over 2m32s) kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-wgn28]: timed out waiting for the condition
[root@master ingress-nginx]#
[root@master ingress-nginx]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:27:13Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
[root@master ingress-nginx]#
tried
Finally, this worked for me. Run kubectl edit configmap kube-proxy -n kube-system and set maxPerCore under conntrack to 0 from null.
[root@master ingress-nginx]#
[root@master ingress-nginx]# grep nginx2 deploy.yaml
- --ingress-class=nginx2
[root@master ingress-nginx]#
[root@master ingress-nginx]#
I also tried kubectl reset ..no luck
I went into this problem too, and I keep the namespace same to solve it.
< --ingress-class=nginx> renamed to < --ingress-class=nginx2 > by editing the deployme
我的这样解决了 v1.22 ; 在新版本未解决 v1.26
NGINX Ingress controller version:0.34.1
Kubernetes version (use kubectl version):1.18.1
Environment:
Cloud provider or hardware configuration:baremetal OS (e.g. from /etc/os-release):ubuntu16.04 Kernel (e.g.
uname -a
):4.4.0-184-generic.x86_64 Install tools:deploy.yamlWhat happened: ingress-nginx-admission-create and ingress-nginx-admission-patch started intially but Ingress-nginx controller failed to start
What you expected to happen: Ingress-nginx controller should start successfully
How to reproduce it:
Install kubernetes cluster using kubeadm
Install the ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml
Anything else we need to know: