search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.27k stars 8.21k forks source link

bind() to 127.0.0.1:10246 failed (98: Address in use) - during initial startup #7819

Closed Stono closed 2 years ago

Stono commented 2 years ago

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.43.0
  Build:         f3f6da12ac7c59b85ae7132f321bc3bcf144af04
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-gke.1200", GitCommit:"ee714a7b695ca42b9bd0c8fe2c0159024cdcba5e", GitTreeState:"clean", BuildDate:"2021-08-03T09:21:57Z", GoVersion:"go1.15.13b5", Compiler:"gc", Platform:"linux/amd64"}

Environment: GKE We have around 1000 ingress resources on the cluster.

What happened:

During a regular rolling restart of ingress-nginx controller, during initial startup it seemed to get stuck in some sort of configuration loop, repeating until the liveness probe killed it.

Once the liveness probe killed it, it came back up fine.

There are 6 pods, and this only happened on one of them, so it's an ephemeral failure that I am struggling to recreate.

Here are the logs:

.... loads of store.go stuff loading the ingress resources
I1018 12:10:10.650404       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:11.006189       8 status.go:84] "New leader elected" identity="ingress-nginx-external-controller-9d846d8c9-zw97q"
I1018 12:10:11.019554       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:11.019641       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:11.019740       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:11 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:11 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:12.020606       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:12.020641       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:12.020663       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="initial-sync"
2021/10/18 12:10:11 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:11 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:11 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:11 [emerg] 33#33: still could not bind()
I1018 12:10:13.984048       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:14.431280       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:14.431360       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:14.431408       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:14 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:14 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:15.432141       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:15.432181       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:15.432202       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="user-preferences-service/app"
2021/10/18 12:10:14 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:14 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:16 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:10:16.454416       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:10:14 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:14 [emerg] 33#33: still could not bind()
I1018 12:10:17.315466       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:17.731203       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:17.731286       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:17.731361       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:17 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:17 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:18.732092       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:18.732131       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:18.732158       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="bulk-retail-check-service/admin"
2021/10/18 12:10:17 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:17 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:17 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:17 [emerg] 33#33: still could not bind()
I1018 12:10:20.648968       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:21.065603       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:21.065660       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:21.065715       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:21 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:21 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:22.066507       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:22.066544       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:22.066566       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="kafka/zookeeper-2"
2021/10/18 12:10:21 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:21 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:21 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:21 [emerg] 33#33: still could not bind()
I1018 12:10:23.982000       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:24.421569       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:24.421632       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:24.421679       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:24 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:24 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:25.422378       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:25.422418       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:25.422437       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="api-admin/app"
2021/10/18 12:10:24 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:24 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:24 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:24 [emerg] 33#33: still could not bind()
I1018 12:10:27.315015       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:27.719793       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:27.719942       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:27.720089       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:27 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:27 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:28.720673       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:28.720718       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:28.720745       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="dealer-service/app"
2021/10/18 12:10:27 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:27 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:27 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:27 [emerg] 33#33: still could not bind()
I1018 12:10:30.649338       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:31.091750       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:31.091821       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:31.091883       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:31 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:31 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:10:31.453662       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:10:31 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:32.092576       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:32.092645       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:32.092671       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="dealerauction-sale-notification/admin"
2021/10/18 12:10:31 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:31 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:31 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:31 [emerg] 33#33: still could not bind()
I1018 12:10:33.982160       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:34.410348       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:34.410411       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:34.410503       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:34 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:34 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:35.411168       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:35.411206       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:35.411224       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="sauron-leasing-app/redis-commander"
2021/10/18 12:10:34 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:34 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:34 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:34 [emerg] 33#33: still could not bind()
I1018 12:10:37.315769       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:37.727971       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:37.728025       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:37.728096       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:37 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:37 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:38.728805       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:38.728837       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:38.728869       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="vds-acceptor/sftp"
2021/10/18 12:10:37 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:37 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:37 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:37 [emerg] 33#33: still could not bind()
I1018 12:10:40.648752       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:41.089793       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:41.089889       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:41.089971       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:41 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:41 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:42.093840       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:42.093897       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:42.093921       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="audience-brand/app"
2021/10/18 12:10:41 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:41 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:41 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:41 [emerg] 33#33: still could not bind()
I1018 12:10:43.983576       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:44.424389       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:44.424442       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:44.424492       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:44 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:44 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:45.425248       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:45.425295       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:45.425321       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="sauron-search-app/app"
2021/10/18 12:10:44 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:44 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:46 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:10:46.458328       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:10:44 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:44 [emerg] 33#33: still could not bind()
I1018 12:10:47.315418       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:47.725272       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:47.725342       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:47.725433       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:47 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:47 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:48.726124       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:48.726158       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:48.726177       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="data-platform/prometheus-pushgateway"
2021/10/18 12:10:47 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:47 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:47 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:47 [emerg] 33#33: still could not bind()
I1018 12:10:50.648595       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:51.081696       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:51.081763       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:51.081986       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:51 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:51 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:52.082476       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:52.082511       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:52.082535       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="external-webhook-forwarder/app"
2021/10/18 12:10:51 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:51 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:51 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:51 [emerg] 33#33: still could not bind()
I1018 12:10:53.981885       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:54.410942       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:54.411015       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:54.411036       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:54 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:54 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:55.411722       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:55.411756       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:55.411775       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="historic-stock-service/app"
2021/10/18 12:10:54 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:54 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:54 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:54 [emerg] 33#33: still could not bind()
I1018 12:10:57.315243       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:10:57.725036       8 controller.go:161] "Backend successfully reloaded"
I1018 12:10:57.725175       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:10:57.725201       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:10:57 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:57 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:10:58.725889       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:58.725928       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:10:58.725951       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="istio-test-app-1/app"
2021/10/18 12:10:57 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:57 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:57 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:10:57 [emerg] 33#33: still could not bind()
I1018 12:11:00.648868       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:11:01.071120       8 controller.go:161] "Backend successfully reloaded"
I1018 12:11:01.071191       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:11:01.071288       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:11:01 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:01 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:11:01.454472       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:11:01 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:11:02.071907       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:11:02.071938       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:11:02.071960       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="permissions-consumer/app"
2021/10/18 12:11:01 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:01 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:01 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:01 [emerg] 33#33: still could not bind()
I1018 12:11:03.982613       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:11:04.426949       8 controller.go:161] "Backend successfully reloaded"
I1018 12:11:04.427002       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:11:04.427018       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:11:04 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:04 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:11:05.427893       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:11:05.427925       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:11:05.427942       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="search-one-read/app"
2021/10/18 12:11:04 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:04 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:04 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:04 [emerg] 33#33: still could not bind()
I1018 12:11:07.315461       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:11:07.715160       8 controller.go:161] "Backend successfully reloaded"
I1018 12:11:07.715248       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:11:07.715333       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:11:07 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:07 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
W1018 12:11:08.715987       8 controller.go:190] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:11:08.716025       8 controller.go:194] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E1018 12:11:08.716048       8 queue.go:130] "requeuing" err="Post \"http://127.0.0.1:10246/configuration/backends\": dial tcp 127.0.0.1:10246: connect: connection refused" key="vds-admin/app"
2021/10/18 12:11:07 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:07 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:07 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:07 [emerg] 33#33: still could not bind()
I1018 12:11:10.648602       8 controller.go:144] "Configuration changes detected, backend reload required"
I1018 12:11:11.079697       8 controller.go:161] "Backend successfully reloaded"
I1018 12:11:11.079775       8 controller.go:172] "Initial sync, sleeping for 1 second"
I1018 12:11:11.079825       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-external-controller-b45b65f5f-xwjfx", UID:"7cac186a-0e87-4c67-901f-054878e8cbf5", APIVersion:"v1", ResourceVersion:"1340290783", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2021/10/18 12:11:11 [emerg] 33#33: bind() to 127.0.0.1:10246 failed (98: Address in use)
2021/10/18 12:11:11 [error] 334#334: *11 [lua] certificate.lua:240: call(): certificate not found, falling back to fake certificate for hostname: _, context: ssl_certificate_by_lua*, client: 35.191.2.156, server: 0.0.0.0:443
2021/10/18 12:11:11 [error] 332#332: *13 [lua] certificate.lua:240: call(): certificate not found, falling back to fake certificate for hostname: _, context: ssl_certificate_by_lua*, client: 35.191.2.150, server: 0.0.0.0:443
2021/10/18 12:11:12 [error] 332#332: *15 [lua] certificate.lua:240: call(): certificate not found, falling back to fake certificate for hostname: _, context: ssl_certificate_by_lua*, client: 35.191.2.148, server: 0.0.0.0:443
I1018 12:12:03.760824       8 main.go:187] "Received SIGTERM, shutting down"
I1018 12:12:03.760864       8 nginx.go:370] "Shutting down controller queues"
W1018 12:12:03.760887       8 status.go:110] skipping update of status of Ingress rules
I1018 12:12:03.760893       8 nginx.go:386] "Stopping NGINX process"
2021/10/18 12:12:03 [notice] 508#508: signal process started
2021/10/18 12:12:16 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:12:16.865500       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:12:31 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:12:31.454099       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:12:46 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:12:46.453500       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:13:01 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:13:01.454247       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:13:16 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:13:16.454058       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:13:31 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:13:31.453417       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:13:46 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:13:46.486273       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:14:01 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:14:01.454448       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:14:16 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:14:16.453928       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:14:31 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:14:31.453874       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:14:46 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:14:46.453931       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:15:01 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:15:01.453076       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:15:16 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:15:16.457454       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:15:31 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:15:31.453568       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:15:46 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:15:46.469768       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
2021/10/18 12:16:01 Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused
W1018 12:16:01.456493       8 nginx_status.go:172] unexpected error obtaining nginx status info: Get "http://127.0.0.1:10246/nginx_status": dial tcp 127.0.0.1:10246: connect: connection refused

What you expected to happen:

Not to crash

How to reproduce it: Unable to reproduce unfortunately!

Anything else we need to know: Noticed this issue: https://github.com/kubernetes/ingress-nginx/issues/4616 - but we don't use hostNetwork

/kind bug

k8s-ci-robot commented 2 years ago

@Stono: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 2 years ago

/remove-kind bug /kind support /triage needs-information

Please upgrade to a recent release (prefer the latest) and check if the problem happens

Evesy commented 2 years ago

@longwuyuan Upgrading is a bit tricky due to https://github.com/kubernetes/ingress-nginx/issues/7753

longwuyuan commented 2 years ago

There have been many checkins/fixes since the version you are using, so unsure how much progress can be made and at what speed on this

timblaktu commented 2 years ago

I'm seeing a similar issue with an ingress-nginx pod installed via this repo's helm chart with the following helm commands:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx-ingress ingress-nginx/ingress-nginx --namespace ingress --set controller.replicaCount=2 --values helm/nginx-values.yaml

where helm/nginx-values.yaml contains the following to workaround the fargate privilege escalation / port issue:

controller: 
  extraArgs: 
    http-port: 8080 
    https-port: 8443 
  containerPort: 
    http: 8080 
    https: 8443 
  service: 
    ports: 
      http: 80 
      https: 443 
    targetPorts: 
      http: 8080 
      https: 8443 
  image: 
    allowPrivilegeEscalation: false
    # https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes
    livenessProbe:
      initialDelaySeconds: 1200  # 30
    readinessProbe:
      initialDelaySeconds: 1200  # 0
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

..and the apparent chart versions:

> helm list -n ingress
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
nginx-ingress   ingress         1               2021-11-12 19:55:30.921251692 +0000 UTC deployed        ingress-nginx-4.0.6     1.0.4

and:

> kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Environment is a fresh Fargate-only EKS cluster that was created with 3 public and 3 private subnets.

Regardless of my probe initialDelay settings, I encounter perpetual "CrashLoopBackOff" and my pod/container logs are littered with:

2021/11/12 20:06:19 [emerg] 28#28: bind() to 0.0.0.0:8443 failed (98: Address in use)
2021/11/12 20:06:19 [emerg] 28#28: bind() to [::]:8443 failed (98: Address in use)

As I describe here I'm able to kubectl exec into the running container and use netstat to confirm that indeed 0/INADDR_ANY:8443 is already bound, but bc I can't get in as root, I cannot see what the process is that's doing it.

Can someone shed some light on why the applications in my ingress-nginx containers are failing to do the most basic requirement, binding to port 8443?

I linked to the fargate privilege escalation / port issue bc it feels related. Thanks.

EDIT: I ended up creating this new issue for what I describe above, but it has been resolved due to another cause.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7819#issuecomment-1100427116): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.