Closed 0xThiebaut closed 5 years ago
aledbf
commented
5 years ago I0313 17:06:33.884525 9 main.go:200] Creating API client for https://172.17.0.1:443
If that's the last line you see in the log it means the ingress controller cannot reach the API server (your node have some networking issue, like CNI plugin or restriction)
Adding the flag --v=10 in the ingress controller deployment you will be able to see the connection attempts.
0xThiebaut
commented
5 years ago Any suggestion on how to modify the API port? I've attempted to define an environment variable as suggested in #3365 due to extra args not working:
helm install --name nginx stable/nginx-ingress --set controller.kind=DaemonSet,controller.daemonset.useHostPort=true,controller.service.type=ClusterIP,controller.dnsPolicy=ClusterFirstWithHostNet,controller.image.tag=0.23.0,controller.extraArgs.v=10,controller.extraEnvs[0].name=KUBERNETES_SERVICE_PORT,controller.extraEnvs[0].value=6443However this crashes as explained by https://github.com/kubernetes/kubernetes/issues/57509#issuecomment-353509068 :
Error: release nginx failed: DaemonSet in version "v1beta1" cannot be handled as a DaemonSet: v1beta1.DaemonSet.Spec: v1beta1.DaemonSetSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 6, error found in #10 byte of ...|,"value":6443}],"ima|..., bigger context ...|ce"}}},{"name":"KUBERNETES_SERVICE_PORT","value":6443}],"image":"quay.io/kubernetes-ingress-controll|...I've attempted to use quotes which result in the same error. I afterwards moved to use escaped quotes, which should be legal per kubernetes' specs, as follow:
helm install --name nginx stable/nginx-ingress --set controller.kind=DaemonSet,controller.daemonset.useHostPort=true,controller.service.type=ClusterIP,controller.dnsPolicy=ClusterFirstWithHostNet,controller.image.tag=0.23.0,controller.extraArgs.v=10,controller.extraEnvs[0].name=KUBERNETES_SERVICE_PORT,controller.extraEnvs[0].value=\"6443\"However then the controller crashes at runtime with the following error:
I0313 18:46:56.877189 8 main.go:229] Unexpected error discovering Kubernetes version (attempt 1): Get https://172.17.0.1:"6443"/version?timeout=32s: invalid URL port "\"6443\""
kfox1111
commented
5 years ago Try setting it as a quoted string in a yaml file. its hard to get it into a string from the command line.
From: Maxime THIEBAUT [notifications@github.com] Sent: Wednesday, March 13, 2019 11:58 AM To: kubernetes/ingress-nginx Cc: Subscribed Subject: Re: [kubernetes/ingress-nginx] Helm DaemonSet deployment fails to probe and enters CrashLoopBackOff (#3892)
Any suggestion on how to modify the API port? I've attempted to define an environment variable as suggested in #3365https://github.com/kubernetes/ingress-nginx/issues/3365 due to extra args not working:
helm install --name nginx stable/nginx-ingress --set controller.kind=DaemonSet,controller.daemonset.useHostPort=true,controller.service.type=ClusterIP,controller.dnsPolicy=ClusterFirstWithHostNet,controller.image.tag=0.23.0,controller.extraArgs.v=10,controller.extraEnvs[0].name=KUBERNETES_SERVICE_PORT,controller.extraEnvs[0].value=6443
However this crashes as explained by kubernetes/kubernetes#57509 (comment)https://github.com/kubernetes/kubernetes/issues/57509#issuecomment-353509068 :
Error: release nginx failed: DaemonSet in version "v1beta1" cannot be handled as a DaemonSet: v1beta1.DaemonSet.Spec: v1beta1.DaemonSetSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 6, error found in #10 byte of ...|,"value":6443}],"ima|..., bigger context ...|ce"}}},{"name":"KUBERNETES_SERVICE_PORT","value":6443}],"image":"quay.io/kubernetes-ingress-controll|...
I've attempted to use quotes which result in the same error. I afterwards moved to use escaped quotes, which should be legal per kubernetes' specs, as follow:
helm install --name nginx stable/nginx-ingress --set controller.kind=DaemonSet,controller.daemonset.useHostPort=true,controller.service.type=ClusterIP,controller.dnsPolicy=ClusterFirstWithHostNet,controller.image.tag=0.23.0,controller.extraArgs.v=10,controller.extraEnvs[0].name=KUBERNETES_SERVICE_PORT,controller.extraEnvs[0].value=\"6443\"
However then the controller crashes at runtime with the following error:
I0313 18:46:56.877189 8 main.go:229] Unexpected error discovering Kubernetes version (attempt 1): Get https://172.17.0.1:"6443"/version?timeout=32s: invalid URL port "\"6443\""
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/kubernetes/ingress-nginx/issues/3892#issuecomment-472560147, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABcWwvz-rC8zeC1cI0XyLtb5kqyOpUXDks5vWUo9gaJpZM4btw_o.
wohckcin
commented
5 years ago I got the same issue. It only happed when I used DaemonSet and set useHostPort=true.
Here are the logs
I0315 11:42:13.584769 8 flags.go:183] Watching for Ingress class: nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.22.0
Build: git-f7c42b78a
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
W0315 11:42:13.585260 8 flags.go:216] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.8
built by gcc 8.2.0 (Debian 8.2.0-13)
built with OpenSSL 1.1.1a 20 Nov 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt='-g -Og -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -I/root/.hunter/_Base/2c5c6fc/b2f767f/92161a9/Install/include -Wno-cast-function-type -m64 -mtune=native' --with-ld-opt='-fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now -L/root/.hunter/_Base/2c5c6fc/b2f767f/92161a9/Install/lib' --user=www-data --group=www-data --add-module=/tmp/build/ngx_devel_kit-0.3.1rc1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.33 --add-module=/tmp/build/nginx-http-auth-digest-274490cec649e7300fea97fed13d84e596bbc0ce --add-module=/tmp/build/ngx_http_substitutions_filter_module-bc58cb11844bc42735bbaef7085ea86ace46d05b --add-module=/tmp/build/lua-nginx-module-1c72f57ce87d4355d546a97c2bd8f5123a70db5c --add-module=/tmp/build/stream-lua-nginx-module-0.0.6rc2 --add-module=/tmp/build/lua-upstream-nginx-module-0.07 --add-module=/tmp/build/nginx-influxdb-module-0e2cb6cbf850a29c81e44be9e33d9a15d45c50e8 --add-dynamic-module=/tmp/build/nginx-opentracing-ea9994d7135be5ad2e3009d0f270e063b1fb3b21/opentracing --add-dynamic-module=/tmp/build/ModSecurity-nginx-fc061a57a8b0abda79b17cbe103d78db803fa575 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-3.2 --add-module=/tmp/build/nginx_ajp_module-bf6cd93f2098b59260de8d494f0f4b1f11a84627 --add-module=/tmp/build/ngx_brotli
W0315 11:42:13.589630 8 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0315 11:42:13.589793 8 main.go:200] Creating API client for https://10.233.0.1:443
I0315 11:42:13.591539 8 main.go:220] Trying to discover Kubernetes version
I0315 11:42:13.591704 8 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.233.0.1:443/version?timeout=32s'
I0315 11:42:13.592008 8 round_trippers.go:405] GET https://10.233.0.1:443/version?timeout=32s in 0 milliseconds
I0315 11:42:13.592017 8 round_trippers.go:411] Response Headers:
I0315 11:42:13.592041 8 main.go:229] Unexpected error discovering Kubernetes version (attempt 0): Get https://10.233.0.1:443/version?timeout=32s: dial tcp 10.233.0.1:443: connect: connection refused
I0315 11:42:14.680348 8 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.233.0.1:443/version?timeout=32s'
I0315 11:42:14.680675 8 round_trippers.go:405] GET https://10.233.0.1:443/version?timeout=32s in 0 milliseconds
I0315 11:42:14.680687 8 round_trippers.go:411] Response Headers:
I0315 11:42:14.680774 8 main.go:229] Unexpected error discovering Kubernetes version (attempt 1): Get https://10.233.0.1:443/version?timeout=32s: dial tcp 10.233.0.1:443: connect: connection refused
I0315 11:42:16.224359 8 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/vnd.kubernetes.protobuf, */*" 'https://10.233.0.1:443/version?timeout=32s'
I0315 11:42:16.224717 8 round_trippers.go:405] GET https://10.233.0.1:443/version?timeout=32s in 0 milliseconds
I0315 11:42:16.224729 8 round_trippers.go:411] Response Headers:
I0315 11:42:16.224752 8 main.go:229] Unexpected error discovering Kubernetes version (attempt 2): Get https://10.233.0.1:443/version?timeout=32s: dial tcp 10.233.0.1:443: connect: connection refused
I0315 11:42:18.534989 8 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.233.0.1:443/version?timeout=32s'
I0315 11:42:18.535351 8 round_trippers.go:405] GET https://10.233.0.1:443/version?timeout=32s in 0 milliseconds
I0315 11:42:18.535363 8 round_trippers.go:411] Response Headers:
I0315 11:42:18.535383 8 main.go:229] Unexpected error discovering Kubernetes version (attempt 3): Get https://10.233.0.1:443/version?timeout=32s: dial tcp 10.233.0.1:443: connect: connection refused
I0315 11:42:22.018942 8 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.233.0.1:443/version?timeout=32s'
I0315 11:42:22.019344 8 round_trippers.go:405] GET https://10.233.0.1:443/version?timeout=32s in 0 millisecondsNGINX Ingress controller version:
0.22.0
Kubernetes version (use kubectl version):
v1.13.4
Environment:
aledbf
commented
5 years ago dial tcp 10.233.0.1:443: connect: connection refused
@nickychow no, you have other problem ^^
wohckcin
commented
5 years ago dial tcp 10.233.0.1:443: connect: connection refused
@nickychow no, you have other problem ^^
@aledbf Sorry, could you please provide more info? Thanks for your attention.
aledbf
commented
5 years ago @nickychow the pod cannot reach the IP:PORT you see in the log. That's the error. Please check you don't have network issues reaching the api server.
wohckcin
commented
5 years ago @aledbf It's weird when I set useHostPort=false, everything was working just fine.
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.22.0
Build: git-f7c42b78a
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
I0315 12:33:09.677826 8 flags.go:183] Watching for Ingress class: nginx
W0315 12:33:09.678312 8 flags.go:216] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.8
built by gcc 8.2.0 (Debian 8.2.0-13)
built with OpenSSL 1.1.1a 20 Nov 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/etc/nginx/modules --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-http_secure_link_module --with-http_gunzip_module --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt='-g -Og -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wno-deprecated-declarations -fno-strict-aliasing -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_FASTOPEN=23 -fPIC -I/root/.hunter/_Base/2c5c6fc/b2f767f/92161a9/Install/include -Wno-cast-function-type -m64 -mtune=native' --with-ld-opt='-fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now -L/root/.hunter/_Base/2c5c6fc/b2f767f/92161a9/Install/lib' --user=www-data --group=www-data --add-module=/tmp/build/ngx_devel_kit-0.3.1rc1 --add-module=/tmp/build/set-misc-nginx-module-0.32 --add-module=/tmp/build/headers-more-nginx-module-0.33 --add-module=/tmp/build/nginx-http-auth-digest-274490cec649e7300fea97fed13d84e596bbc0ce --add-module=/tmp/build/ngx_http_substitutions_filter_module-bc58cb11844bc42735bbaef7085ea86ace46d05b --add-module=/tmp/build/lua-nginx-module-1c72f57ce87d4355d546a97c2bd8f5123a70db5c --add-module=/tmp/build/stream-lua-nginx-module-0.0.6rc2 --add-module=/tmp/build/lua-upstream-nginx-module-0.07 --add-module=/tmp/build/nginx-influxdb-module-0e2cb6cbf850a29c81e44be9e33d9a15d45c50e8 --add-dynamic-module=/tmp/build/nginx-opentracing-ea9994d7135be5ad2e3009d0f270e063b1fb3b21/opentracing --add-dynamic-module=/tmp/build/ModSecurity-nginx-fc061a57a8b0abda79b17cbe103d78db803fa575 --add-dynamic-module=/tmp/build/ngx_http_geoip2_module-3.2 --add-module=/tmp/build/nginx_ajp_module-bf6cd93f2098b59260de8d494f0f4b1f11a84627 --add-module=/tmp/build/ngx_brotli
W0315 12:33:09.685785 8 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0315 12:33:09.686529 8 main.go:200] Creating API client for https://10.233.0.1:443
I0315 12:33:09.687614 8 main.go:220] Trying to discover Kubernetes version
I0315 12:33:09.687845 8 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.233.0.1:443/version?timeout=32s'
I0315 12:33:09.696778 8 round_trippers.go:405] GET https://10.233.0.1:443/version?timeout=32s 200 OK in 8 milliseconds
I0315 12:33:09.696801 8 round_trippers.go:411] Response Headers:
I0315 12:33:09.696805 8 round_trippers.go:414] Date: Fri, 15 Mar 2019 12:33:09 GMT
I0315 12:33:09.696809 8 round_trippers.go:414] Content-Type: application/json
I0315 12:33:09.696812 8 round_trippers.go:414] Content-Length: 263
I0315 12:33:09.696842 8 request.go:942] Response Body: {
"major": "1",
"minor": "13",
"gitVersion": "v1.13.4",
"gitCommit": "c27b913fddd1a6c480c229191a087698aa92f0b1",
"gitTreeState": "clean",
"buildDate": "2019-02-28T13:30:26Z",
"goVersion": "go1.11.5",
"compiler": "gc",
"platform": "linux/amd64"
}
I0315 12:33:09.696915 8 main.go:244] Running in Kubernetes cluster version v1.13 (v1.13.4) - git (clean) commit c27b913fddd1a6c480c229191a087698aa92f0b1 - platform linux/amd64
I0315 12:33:09.696972 8 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.233.0.1:443/api/v1/namespaces/ingress/services/nginx-ingress-default-backend'
I0315 12:33:09.698736 8 round_trippers.go:405] GET https://10.233.0.1:443/api/v1/namespaces/ingress/services/nginx-ingress-default-backend 200 OK in 1 millisecondsAfter a bit of search, I think we are hitting this one.
https://github.com/kubernetes/kubernetes/issues/66103
0xThiebaut
commented
5 years ago The weird part is that switching form ingress controller made it work.
fejta-bot
commented
5 years ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
fejta-bot
commented
5 years ago Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten
fejta-bot
commented
5 years ago Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close
k8s-ci-robot
commented
5 years ago @fejta-bot: Closing this issue.
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):
No
What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):
Readiness probe failed
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug Report
NGINX Ingress controller version:
0.22.0, 0.23.0
Kubernetes version (use
kubectl version):v1.13.4
Environment:
uname -a):Linux scw-par-1-1 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/LinuxWhat happened:
Nginx Ingress was deployed through the following Helm command:
When checking the pod statuses through
kubectl get pods -n nginx-ingressthey were displayed as not ready:Describing the pods showed that both the readiness and liveness probe failed repeatedly:
What you expected to happen:
I expected
kubectl get pods -n nginx-ingressto show all pods as ready.How to reproduce it (as minimally and precisely as possible):
Deploy Nginx Ingress through Helm with the following added values (desired nginx and tiller namespaces assumed default):
Anything else we need to know:
I've attempted to increase the probe delays to 120 seconds which had no effect on the end result. Furthermore, the controller's logs seem clear: