Default Helm chart installation fails because of Kong configuration issue: Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1):

pzi123 commented 2 months ago

What happened?

I tried installing the Kubernetes Dashboard helm chart version 7.5.0 The Kong pod does not come up, I see the following error in the logs:

$ kubectl logs -n kubernetes-dashboard pod/kubernetes-dashboard-kong-d59b89c4f-4bkxr Defaulted container "proxy" out of: proxy, clear-stale-pid (init) Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1): nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7 nginx: the configuration file /kong_prefix/nginx.conf syntax is ok nginx: [emerg] socket() [::]:8000 failed (97: Address family not supported by protocol) nginx: configuration file /kong_prefix/nginx.conf test failed

Run with --v (verbose) or --vv (debug) for more details

The system running control node does not have IPV6 enabled. Still the kubernetes-dashboard should work with just IPV4.

$ kubectl get po -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE kubernetes-dashboard-api-7f75f559c9-j4kcq 1/1 Running 0 71m kubernetes-dashboard-auth-5475584774-2957l 1/1 Running 0 71m kubernetes-dashboard-kong-6d79b95f7d-msvrj 0/1 CrashLoopBackOff 18 (3m38s ago) 71m kubernetes-dashboard-kong-d59b89c4f-4bkxr 0/1 CrashLoopBackOff 18 (3m31s ago) 71m kubernetes-dashboard-metrics-scraper-5b8f478ddc-j9np8 1/1 Running 0 71m kubernetes-dashboard-web-78b7b4ff86-kc9tp 1/1 Running 0 71m

What did you expect to happen?

The application should work.

How can we reproduce it (as minimally and precisely as possible)?

$ helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard

Anything else we need to know?

$ helm list -n kubernetes-dashboard NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION kubernetes-dashboard kubernetes-dashboard 1 2024-08-28 21:01:51.050263828 -0600 MDT deployed kubernetes-dashboard-7.5.0

What browsers are you seeing the problem on?

No response

Kubernetes Dashboard version

7.5.0

Kubernetes version

v1.30.4

Dev environment

$ kubectl get no NAME STATUS ROLES AGE VERSION p32-ka21 Ready control-plane 26m v1.30.4 p32-ka22 Ready 25m v1.30.4

$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

georglauterbach commented 2 months ago

I'm seeing the same issue. I have IPv6 disabled by the CNI and in the kernel.

floreks commented 2 months ago

Try overriding this value and remove AAAA. This should disable IPv6 resolution in kong.

https://github.com/kubernetes/dashboard/blob/a12c8099c7ecbf685a6d43aff73d3a9355a8e7d6/charts/kubernetes-dashboard/values.yaml#L358

siva9941166 commented 2 months ago

I am facing same issue, any solutions for the below error?

Warning: Helm release "" was created but has a failed status. Use the helm command to investigate the error, correct it, then run Terraform again. │ │ with helm_release.kube-ui, │ on 05-gke-add-on.tf line 64, in resource "helm_release" "kube-ui": │ 64: resource "helm_release" "kube-ui" { │ ╵ ╷ │ Error: context deadline exceeded │ │ with helm_release.kube-ui, │ on 05-gke-add-on.tf line 64, in resource "helm_release" "kube-ui": │ 64: resource "helm_release" "kube-ui" {

Logs:

2024-09-05 11:15:32.828 IST nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address) 2024-09-05 11:15:32.828 IST nginx: configuration file /kong_prefix/nginx.conf test failed 2024-09-05 11:15:32.828 IST 2024-09-05 11:15:32.828 IST 2024-09-05 11:15:32.828 IST Run with --v (verbose) or --vv (debug) for more details 2024-09-05 11:20:44.874 IST Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1): 2024-09-05 11:20:44.874 IST nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7 2024-09-05 11:20:44.874 IST nginx: the configuration file /kong_prefix/nginx.conf syntax is ok 2024-09-05 11:20:44.874 IST nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)

pzi123 commented 2 months ago

I enabled ipv6 for the k8s nodes - which is a security threat since ipv6 is not maintained and no one needs it.

On 9/5/2024 12:05 AM, siva9941166 wrote:

I am facing same issue, any solutions for the below error?

Warning: Helm release "" was created but has a failed status. Use the |helm| command to investigate the error, correct it, then run Terraform again. │ │ with helm_release.kube-ui, │ on 05-gke-add-on.tf line 64, in resource "helm_release" "kube-ui": │ 64: resource "helm_release" "kube-ui" { │ ╵ ╷ │ Error: context deadline exceeded │ │ with helm_release.kube-ui, │ on 05-gke-add-on.tf line 64, in resource "helm_release" "kube-ui": │ 64: resource "helm_release" "kube-ui" {

Logs:

2024-09-05 11:15:32.828 IST nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address) 2024-09-05 11:15:32.828 IST nginx: configuration file /kong_prefix/nginx.conf test failed 2024-09-05 11:15:32.828 IST 2024-09-05 11:15:32.828 IST 2024-09-05 11:15:32.828 IST Run with --v (verbose) or --vv (debug) for more details 2024-09-05 11:20:44.874 IST Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1): 2024-09-05 11:20:44.874 IST nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7 2024-09-05 11:20:44.874 IST nginx: the configuration file /kong_prefix/nginx.conf syntax is ok 2024-09-05 11:20:44.874 IST nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/dashboard/issues/9420#issuecomment-2330672763, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOUAJB3VRLYEP3W64NEMU3ZU7YBBAVCNFSM6AAAAABNJRXQLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZQGY3TENZWGM. You are receiving this because you authored the thread.Message ID: @.***>

-- Peter

--------------5SXbVsgKVxLZtNDI3Z4A8m0K Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

I enabled ipv6 for the k8s nodes - which is a security threat since ipv6 is not maintained and no one needs it.

On 9/5/2024 12:05 AM, siva9941166 wrote:

I am facing same issue, any solutions for the below error?

Warning: Helm release "" was created but has a failed status. Use the helm command to investigate the error, correct it, then run Terraform again.
│
│ with helm_release.kube-ui,
│ on 05-gke-add-on.tf line 64, in resource "helm_release" "kube-ui":
│ 64: resource "helm_release" "kube-ui" {
│
╵
╷
│ Error: context deadline exceeded
│
│ with helm_release.kube-ui,
│ on 05-gke-add-on.tf line 64, in resource "helm_release" "kube-ui":
│ 64: resource "helm_release" "kube-ui" {

Logs:

2024-09-05 11:15:32.828 IST
nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)
2024-09-05 11:15:32.828 IST
nginx: configuration file /kong_prefix/nginx.conf test failed
2024-09-05 11:15:32.828 IST
2024-09-05 11:15:32.828 IST
2024-09-05 11:15:32.828 IST
Run with --v (verbose) or --vv (debug) for more details
2024-09-05 11:20:44.874 IST
Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1):
2024-09-05 11:20:44.874 IST
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
2024-09-05 11:20:44.874 IST
nginx: the configuration file /kong_prefix/nginx.conf syntax is ok
2024-09-05 11:20:44.874 IST
nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.

-- 
Peter Ziobrzynski, , tel: 720-514-9877

--------------5SXbVsgKVxLZtNDI3Z4A8m0K--

siva9941166 commented 2 months ago

If I enable ipv6 for the k8s nodes, kube ui will work?

pzi123 commented 2 months ago

yes

On 9/5/2024 12:12 AM, siva9941166 wrote:

If I enable ipv6 for the k8s nodes, kube ui will work?

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/dashboard/issues/9420#issuecomment-2330682459, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOUAJASTX43ENUVP4QACFTZU7Y4JAVCNFSM6AAAAABNJRXQLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZQGY4DENBVHE. You are receiving this because you authored the thread.Message ID: @.***>

-- Peter

--------------pKFCeAk5fyF6iT7LuVwnd7yP Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

yes

On 9/5/2024 12:12 AM, siva9941166 wrote:

If I enable ipv6 for the k8s nodes, kube ui will work?

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.

-- 
Peter Ziobrzynski, , tel: 720-514-9877

--------------pKFCeAk5fyF6iT7LuVwnd7yP--

siva9941166 commented 2 months ago

is there any solution when we use ipv4 ? in my organization we have created ipv4 only. you have any solution for it.

pzi123 commented 2 months ago

Try overriding this value and remove AAAA. This should disable IPv6 resolution in kong.

https://github.com/kubernetes/dashboard/blob/a12c8099c7ecbf685a6d43aff73d3a9355a8e7d6/charts/kubernetes-dashboard/values.yaml#L358

Have you tried it? What value did you use?

rskDam commented 2 months ago

Try overriding this value and remove AAAA. This should disable IPv6 resolution in kong.

https://github.com/kubernetes/dashboard/blob/a12c8099c7ecbf685a6d43aff73d3a9355a8e7d6/charts/kubernetes-dashboard/values.yaml#L358

I have tried this, it did not work. Same error

floreks commented 2 months ago

This is actually coming from nginx trying to listen on IPv6 interface.

nginx: [emerg] socket() [::]:8000 failed (97: Address family not supported by protocol)

It should be possible to override listen config via proxy_listen. In values.yaml file it can be configured under kong.env.proxy_listen. https://github.com/kubernetes/dashboard/blob/fb9949f7234e72e33363f7abd74ae39e5e7190d6/charts/kubernetes-dashboard/values.yaml#L354-L360

There could be more, so it is best to check the configuration reference both for nginx and kong gateway.

pzi123 commented 2 months ago

This is actually coming from nginx trying to listen on IPv6 interface.

nginx: [emerg] socket() [::]:8000 failed (97: Address family not supported by protocol)

It should be possible to override listen config via proxy_listen. In values.yaml file it can be configured under kong.env.proxy_listen.

https://github.com/kubernetes/dashboard/blob/fb9949f7234e72e33363f7abd74ae39e5e7190d6/charts/kubernetes-dashboard/values.yaml#L354-L360

There could be more, so it is best to check the configuration reference both for nginx and kong gateway.

What needs to happen here is a triage by Dashboard maintainers, looking for a solution, testing, releasing the fix. @floreks your untested suggestions pointing at random provide little value here.

floreks commented 2 months ago

This is not a bug, but a configuration issue on your side. Our defaults support IPv4+IPv6 and IPv6 only clusters by default. There is no way to also support IPv4 only clusters with the same default values.

We will not be releasing a "fix" and changing the defaults just to flip on the IPv6 only clusters and break those installations.

I have tried to assist with finding the correct combination of values for you even though we do not really provide support via issues. Helm allows overriding values and that is what users with custom requirements/clusters should use to make their installations work.

Closing as this is a configuration issue and can be solved via overriding helm values.

/close

k8s-ci-robot commented 2 months ago

@floreks: Closing this issue.

In response to [this](https://github.com/kubernetes/dashboard/issues/9420#issuecomment-2333260346): >This is not a bug, but a configuration issue on your side. Our defaults support IPv4+IPv6 and IPv6 only clusters by default. There is no way to also support IPv4 only clusters with the same default values. > >We will not be releasing a "fix" and changing the defaults just to flip on the IPv6 only clusters and break those installations. > >I have tried to assist with finding the correct combination of values for you even though we do not really provide support via issues. Helm allows overriding values and that is what users with custom requirements/clusters should use to make their installations work. > >Closing as this is a configuration issue and can be solved via overriding helm values. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

pzi123 commented 2 months ago

dual stack was introduced in July (2024). The default in vast majority of installations is ipv4 only.

floreks commented 2 months ago

From the Kubernetes docs:

IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in 1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses.

The latest version of Dashboard will always try to match the Kubernetes defaults for the version it currently supports. Right now it is K8S 1.28 or 1.29.

PS. Kubernetes 1.21 was released on 08 Apr 2021.

kubernetes / dashboard