search

openshift / console

OpenShift Cluster Console UI
https://www.openshift.org
Apache License 2.0
401 stars 612 forks source link

Internal k8s proxy goes into broken state after high usage #12526

Closed djcass44 closed 1 year ago

djcass44 commented 1 year ago

Kubernetes version: v1.24.8-eks-ffeb93d (EKS on AWS), v1.24.8+k3s1 (K3S on Bare metal) OpenShift Console version: 4.13.0, 4.12.0

After extended usage, the console stops working. The Web UI shows a blank white screen or a "Bad Gateway" message and the server reports the following error:

http: proxy error: dial tcp 1.2.3.4:443: connect: cannot assign requested address

It shows this error over and over until a human restarts the deployment. The liveness probes to /health continue to succeed even though the pod is in a broken state.

The issue is present with version 4.13 and 4.12. I had used 4.12 without the issue for quite a while with no issues and suddenly it appeared when a newer version of the image was pulled.

I'll be testing older versions to see if there was an introduction point, however since it requires a decent amount of usage by human users that may take a while to confirm.

openshift-bot commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

huangkaoya commented 1 year ago

Yes, we have encountered this issue as well. In fact, I think the bridge-based entry provided by OpenShift has poor performance.

huangkaoya commented 1 year ago

/remove-lifecycle stale

openshift-bot commented 1 year ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented 1 year ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot commented 1 year ago

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci[bot] commented 1 year ago

@openshift-bot: Closing this issue.

In response to [this](https://github.com/openshift/console/issues/12526#issuecomment-1746343262): >Rotten issues close after 30d of inactivity. > >Reopen the issue by commenting `/reopen`. >Mark the issue as fresh by commenting `/remove-lifecycle rotten`. >Exclude this issue from closing again by commenting `/lifecycle frozen`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.