upstream connect error or disconnect/reset before headers. reset reason: connection termination

[kumy]

Hi, we're trying to deploy the stack. We followed https://openunison.github.io/deployauth/#alternate-deployment-methods page.

We can successfully login on orchestra using GitHub and were presented with the landing page

When we try to access https://k8sou.xxx/k8stoken/ we get error

upstream connect error or disconnect/reset before headers. reset reason: connection termination

We didn't find any error in the logs from the pods in the openunison namespace when we hit/refresh that page.

However, we found this one on startup, could it be related :shrug: ?

openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra [2024-10-23 08:25:04,042][main] WARN  UrlHolder - Could not process url : ''
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra java.net.MalformedURLException: no protocol: 
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at java.base/java.net.URL.<init>(URL.java:772) ~[?:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at java.base/java.net.URL.<init>(URL.java:654) ~[?:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at java.base/java.net.URL.<init>(URL.java:590) ~[?:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.config.util.UrlHolder.<init>(UrlHolder.java:125) [unison-sdk-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.config.util.UnisonConfigManagerImpl.addAppInternal(UnisonConfigManagerImpl.java:882) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.config.util.UnisonConfigManagerImpl.addApplication(UnisonConfigManagerImpl.java:799) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.proxy.dynamicconfiguration.LoadApplicationsFromK8s.addObject(LoadApplicationsFromK8s.java:476) [unison-applications-k8s-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.k8s.watch.K8sWatcher.initalRun(K8sWatcher.java:161) [unison-applications-k8s-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.proxy.dynamicconfiguration.LoadApplicationsFromK8s.loadDynamicApplications(LoadApplicationsFromK8s.java:447) [unison-applications-k8s-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.config.util.UnisonConfigManagerImpl.initialize(UnisonConfigManagerImpl.java:587) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.filter.UnisonServletFilter.init(UnisonServletFilter.java:369) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.openunison.OpenUnisonServletFilter.init(OpenUnisonServletFilter.java:118) [open-unison-classes-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:111) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.ManagedFilter.createFilter(ManagedFilter.java:86) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:598) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:605) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra  at com.tremolosecurity.openunison.undertow.OpenUnisonOnUndertow.main(OpenUnisonOnUndertow.java:357) [openunison-on-undertow-1.0.41.jar:?]

Any hint on what we should check, or how to enable more debug logs? (We're using Istio as Ingress controller.) Thanks for your help :pray:

Startup logs: openunison-orchestra.txt

[kumy]

BTW the oulogin plugin works fine to configure our kube config

# ~
$ kubectl oulogin --host=k8sou.ou.xxxxxx
oulogin 0.0.7
Checking for existing issuer https://k8sou.ou.xxxxxx/auth/idp/k8sIdp
Invalid context or does not exist, launching browser to login
Starting OpenID Connect for host k8sou.ou.xxxxxx
kubectl configuration created

# ~
$ k get po
NAME                                            READY   STATUS      RESTARTS   AGE
kube-oidc-proxy-orchestra-6989c8ffbb-wlglv      2/2     Running     0          39m
openunison-operator-6f5d4b7cc6-dn9w7            2/2     Running     0          45m
openunison-orchestra-b459ffb77-tb7vz            2/2     Running     0          39m
ouhtml-orchestra-login-portal-5fd446c89-l2k74   2/2     Running     0          36m
test-orchestra-orchestra                        0/1     Completed   0          39m

[mlbiam]

Does the dashboard work?

[kumy]

We didn't deployed the kube dashboard if it's what you mean

[mlbiam]

We didn't deployed the kube dashboard if it's what you mean

Understood, what happens if you go to https://k8s.ou.XXX/k8stoken/token/user, do you get a bunch of JSON? Also, do you have network policies enabled in your values.yaml?

[kumy]

Yes we see json, and no network policies enabled

network_policies:
  enabled: false
  ingress:
    enabled: true
    labels: []
  monitoring:
    enabled: true
    labels: []
  apiserver:
    enabled: true
    labels: []
  namespace_label: kubernetes.io/metadata.name

[mlbiam]

That's so odd. The original error comes usually from the virtual gateway not being able to connect or the certificate isn't setup. but you're getting to the OpenUnison service and pod. The "slow" part, generating tokens, works. It appears the issue is just the conection from OpenUnison --> html pod, but you're able to login to the openunison portal. is it possible that the service mesh is blocking the URL?

[mlbiam]

oh, i just saw your screen shot, that's REALLY old. What version of OpenUnison are you deploying (charts, images, etc)

[kumy]

We're using helm as:

• helm template openunison tremolo/openunison-operator --namespace openunison | tee openunison-base.yaml
• helm template orchestra tremolo/orchestra --namespace openunison -f values.yaml | tee orchestra-base.yaml
• helm template orchestra-login-portal tremolo/orchestra-login-portal --namespace openunison -f values.yaml | tee orchestra-login-portal-base.yaml

from

• https://github.com/OpenUnison/helm-charts/tree/master

That deploys images:

$ k get po -o yaml|grep image:|sort -u
      image: ghcr.io/openunison/openunison-k8s:1.0.41
      image: ghcr.io/openunison/openunison-k8s-html:1.0.0
      image: ghcr.io/openunison/openunison-kubernetes-operator:1.0.6
      image: ghcr.io/tremolosecurity/kube-oidc-proxy:1.0.7
      image: ghcr.io/tremolosecurity/python3:1.0.0
      image: upm-istio/proxyv2:1.19.7-distroless

is it possible that the service mesh is blocking the URL?

we were suspecting our base istio rules to block something, but we didn't have any to what/where to start our investigation

[mlbiam]

ghcr.io/openunison/openunison-k8s-html:1.0.0

odd, where did that come from? Try setting openunison.html.image to ghcr.io/openunison/openunison-k8s-react:1.0.0 in your values.yaml. also, make sure that openunison.html.legacy is false.

we were suspecting our base istio rules to block something, but we didn't have any to what/where to start our investigation

Do you have Kiali deployed? That's usually a good place. What version of istio?