Closed mirisu2 closed 2 years ago
Is the 404 coming from nginx? Can you run helm list -n openunison
. Those errors usually mean the orchestra-login-portal
chart hasn't deployed.
orchestra-login-portal
pod is started
# kubectl -n openunison get po
NAME READY STATUS RESTARTS AGE
openunison-operator-c8b7966fb-fjvmw 1/1 Running 0 15h
openunison-orchestra-65c7f8849f-8lr8x 1/1 Running 1 (8m29s ago) 15m
ouhtml-orchestra-login-portal-5655959dc-zbh6j 1/1 Running 1 (8m29s ago) 15m
test-orchestra-orchestra 0/1 Completed 0 15m
and orchestra-login-portal
's logs show
# kubectl -n openunison logs -f ouhtml-orchestra-login-portal-5655959dc-zbh6j
192.168.31.103 - - [14/Apr/2022:07:38:29 +0000] "GET /nginx-health HTTP/1.1" 200 8 "-" "kube-probe/1.23" "-"
192.168.31.103 - - [14/Apr/2022:07:38:29 +0000] "GET /nginx-health HTTP/1.1" 200 8 "-" "kube-probe/1.23" "-"
192.168.31.103 - - [14/Apr/2022:07:38:35 +0000] "GET /nginx-health HTTP/1.1" 200 8 "-" "kube-probe/1.23" "-"
192.168.31.103 - - [14/Apr/2022:07:38:59 +0000] "GET /nginx-health HTTP/1.1" 200 8 "-" "kube-probe/1.23" "-"
but logs of openunison-orchestra-65c7f8849f-8lr8x
pod:
[2022-04-14 07:38:40,434][main] INFO threads - JBoss Threads version 2.3.6.Final
[2022-04-14 07:38:59,846][XNIO-1 task-1] INFO AccessLog - [Error] - UNKNOWN - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [127.0.0.1] - [fcbdae9b363fda5541ec734bed5afcc2cca01b11b]
[2022-04-14 07:38:59,846][XNIO-1 task-1] ERROR ConfigSys - Could not process request
javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration
at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:116) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:89) ~[unison-sdk-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:88) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:269) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.29.jar:?]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280) ~[xnio-api-3.8.6.Final.jar:3.8.6.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
[2022-04-14 07:39:00,016][local_Worker-1] WARN ClearSessions - No openid connect identity providers available yet
[2022-04-14 07:39:00,450][XNIO-1 task-1] INFO AccessLog - [AzSuccess] - CheckAlive - https://127.0.0.1:8443/check_alive - uid=Anonymous,o=Tremolo - [127.0.0.1] - [facec99adb5c7975455ce01648280743337fcf1ca]
[2022-04-14 07:39:02,389][XNIO-1 task-1] INFO AccessLog - [Error] - UNKNOWN - https://k8sou.stage.mydomain.com/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [10.233.21.194] - [ffd6309c01fad35508cb50e37230d88c5ead29160]
[2022-04-14 07:39:02,389][XNIO-1 task-1] ERROR ConfigSys - Could not process request
javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration
at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:116) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:89) ~[unison-sdk-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:88) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:269) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.29.jar:?]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100) ~[undertow-servlet-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852) ~[undertow-core-2.2.16.Final.jar:2.2.16.Final]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280) ~[xnio-api-3.8.6.Final.jar:3.8.6.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
[2022-04-14 07:39:06,169][XNIO-1 task-1] INFO AccessLog - [Error] - UNKNOWN - https://k8sou.stage.mydomain.com/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [10.233.21.194] - [f248f08340d8645563a0dd3f77ae5d4bde822be18]
[2022-04-14 07:39:06,169][XNIO-1 task-1] ERROR ConfigSys - Could not process request
javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration
at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:116) ~[unison-server-core-1.0.29.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.29.jar:?]
Why this errors happened?
[2022-04-14 07:39:00,016][local_Worker-1] WARN ClearSessions - No openid connect identity providers available yet
[2022-04-14 07:38:59,846][XNIO-1 task-1] INFO AccessLog - [Error] - UNKNOWN - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [127.0.0.1] - [fcbdae9b363fda5541ec734bed5afcc2cca01b11b]
[2022-04-14 07:38:59,846][XNIO-1 task-1] ERROR ConfigSys - Could not process request javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration
Service openunison-orchestra
has endpoint
Port: openunison-secure-orchestra 443/TCP
TargetPort: 8443/TCP
Endpoints: 10.233.58.202:8443
Port: openunison-insecure-orchestra 80/TCP
TargetPort: 8080/TCP
Endpoints: 10.233.58.202:8080
Ingresses are created.
Name: openunison-orchestra
Labels: app.kubernetes.io/component=ingress-nginx
app.kubernetes.io/instance=openunison-orchestra
app.kubernetes.io/name=openunison
app.kubernetes.io/part-of=openunison
Namespace: openunison
Address: 192.168.31.102
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
ou-tls-certificate terminates k8sou.stage.mydomain.com,dashboard.stage.mydomain.com
Rules:
Host Path Backends
---- ---- --------
k8sou.stage.mydomain.com
/ openunison-orchestra:443 (10.233.58.202:8443)
dashboard.stage.mydomain.com
/ openunison-orchestra:443 (10.233.58.202:8443)
Annotations: kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/backend-protocol: https
nginx.ingress.kubernetes.io/secure-backends: true
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
nginx.ingress.kubernetes.io/session-cookie-name: openunison-orchestra
nginx.org/ssl-services: openunison-orchestra
And one more moment, I don't know is it important or not. I cannot use helm in new environment. Preliminarily I have to do
helm pull tremolo/orchestra --untar
helm template orchestra -n openunison -f openunison-default-values.yaml orchestra/. > orchestra.yaml
helm pull tremolo/orchestra-login-portal --untar
helm template orchestra-login-portal -n openunison -f openunison-default-values.yaml orchestra-login-portal/. > orchestra-login-portal.yaml
after that in my restricted environment:
kubectl apply -f orchestra.yaml
kubectl apply -f orchestra-login-portal.yaml
nginx pod logs: 404 error - when I access page https://k8sou.stage.mydomain.com
10.233.24.0 - - [14/Apr/2022:09:26:10 +0000] "GET / HTTP/2.0" 404 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0" 399 0.002 [openunison-openunison-orchestra-443] [] 10.233.58.202:8443 0 0.004 404 7b6b2bbabd46663423c318f33e05583e
500 error - this errors periodically
10.233.24.0 - - [14/Apr/2022:09:26:12 +0000] "GET /auth/idp/k8sIdp/.well-known/openid-configuration HTTP/2.0" 500 2302 "-" "Go-http-client/2.0" 6 0.003 [openunison-openunison-orchestra-443] [] 10.233.58.202:8443 2302 0.004 500 5f6005ddf99c6441aafbaf72cdd2571c
Ok, I found out the reason of errors. CI run step-by-step tasks. orchestra-login-portal
deploy starts before orchestra
pod was started. I added a condition to check if pod in runnnig state before continue. It is fixed.
Now I can access auth page. I enter my login/password. I click Sign in
and got an error Not Authorized
In logs of openunison-orchestra-65c7f8849f-lt64w
pod I see
[2022-04-14 13:14:56,969][Thread-10] WARN LDAPConnectionPool - Could not execute ldap heartbeat for srvdc1.mydomain.com/389, recreating connection
com.novell.ldap.LDAPException: Invalid Credentials
Now I have another question. Invalid Credentials
- does it mean what credentials to connect ldap server is wrong?
I deployed yet another cluster and openunison deployment. The only changes I made is another domain name and ldap server. But this time I got an error.
When I accessed k8sou page - I receive 404 error. Log from
nginx ingress controller
:Log from
openunison-orchestra-65c7f8849f-cg6vr
:Tell me please, what does it mean? How to fix this errors?