Open plcarterco opened 6 months ago
'[2024-01-09 18:07:48,570][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.redacted.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]' '[2024-01-09 18:07:48,627][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.redacted.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]'
is there a stack trace after this? It would start with "Sendnig JSON Error"?
Also, what verion of the helm charts are you on?
hello,
we're using orchestra-login-portal-argocd via the ArgoCD Application Object - version: 2.3.38
(saw the recent versions to 2.3.41 but it breaks in other areas regarding groups and how we are using them I believe....We haven't had time to investigate and didn't want to introduce new variables into the mix. )
there's no stack trace, but here's fuller log context before and after:
[2024-01-09 18:07:47,153][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:47,211][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:47,211][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]
[2024-01-09 18:07:48,006][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]
[2024-01-09 18:07:48,494][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:48,570][XNIO-1 task-5] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:07:48,570][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde]
[2024-01-09 18:07:48,627][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]
[2024-01-09 18:07:51,196][Thread-8] INFO K8sWatcher - Resource 36753 already processed, skipping
[2024-01-09 18:07:52,584][Thread-13] INFO K8sWatcher - Resource 37101 already processed, skipping
[2024-01-09 18:07:54,161][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - k8sidp - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - uid=Anonymous,o=Tremolo - NONE [127.0.0.1] - [f6dfd36ec006b1545c4767118e3f52195449e143a]
[2024-01-09 18:07:54,162][XNIO-1 task-6] INFO AccessLog - [AzSuccess] - k8sidp - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - uid=Anonymous,o=Tremolo - NONE [127.0.0.1] - [f7b242b275c8bafed4e91e16f594eef2134f4eb0c]
[2024-01-09 18:07:54,386][Thread-9] INFO K8sWatcher - Resource 34772 already processed, skipping
[2024-01-09 18:07:57,362][Thread-12] INFO K8sWatcher - Resource 37550 already processed, skipping
[2024-01-09 18:07:58,096][XNIO-1 task-6] INFO AccessLog - [AzSuccess] - scale-session-check - https://k8sou.athens-aks.patb68.REDACTED.ai/scale/sessioncheck - uid=Anonymous,o=Tremolo - NONE [10.244.0.26] - [f015c2e843b69a5404c633b2011cbccd98aa68320]
[2024-01-09 18:07:58,106][XNIO-1 task-6] WARN OpenShiftTarget - Unexpected result calling 'https://10.0.0.1:443/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/x90a54ae8-5757-4a94-84c1-eec6980029c9x' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"x90a54ae8-5757-4a94-84c1-eec6980029c9x\" not found","reason":"NotFound","details":{"name":"x90a54ae8-5757-4a94-84c1-eec6980029c9x","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}
[2024-01-09 18:08:04,156][XNIO-1 task-6] INFO AccessLog - [AzSuccess] - k8sidp - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - uid=Anonymous,o=Tremolo - NONE [127.0.0.1] - [fc9e5a9078ac66f09a8c4d5c8d6e260f931de5bc3]
here's the OU config:
network:
openunison_host: "k8sou.${OU_BASEDOMAIN}"
dashboard_host: "k8sdb.${OU_BASEDOMAIN}"
api_server_host: "k8sapi.${OU_BASEDOMAIN}"
session_inactivity_timeout_seconds: 900
k8s_url: "https://kubernetes.default.svc"
force_redirect_to_tls: false
createIngressCertificate: false
ingress_certificate: ou-tls-certificate
ingress_type: traefik
ingress_annotations:
kubernetes.io/ingress.class: traefik0
traefik:
secure: true
entrypoints:
plaintext: web
tls: websecure
cert_template:
ou: "Kubernetes"
o: "NeoEnvironments"
l: "${OU_BASEDOMAIN}"
st: "Colorado"
c: "USA"
myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: "${OU_CLUSTER_NAME}"
enable_impersonation: true
impersonation:
use_jetstack: true
jetstack_oidc_proxy_image: docker.io/tremolosecurity/kube-oidc-proxy:latest
explicit_certificate_trust: true
dashboard:
enabled: true
namespace: "kubernetes-dashboard"
cert_name: "kubernetes-dashboard-certs"
label: "k8s-app=kubernetes-dashboard"
service_name: kubernetes-dashboard
require_session: true
certs:
use_k8s_cm: false
trusted_certs: []
monitoring:
prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s
oidc:
client_id_is_secret: true
issuer: "${OU_ISSUER}"
user_in_idtoken: true
domain: ""
scopes: openid email profile groups
claims:
sub: email
email: email
given_name: given_name
family_name: family_name
display_name: name
groups: ${OU_CLAIM_NAMESPACE}/groups
network_policies:
enabled: false
ingress:
enabled: true
labels:
app.kubernetes.io/name: traefik
monitoring:
enabled: true
labels:
app.kubernetes.io/name: monitoring
apiserver:
enabled: true
labels:
app.kubernetes.io/name: kube-system
services:
enable_tokenrequest: false
token_request_audience: api
token_request_expiration_seconds: 600
node_selectors: []
openunison:
replicas: 1
non_secret_data:
K8S_DB_SSO: oidc
PROMETHEUS_SERVICE_ACCOUNT: system:serviceaccount:monitoring:prometheus-k8s
SHOW_PORTAL_ORGS: "false"
secrets: []
enable_provisioning: false
use_standard_jit_workflow: true
az_groups: []
html:
image: docker.io/tremolosecurity/openunison-k8s-html
logosConfigMap: custom-logos
operator:
validators: []
mutators: []
imagePullPolicy: Always
resources:
requests:
memory: 200Mi
cpu: 0.3
limits:
memory: 200Mi
cpu: 1.0
nothing stands out. it's odd that there's no error in the logs. i've never used auth0 forward, so i'll try to get it up and running and see if i can reproduce
If you can email me directly, we can easily spin up a full temporary environment with everything in place and provide you access.
From: Marc Boorshtein @.> Sent: Tuesday, January 9, 2024 3:40 PM To: OpenUnison/openunison-k8s @.> Cc: plcarterco @.>; Author @.> Subject: Re: [OpenUnison/openunison-k8s] OpenUnison Server error when forward-auth0 requests access token (Issue #97)
nothing stands out. it's odd that there's no error in the logs. i've never used auth0 forward, so i'll try to get it up and running and see if i can reproduce
— Reply to this email directly, view it on GitHubhttps://github.com/OpenUnison/openunison-k8s/issues/97#issuecomment-1883912701, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEWCXAUJ3YURTSL5WKSUKALYNXBMJAVCNFSM6AAAAABBTSXDKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTHEYTENZQGE. You are receiving this because you authored the thread.Message ID: @.**@.>>
Thank you for the offer, but we reserve direct interaction for customers with support contracts, as we don't produce any closed source or "enterprise" editions that require licensing. If you're interested in a support contract, please reach out at https://www.tremolosecurity.com/contact/contact-us.
That said, I put together a debug build of 1.0.38 for you (the same version as is running per your helm chart) - docker.io/tremolosecurity/openunison-k8s-debug:1.0.38
This should have the build number 1.0.38-2023120501-DEBUG
in the logs. I added an extra catch for errors, for some reason the error is being caught but not logged so hopefully this will find it. You can set the top level image
option in your values.yaml and redeploy.
Completely understand, and thank you for the debug build. We'll try it out and let you know.
Hello,
Background we've been using Openunison successfully in regards to k8s dashbard, grafana, argocd, etc. we also successfully used forward-auth0 (not to be confused with traefik-forward-auth) for authentication against Auth0 as idp we're now trying to have forward-auth0 utilize Openunison (via Application CR) as idp within the cluster.
Issue Authentication is successful. But the forward-auth0 access token request to openunison fails with OpenUnison server error.
We were wondering if you all had encountered similar before or had any thoughts or advice. (We couldn't determine the mechanism to increase Opeunison logging level.)
OpenUnison log entries:
[2024-01-09 18:07:48,570][XNIO-1 task-5] INFO AccessLog - [AzSuccess] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/completefed - uid=patrickx-64-xneosavant.ai,ou=shadow,o=Tremolo - NONE [10.244.0.26] - [f80611c3db9a5fff7039260ae062fa712855a0cde] [2024-01-09 18:07:48,627][XNIO-1 task-5] INFO AccessLog - [Error] - forward-auth - https://k8sou.athens-aks.patb68.REDACTED.ai/auth/idp/forward-auth/token - cn=none - NONE [10.244.0.26] - [f8f6134e320a1bdfd6713491ccf6a63827721dbb3]
forward-auth0 log entires:
2024-01-09 18:07:47.304 INFO 1 --- [nio-8080-exec-7] d.f.infrastructure.auth0.Auth0Service : AuthorizationCodeExchange: code=LZK5rqNIAAD/xSkrmfvYzGAabMDcV2sSHrQBQ/MacxgYzb/vk3aSykqqoH6f2vX07wl86dZCDSsn+/d+NFNQpS2sfy00zVX/8/TPCQ3leyczqgI0LmiafzylYEZmO7/2B2wOpurQ4h6bVZKOitrwM+L0zaTiFmjFrkXtZTXyZEbGIbPXeASGYoOq5Acc5nCXzhLC7pgVlmYfsouKCdTD+hSvpmnGaiArvNAIo+K/uKR2S/pNT3rIFArE9KMBwvlR1CDPD4XGUyeNZmTsji8ukmLIF+AuWc/0QtOO3M1hpas9PAfMKuUX2ns+E4eyKkbQ6rUT9SVV8BwfOE7V3olA5d8ARp4BKd15HakkghTR2cPuEfRL88K+ENGyxnzHTTgxsUWPtOH4dSvPkCjbJwQiVzYa2T4RJ1At00kKIxIQqXjoWuMnG2kUlO9pm5AkjiN2Ct1Df6T+5YKHFTvJ7fvNMm1c2rKgJ1/nOfjUccUegrXGixK7qrEHhLexjIbOPs6UOkTB1Ze0HiZHOUYmM7l1juWGyl09zoEkX1RnUW6kaGM1k/2XI+ryM9Nels26395bNKgzCSHa7RE4xBQW3MmQ2d3zcgCGzCX08pwQy05Wnka1pZp0AFsvEqJGTGfhuuamx2HOy5+QEYxsZIXrPTs+8nfS0ccN9TjkA3qIVog299FdQu1ZBptPd7P1+oAu44uz8ZiaHt5yYk/TnCXva7VvXu7MvNW9klnHqdPNnuc91GhOluSWtAhfidnAkOpVydI0KgjC99NBONnNDwpVfxV3f1vvbK2r7KzhvzOf/vwH 2024-01-09 18:07:48.030 ERROR 1 --- [nio-8080-exec-7] d.f.i.e.ServerFaultExceptionMapper : ServerFaultExceptionMapper.toResponse: error=JSONObject["error_description"] not found.
Openunison Application CR configuration