Closed santiagon610 closed 5 months ago
mlbiam
commented
5 months ago I think spec.sources[1].helm.valueObjects.services.token_request_audience is the issue:
token_request_audience: apiTry changing this to token_request_audience: https://kubernetes.default.svc. Once upon a time api was an accepted audience, but most distros have fixed that.
santiagon610
commented
5 months ago Thanks, @mlbiam - I made that change and redeployed (removed the app entirely from Argo and let it recreate, just in case there were any gremlins remaining), and no change in behavior.
[2024-04-15 14:10:24,175][main] WARN BrokerRegistry - Broker localhost not started so using local instead
[2024-04-15 14:10:24,245][main] INFO OpenShiftTarget - Config url='https://kubernetes.default.svc'
[2024-04-15 14:10:24,253][main] INFO OpenShiftTarget - Use Token: 'true'
[2024-04-15 14:10:24,254][main] INFO OpenShiftTarget - tokenType: 'tokenapi'
[2024-04-15 14:10:24,257][main] INFO OpenShiftTarget - Config tokenPath='/var/run/secrets/tokens/ou-token'
[2024-04-15 14:10:24,531][main] INFO OpenShiftTarget - label: 'Local Deployment'
[2024-04-15 14:10:24,535][main] WARN OpenShiftTarget - gitUrl not found
[2024-04-15 14:10:24,536][main] INFO OpenShiftTarget - drqueues: ''
com.tremolosecurity.provisioning.core.ProvisioningException: Could not load CRDs
at com.tremolosecurity.k8s.watch.K8sWatcher.initalRun(K8sWatcher.java:176)
at com.tremolosecurity.provisioning.targets.LoadTargetsFromK8s.loadDynamicTargets(LoadTargetsFromK8s.java:253)
at com.tremolosecurity.provisioning.core.ProvisioningEngineImpl.generateTargets(ProvisioningEngineImpl.java:897)
at com.tremolosecurity.provisioning.core.ProvisioningEngineImpl.<init>(ProvisioningEngineImpl.java:492)
at com.tremolosecurity.config.util.UnisonConfigManagerImpl.initialize(UnisonConfigManagerImpl.java:484)
at com.tremolosecurity.filter.UnisonServletFilter.init(UnisonServletFilter.java:369)
at com.tremolosecurity.openunison.OpenUnisonServletFilter.init(OpenUnisonServletFilter.java:118)
at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:111)
at io.undertow.servlet.core.ManagedFilter.createFilter(ManagedFilter.java:86)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:598)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:605)
at com.tremolosecurity.openunison.undertow.OpenUnisonOnUndertow.main(OpenUnisonOnUndertow.java:357)
Caused by: java.io.IOException: Unexpected result calling 'https://kubernetes.default.svc/apis/apiextensions.k8s.io/v1/customresourcedefinitions/targets.openunison.tremolo.io' - 401 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
at com.tremolosecurity.unison.openshiftv3.OpenShiftTarget.callWS(OpenShiftTarget.java:546)
at com.tremolosecurity.k8s.watch.K8sWatcher.findCrdUri(K8sWatcher.java:198)
at com.tremolosecurity.k8s.watch.K8sWatcher.initalRun(K8sWatcher.java:105)
... 14 more
Exception in thread "main" jakarta.servlet.ServletException: com.tremolosecurity.provisioning.core.ProvisioningException: Could not load CRDs
at com.tremolosecurity.filter.UnisonServletFilter.init(UnisonServletFilter.java:400)
at com.tremolosecurity.openunison.OpenUnisonServletFilter.init(OpenUnisonServletFilter.java:118)
at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:111)
at io.undertow.servlet.core.ManagedFilter.createFilter(ManagedFilter.java:86)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:598)
at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:605)
at com.tremolosecurity.openunison.undertow.OpenUnisonOnUndertow.main(OpenUnisonOnUndertow.java:357)
Caused by: com.tremolosecurity.provisioning.core.ProvisioningException: Could not load CRDs
at com.tremolosecurity.k8s.watch.K8sWatcher.initalRun(K8sWatcher.java:176)
at com.tremolosecurity.provisioning.targets.LoadTargetsFromK8s.loadDynamicTargets(LoadTargetsFromK8s.java:253)
at com.tremolosecurity.provisioning.core.ProvisioningEngineImpl.generateTargets(ProvisioningEngineImpl.java:897)
at com.tremolosecurity.provisioning.core.ProvisioningEngineImpl.<init>(ProvisioningEngineImpl.java:492)
at com.tremolosecurity.config.util.UnisonConfigManagerImpl.initialize(UnisonConfigManagerImpl.java:484)
at com.tremolosecurity.filter.UnisonServletFilter.init(UnisonServletFilter.java:369)
... 9 more
Caused by: java.io.IOException: Unexpected result calling 'https://kubernetes.default.svc/apis/apiextensions.k8s.io/v1/customresourcedefinitions/targets.openunison.tremolo.io' - 401 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
at com.tremolosecurity.unison.openshiftv3.OpenShiftTarget.callWS(OpenShiftTarget.java:546)
at com.tremolosecurity.k8s.watch.K8sWatcher.findCrdUri(K8sWatcher.java:198)
at com.tremolosecurity.k8s.watch.K8sWatcher.initalRun(K8sWatcher.java:105)
... 14 moreI may need to take a deeper dive into the docs to make sure that the app has all of the permissions that it's expecting to have, though 401 seems less like permissions but more like the OpenUnison service authenticating against Kubernetes itself.
mlbiam
commented
5 months ago Hm, what kubernetes distro? Try setting it to https://kubernetes.default.svc.cluster.local
santiagon610
commented
5 months ago Thanks, @mlbiam - this is running in DigitalOcean Managed Kubernetes. Though changing the URL didn't end up changing the downstream call from the OpenUnison pod (it was still targeting https://kubernetes.default.svc), I was able to work around this behavior by simply disabling spec.sources[1].helm.valuesObject.services.enable_tokenrequest.
mlbiam
commented
5 months ago by simply disabling spec.sources[1].helm.valuesObject.services.enable_tokenrequest
Yeah, that's where i was going to go next. If you take a look at your pod's token and see the aud claim that would be the correct value to use. The target URL won't change, that's standard across implementations, but if the required audiences is customized by digital ocean that could cause issues. Disabling tokenrequest will still work, its just the tokens used by kube-oidc-proxy are now good for an hour i think instead of 10 min (minimum kube will create them for)
santiagon610
commented
5 months ago Awesome, thanks for the help! Looking forward to getting this working in my POC cluster.
I'm currently in the process of installing OpenUnison on a Kubernetes cluster, and am getting an error in the OpenUnison pod where it appears to be getting a 401 when attempting to load its CRDs:
This is running on Kubernetes v1.29.1 in DigitalOcean.
I deployed this using an ArgoCD application wrapper around the ArgoCD application provided in the docs. The
repoURLare leading to a mirror of the Helm chart, and therawchart referenced is simply to generate the secrets and the cluster role binding for the user account being passed in via the OIDC claim.I'm open to suggestions - I've already tried disabling
openunison.enable_provisioningand cleaned up any references to service accounts that aren't present in this cluster (like Prometheus, for example).Has anyone else run into something similar? Thanks!