Closed davisonja closed 5 months ago
mlbiam
commented
6 months ago Can you provide your values.yaml? The logs for the openunison-orchestra pod may have some information. Most likely cause is that the workflows weren't deployed or integrated. What does kubectl get authchain google-ws-load-groups -n openunison return?
davisonja
commented
6 months ago ❯ kubectl get authchain google-ws-load-groups -n openunison
NAME AGE
google-ws-load-groups 46hThere wasn't anything directly obvious as relating to groups in the orchestra pod logs, tho I'm not sure what this relates to:
[2024-05-22 01:08:10,267][Thread-21] WARN OpenShiftTarget - Unexpected result calling 'https://kubernetes.default.svc/apis/openunison.tremolo.io/v2/namespaces/openunison/oidc-sessions/xabcdefgh-abcd-abcd-abcd-abcdefghijklx' - 404 / {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"oidc-sessions.openunison.tremolo.io \"xabcdefgh-abcd-abcd-abcd-abcdefghijklx\" not found","reason":"NotFound","details":{"name":"xabcdefgh-abcd-abcd-abcd-abcdefghijklx","group":"openunison.tremolo.io","kind":"oidc-sessions"},"code":404}They appear in groups of three, each with their own value between the xs (which I've alphabetised above)
values.yaml, with commented out things removed, and ids cropped; most of it is the same as the defaults:
network:
openunison_host: "k8sou-staging.fred.com"
dashboard_host: "k8sdb-staging.fred.com"
api_server_host: "k8sapi-staging.fred.com"
session_inactivity_timeout_seconds: 900
k8s_url: https://k8s-api-staging.internal.fred.com:6443
force_redirect_to_tls: false
createIngressCertificate: false
ingress_type: nginx
ingress_annotations: {cert-manager.io/issuer: "letsencrypt-staging"}
cert_template:
ou: "Kubernetes"
o: "MyOrg"
l: "My Cluster"
st: "State of Cluster"
c: "MyCountry"
myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: openunison-cp
enable_impersonation: true
impersonation:
use_jetstack: true
explicit_certificate_trust: true
dashboard:
namespace: "kubernetes-dashboard"
cert_name: "kubernetes-dashboard-certs"
label: "k8s-app=kubernetes-dashboard"
service_name: kubernetes-dashboard
require_session: true
certs:
use_k8s_cm: false
trusted_certs: []
monitoring:
prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s
oidc:
client_id: id-body.apps.googleusercontent.com
issuer: https://accounts.google.com
user_in_idtoken: false
domain: ""
scopes: openid email profile
claims:
sub: email
email: email
given_name: given_name
family_name: family_name
display_name: name
groups: groups
network_policies:
enabled: false
ingress:
enabled: true
labels:
app.kubernetes.io/name: ingress-nginx
monitoring:
enabled: true
labels:
app.kubernetes.io/name: monitoring
apiserver:
enabled: true
labels:
app.kubernetes.io/name: kube-system
services:
enable_tokenrequest: false
token_request_audience: api
token_request_expiration_seconds: 600
node_selectors: []
openunison:
replicas: 1
non_secret_data:
K8S_DB_SSO: oidc
PROMETHEUS_SERVICE_ACCOUNT: system:serviceaccount:monitoring:prometheus-k8s
SHOW_PORTAL_ORGS: "false"
secrets: []
enable_provisioning: false
use_standard_jit_workflow: true
#az_groups:
#- CN=k8s-users,CN=Users,DC=ent2k12,DC=domain,DC=com
google_ws:
admin_email: "julian@fred.com"
service_account_email: "service.account@app.iam.gserviceaccount.com"
openunison.include_auth_chain: google-ws-load-groupsI think I see the issue. openunison.include_auth_chain: google-ws-load-groups needs to be adding include_auth_chain: google-ws-load-groups to the openunison section of your values.yaml.
davisonja
commented
6 months ago That'll teach me for too blindly copy and pasting, rather than thinking about what the aim was!
That certainly changed the behaviour, tho now I'm getting
[2024-05-22 03:15:10,066][XNIO-1 task-1] ERROR ConfigSys - Could not process request
org.graalvm.polyglot.PolyglotException: null
at java.base/java.util.Base64$Decoder.decode(Base64.java:561) ~[?:?]
at com.tremolosecurity.util.JSUtils.base64Decode(JSUtils.java:38) ~[unison-sdk-1.0.40.jar:?]
at <js>.doAuth(Unnamed:61) ~[?:?]
at org.graalvm.polyglot.Value.execute(Value.java:880) ~[graal-sdk-22.3.4.jar:?]
at com.tremolosecurity.proxy.auth.JavaScriptAuth.doGet(JavaScriptAuth.java:66) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.execAuth(AuthManagerImpl.java:452) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:134) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:88) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.FullMappingAuthMech.doGet(FullMappingAuthMech.java:85) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.execAuth(AuthManagerImpl.java:452) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:134) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.sys.AuthManagerImpl.nextAuth(AuthManagerImpl.java:88) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.unison.proxy.auth.openidconnect.OpenIDConnectAuthMech.doGet(OpenIDConnectAuthMech.java:477) ~[unison-auth-openidconnect-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:196) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:89) ~[unison-sdk-1.0.40.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:88) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:296) [unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) [unison-server-core-1.0.40.jar:?]
at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) [unison-server-core-1.0.40.jar:?]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101) [undertow-servlet-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859) [undertow-core-2.3.12.Final.jar:2.3.12.Final]
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) [xnio-api-3.8.13.Final.jar:3.8.13.Final]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]errors when I attempt to login to the portal (I can choose my Google Workspace account, but then get an error page, and the orchestra pod shows several of those errors)
mlbiam
commented
6 months ago Did you create the googlews Secret in the openunison namespace with the key key with your Google SA?
davisonja
commented
6 months ago Oh dear, in chasing down problems with the Google SA that value got loaded with the key data not key.
Switching that so the value was called key instead sorted it all, thank you very much! It works beautifully now.
I did initially (once thek ey was sorted) get a 403 from Google (in the orchestra pod):
Admin SDK API has not been used in project <id-number> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/admin.googleapis.com/overview?project=<id-number> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.Enabling the API has it all working and groups coming through nicely. But... Is that a sign I've messed something else up?
davisonja
commented
5 months ago This is all working nicely, since I fixed up the Secret, thanks for all the help!
vilnitsky
commented
2 weeks ago wanted to leave a note for this article https://www.tremolo.io/post/kubernetes-and-google-workspace
Give your service account a name, id, and description. Don't worry about granting any service access or user access. Once the service account is created, click on it and get the Unique ID on the next screen. Also, click on KEYS and then ADD KEY. Choose a JSON file and then download it. The JSON file will contain a PEM encoded private key. Create a Secret in the openunison namespace called googlews with a single key called key that contains the private key (and ONLY the private key, do not include the rest of the json). It should look like:
apiVersion: v1
kind: Secret
metadata:
name: googlews
namespace: openunison
type: Opaque
data:
key: LS0tLS1CRUdJ..not sure if it's obvious for everyone, I had an issue because json file also contains new lines in pem like this:
z/RSQ/b+\n-----END PRIVATE KEY-----\n",
it leads to a broken encoded certificate in base64
I had to replace all new lines '\n' with actual new lines first and then encode to base64 to make it working
I've been following https://www.tremolosecurity.com/post/kubernetes-and-google-workspace, which has worked well for using Google Workspace as the source of our user info.
However, the portal is showing nothing for the "Roles" section of the user profile, and the group names are (not unexpectedly, given nothing is showing on the profile) not working for RBAC dashboard access.
Is there somewhere I can find some logs on the group fetching authentication chain?