Webhook error when deploying on bare metal k8s cluster

[BrentRose]

I've been able to deploy OU on a local minikube cluster fine. In this local test I'm using a self signed CA with a cert/key I've generated from it. When I try to deploy to a bare metal cluster I'm getting the following error.

2022/07/08 20:26:18 warning: Upgrade "orchestra-login-portal" failed: failed to create resource: Internal error occurred: failed calling webhook "applications-openunison.tremolo.io": Post "https://openunison-orchestra.openunison.svc:443/k8s/webhooks/v1/applications?timeout=5s": x509: certificate signed by unknown authority panic: failed to create resource: Internal error occurred: failed calling webhook "applications-openunison.tremolo.io": Post "https://openunison-orchestra.openunison.svc:443/k8s/webhooks/v1/applications?timeout=5s": x509: certificate signed by unknown authority

I've created the ou-tls-certificate secret with the full chain (root/intermediate/issuing/cert) and the key. This is different from my minikube test where I only had a root (no intermediate or issuing).

In my values.yaml file I added the unison-ca in the trusted_certs section. I've tried to use the root as well as issuing cert here but both had the same result.

My cert includes SANs for my k8sou and k8sdb sites.

My ingress seems to work prior to installing ou.

I'm probably missing something silly but I haven't been able to figure it out.

[mlbiam]

2022/07/08 20:26:18 warning: Upgrade "orchestra-login-portal" failed: failed to create resource: Internal error occurred: failed calling webhook "applications-openunison.tremolo.io": Post "https://openunison-orchestra.openunison.svc:443/k8s/webhooks/v1/applications?timeout=5s": x509: certificate signed by unknown authority panic: failed to create resource: Internal error occurred: failed calling webhook "applications-openunison.tremolo.io": Post "https://openunison-orchestra.openunison.svc:443/k8s/webhooks/v1/applications?timeout=5s": x509: certificate signed by unknown authority

This happens when the orchestra-login-portal chart is deployed before the new openunison-orchestra Pod is in a Ready state.

Kubernetes requires you to have a certificate for admission controller configuration, but it's bad form to embed a certificate in your helm charts, so to configure the admission controller that validates OpenUnison's configuration objects the helm chart templet embeds a dummy cert in the ValidatingWebhookConfiguration configuration's caBundle attribute for each webhook, then the operator updates the ValidatingWebhookConfiguration with the certificate from the unison-tls Secret (not ou-tls-certificate). The unison-tls certificate is used by the OpenUnison Pod, but doesn't get included in Ingress. Since the API server needs to explicitly trust the admission controller's cert, unison-tls needs to be used for each deployment.

Are you deploying the charts manually or using the ouctl tool?

[BrentRose]

@mlbiam I'm using ouctl

[BrentRose]

Does this mean I should deploy without ou-tls-cetificate an then follow the process to update it via https://openunison.github.io/knowledgebase/certificates/ ?

[mlbiam]

I'm using ouctl

hm, seems like a timing issue. What happens when you wait a min or two and manually run helm upgrade orchestra-login-portal tremolo/orchestra-login-portal -f /path/to/values.yaml? What are you using to deploy the bare metal cluster? (ie kubeadm or some other tool?)

Does this mean I should deploy without ou-tls-cetificate an then follow the process to update it via https://openunison.github.io/knowledgebase/certificates/ ?

no, these are different certificates. The issue you are having is from the API server --> OpenUnison in the container. Those certs are from the client --> Ingress --> OpenUnison.

[BrentRose]

I cleaned things up and redeployed using the following steps

Cleanup

1. Delete openunison namespace
2. Delete ingress-nginx namespace
3. Re-deploy ingress Note: the only thing special about my ingress is I put it on a specific worker and use the hostNetwork: true. This lets me point my OU dns records at a single worker and use 80/443. My browser wont let me connect at this point but you can see the ingress is working and using the default nginx cert.

Prep work

2. Create openunison namespace
3. Create the ou-tls-certificate secrete with my key and the full chain (root, intermediate, issuing, oucert with SANs for ou and db)
4. Update values.yaml
   • openunison_host
   • dashboard_host
   • k8s_url
   • createIngressCertificate: false
   • unison-ca (in trusted_certs section I add unison-ca with a pem_b64 of my base64 encoded root CA cert)
   • uncomment saml and for the idp_url I point to my SAML test lab instance in the tremolo lab).

Note: I do NOT set the api_server_host. I didn't use this in my local testing so I don't think it's necessary. Correct me if I'm wrong.

Deploy with Ouctl

1. ./ouctl install-auth-portal values.yaml (I get this error)

2. Wait for a few and try to helm upgrade portal
   • helm upgrade orchestra-login-portal tremolo/orchestra-login-portal -f values.yaml

At this point I'd expect the portal to be up but nginx is still using default cert and says k8sou-vmt.qa.local isn't found.

If I look at the openunison-operator logs I see the following repeated:

[2022-07-11 14:52:41,125][XNIO-1 task-6] INFO AccessLog - [Error] - UNKNOWN - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [127.0.0.1] - [f105ab0a8b812f65cbb5e242f35c7314fa3683878] [2022-07-11 14:52:41,125][XNIO-1 task-6] ERROR ConfigSys - Could not process request javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:116) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:89) ~[unison-sdk-1.0.31.jar:?] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:88) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:269) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.31.jar:?] at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.31.jar:?] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final] at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) ~[xnio-api-3.8.7.Final.jar:3.8.7.Final] at java.lang.Thread.run(Thread.java:829) ~[?:?]

[mlbiam]

At this point I'd expect the portal to be up but nginx is still using default cert and says k8sou-vmt.qa.local isn't found.

Can you post your values.yaml?

Note: I do NOT set the api_server_host. I didn't use this in my local testing so I don't think it's necessary. Correct me if I'm wrong.

You need this setting if you want your API server to integrate with OpenUnison via OIDC. If you set enable_impersonation to true this value gets ignored.

If I look at the openunison-operator logs I see the following repeated:

[2022-07-11 14:52:41,125][XNIO-1 task-6] INFO AccessLog - [Error] - UNKNOWN - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [127.0.0.1] - [f105ab0a8b812f65cbb5e242f35c7314fa3683878] [2022-07-11 14:52:41,125][XNIO-1 task-6] ERROR ConfigSys - Could not process request javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration

Even after updating the orchestra-login-portal helm chart?

[mlbiam]

with large quote blocks, can you paste it between ```?

for instances:

this: 
  is:
    a:
      large: code-block

[BrentRose]

network:
  openunison_host: "k8sou-vmt.qa.local"
  dashboard_host: "k8sdb-vmt.qa.local"
  api_server_host: "k8sapi.apps.ou.tremolo.dev"
  session_inactivity_timeout_seconds: 900
  k8s_url: https://atl1vmtk8s.qa.local:6443
  force_redirect_to_tls: false
  createIngressCertificate: false
  ingress_type: nginx
  ingress_annotations: {}

cert_template:
  ou: "Kubernetes"
  o: "MyOrg"
  l: "My Cluster"
  st: "State of Cluster"
  c: "MyCountry"

image: docker.io/tremolosecurity/openunison-k8s
myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: kubernetes
enable_impersonation: false

impersonation:
  use_jetstack: true
  jetstack_oidc_proxy_image: docker.io/tremolosecurity/kube-oidc-proxy:latest
  explicit_certificate_trust: true

dashboard:
  namespace: "kubernetes-dashboard"
  cert_name: "kubernetes-dashboard-certs"
  label: "k8s-app=kubernetes-dashboard"
  service_name: kubernetes-dashboard
  require_session: true

certs:
  use_k8s_cm: false

trusted_certs:
  - name: unison-ca
    pem_b64: REDACTED

  - name: ldaps
    pem_b64: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURORENDQWh5Z0F3SUJBZ0lRYlJOajZSS3F0cVZQdlc2NXFaeFhYakFOQmdrcWhraUc5dzBCQVFVRkFEQWkKTVNBd0hnWURWUVFEREJkQlJFWlRMa1ZPVkRKTE1USXVSRTlOUVVsT0xrTlBUVEFlRncweE5EQXpNamd3TVRBMQpNek5hRncweU5EQXpNalV3TVRBMU16TmFNQ0l4SURBZUJnTlZCQU1NRjBGRVJsTXVSVTVVTWtzeE1pNUVUMDFCClNVNHVRMDlOTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUEyczlKa2VOQUhPa1EKMVFZSmdqZWZVd2Nhb2dFTWNhVy9rb0ErYnU5eGJyNHJIeS8yZ04va2M4T2tvUHV3Si9uTmxPSU8rcytNYm5YUwpMOW1VVEM0T0s3dHJrRWppS1hCK0QrVlNZeTZpbVhoNnpwQnROYmVaeXgrcmRCbmFPdjNCeVpSbm5FQjhMbWhNCnZIQSs0Zi90OWZ4LzJ2dDZ3UHgvL1ZnSXE5eXVZWVVRUkxtMVdqeVVCRnJaZUdvU3BQbTBLZXdtK0IwYmhtTWIKZHlDKzNmaGFLQytVazFOUG9kRTI5NzNqTEJaSmVsWnhzWlk0MFd3OHpZUXdkR1lJYlhxb1RjKzFhL3g0ZjFFbgptNEFOcWdnSHR3K05xOHpoc3MzeVR0WStVWUtEUkJJTGRMVlpRaEhKRXhlMGtBZWlzZ014SS9iQndPMUhickZWCit6U25rK252Z1FJREFRQUJvMll3WkRBekJnTlZIU1VFTERBcUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SUcKQ2lzR0FRUUJnamNVQWdJR0NDc0dBUVVGQndNRE1CMEdBMVVkRGdRV0JCVHlKVWZZNjZ6WWJtOWkweGVZSHVGSQo0TU43dURBT0JnTlZIUThCQWY4RUJBTUNCU0F3RFFZSktvWklodmNOQVFFRkJRQURnZ0VCQU01a3o5T0tOU3VYCjh3NE5PZ25mSUZkYXpkMG5QbElVYnZEVmZRb055OVEwUzFTRlVWTWVrSVBOaVZoZkd6eWE5SXdSdEdiMVZhQlEKQVEyT1JJekhyOEEycjVVTkx4M21GanBKbWVPeFF3bFYwWCtnOHMrMjUzS1ZGeE9wUkU2eXlhZ24vQnh4cHRUTAphMVo0cWVRSkxENDJsZDFxR2xSd0Z0VlJtVkZaelZYVnJwdTdOdUZkM3Zsbm5PL3FLV1hVK3VNc2ZYdHNsMTNuCmVjMWt3MUV3cTJqbks4V0ltS1RRNy85V2JhSVkwZ3g4bW93Q0pTT3NScTBURTd6Sy9ONTVkck4xd1hKVnhXZTUKNE4zMmVDcW90WHk5ajlsemRrTmE3YXdiOXEzOG5XVnhQK3ZhNWpxTklEbGxqQjZ0RXh5NW4zczd0NktLNmc1agpUWmdWcXJaMyttcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQoK

monitoring:
  prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s

# Uncomment one of the below options for authentication

#active_directory:
#  base: cn=users,dc=ent2k12,dc=domain,dc=com
#  host: "192.168.2.75"
#  port: "636"
#  bind_dn: "cn=Administrator,cn=users,dc=ent2k12,dc=domain,dc=com"
#  con_type: ldaps
#  srv_dns: "false"

#oidc:
#  client_id: xxxxxx
#  issuer: https://xxxxxx.okta.com/
#  user_in_idtoken: false
#  domain: ""
#  scopes: openid email profile groups
#  claims:
#    sub: sub
#    email: email
#    given_name: given_name
#    family_name: family_name
#    display_name: name
#    groups: groups

#github:
#  client_id: d85d77c55a08c9bcbb15
#  teams: TremoloSecurity/

saml:
  idp_url: "https://portal.apps.tremolo.io/idp-test/metadata/REDACTED"

network_policies:
  enabled: false
  ingress:
    enabled: true
    labels:
      app.kubernetes.io/name: ingress-nginx
  monitoring:
    enabled: true
    labels:
      app.kubernetes.io/name: monitoring
  apiserver:
    enabled: false
    labels:
      app.kubernetes.io/name: kube-system

services:
  enable_tokenrequest: false
  token_request_audience: api
  token_request_expiration_seconds: 600
  node_selectors: []

openunison:
  replicas: 1
  non_secret_data:
    K8S_DB_SSO: oidc
    PROMETHEUS_SERVICE_ACCOUNT: system:serviceaccount:monitoring:prometheus-k8s
    SHOW_PORTAL_ORGS: "false"
  secrets: []
  html:
    image: docker.io/tremolosecurity/openunison-k8s-html
  enable_provisioning: false
  #az_groups:
  #- CN=k8s-users,CN=Users,DC=ent2k12,DC=domain,DC=com

#myvd_configmap: myvdconfig

# For Namespace as a Service

#database:
#  hibernate_dialect: org.hibernate.dialect.MySQL5InnoDBDialect
#  quartz_dialect: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
#  driver: com.mysql.jdbc.Driver
#  url: jdbc:mysql://mariadb.mariadb.svc.cluster.local:3306/unison
#  user: unison
#  validation: SELECT 1

#smtp:
#  host: blackhole.blackhole.svc.cluster.local
#  port: 1025
#  user: "none"
#  from: donotreply@domain.com
#  tls: false

[mlbiam]

You need this setting if you want your API server to integrate with OpenUnison via OIDC. If you set enable_impersonation to true this value gets ignored.

Ignore this. I have it backwards. Sorry about that.

Your config looks OK. We'll add something to ouctl that validates the webhooks are ready before launching the last helm chart.

The error [2022-07-11 14:52:41,125][XNIO-1 task-6] INFO AccessLog - [Error] - UNKNOWN - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [127.0.0.1] - [f105ab0a8b812f65cbb5e242f35c7314fa3683878] is expected until the orchestra-login-portal chart is deployed because that's when the identity provider is configured. Is it still happening after the orchestra-login-portal chart is deployed? If so, can you look in your logs for something like:

[2022-07-11 15:38:04,205][main] INFO LoadApplicationsFromK8s - Adding application anonfiles?

[BrentRose]

Still getting this repeating in the logs. I also grep'ed for LoadApplicatioinFromK8s and didn't see any results.

[2022-07-11 15:48:11,134][XNIO-1 task-14] INFO  AccessLog - [Error] - UNKNOWN - https://127.0.0.1:8443/auth/idp/k8sIdp/.well-known/openid-configuration - cn=none - NONE [127.0.0.1] - [f4c0f4986134c0831a2b1a640b5574720dbb06632]
[2022-07-11 15:48:11,134][XNIO-1 task-14] ERROR ConfigSys - Could not process request
javax.servlet.ServletException: Unknown URI : /auth/idp/k8sIdp/.well-known/openid-configuration
        at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:116) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:89) ~[unison-sdk-1.0.31.jar:?]
        at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:88) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:269) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) ~[unison-server-core-1.0.31.jar:?]
        at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) ~[unison-server-core-1.0.31.jar:?]
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100) ~[undertow-servlet-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852) ~[undertow-core-2.2.18.Final.jar:2.2.18.Final]
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) ~[jboss-threads-2.3.6.Final.jar:2.3.6.Final]
        at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) ~[xnio-api-3.8.7.Final.jar:3.8.7.Final]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
[2022-07-11 15:48:11,156][XNIO-1 task-14] INFO  AccessLog - [AzSuccess] - CheckAlive - https://127.0.0.1:8443/check_alive - uid=Anonymous,o=Tremolo -  [127.0.0.1] - [fb97b539178113e9f123a4f23bb8e68aaf0d1bb67]

[mlbiam]

Odd. kubectl get applications -n openunison?

[BrentRose]

[QA\brose@atl1q51k8sb5n05 ou]$ kubectl get applications.openunison.tremolo.io -n openunison
No resources found in openunison namespace.

[BrentRose]

ah, here is something interesting.

[QA\brose@atl1q51k8sb5n05 ou]$ kubectl get applications.openunison.tremolo.io -A
NAMESPACE   NAME                     AGE
users       anonfiles                141m
users       completelogin            141m
users       completelogin-cli        141m
users       dashboard                141m
users       k8s-login-cli            141m
users       k8s-login-cli-redirect   141m
users       k8s-login-tokens         141m
users       k8sidp                   141m
users       logout                   141m
users       metrics                  141m
users       root-redirect            141m
users       scale                    141m
users       scale-session-check      141m
users       token                    141m

[mlbiam]

Yeah, that's not right. Looks like I didn't give you the ns switch. Try this:

helm upgrade orchestra-login-portal tremolo/orchestra-login-portal -f /path/to/values.yaml -n openunison

[BrentRose]

The users namespaces is where the service account I'm currently using to authenticate resides.

[BrentRose]

[QA\brose@atl1q51k8sb5n05 ou]$ helm upgrade orchestra-login-portal tremolo/orchestra-login-portal -f values.yaml -n openunison
Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "applications-openunison.tremolo.io": Post "https://openunison-orchestra.openunison.svc:443/k8s/webhooks/v1/applications?timeout=5s": x509: certificate signed by unknown authority

[mlbiam]

Well, the good news is we can rule out a timing issue. You mentioned using a ServiceAccount, does this account have an RBAC binding to cluster-admin?

Run kubectl get validatingwebhookconfigurations openunison-workflow-validation-orchestra -o json | jq -r '.webhooks[0].clientConfig.caBundle' | base64 -d. The output should match the output from kubectl get secret unison-tls -n openunison -o json | jq -r '.data["tls.crt"]' | base64 -d ?

[BrentRose]

Ah... they do not match

[mlbiam]

Ah... they do not match

great! that gives us a good idea as to the issue. look in the logs for the operator (label openunison-operator), any errors or exceptions in there?

[BrentRose]

Attaching the operator log.

operator_logs.txt

[BrentRose]

Would it help if I try to remove the portal and deploy with helm using the correct namespace? Maybe we got out of whack when I used the helm upgrade command without specifying the namespace?

[BrentRose]

It did create other resources in 'users'

[QA\brose@atl1q51k8sb5n05 ou]$ kubectl get all -n users NAME READY STATUS RESTARTS AGE pod/ouhtml-orchestra-login-portal-55ccc56cc6-ttnb8 1/1 Running 0 3h16m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ouhtml-orchestra-login-portal ClusterIP 10.3.21.155 8080/TCP 3h16m

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ouhtml-orchestra-login-portal 1/1 1 1 3h16m

NAME DESIRED CURRENT READY AGE replicaset.apps/ouhtml-orchestra-login-portal-55ccc56cc6 1 1 1 3h16m

[mlbiam]

What's your cluster built with? Is it OpenShift?

[BrentRose]

No, it's a stick built k8s cluster running on FCOS.

[mlbiam]

That is so odd, the operator thinks its running on OpenShift. in your values.yaml, under openunison.non_secret_data add:

IGNORE_OPENSHIFT: "true"

and try re-running ouctl

[BrentRose]

bazinga!

[mlbiam]

huh, i mean YAY! now, why does the operator think you're on openshift???

[BrentRose]

Possibly something to do with my nodes running on Fedora Core OS?

[mlbiam]

can you run kubectl api-resources -o wide | grep openshift ?

[BrentRose]

[QA\brose@atl1q51k8sb5n05 ou]$ kubectl api-resources | grep openshift
volumereplicationclasses          vrc               replication.storage.openshift.io/v1alpha1   false        VolumeReplicationClass
volumereplications                vr                replication.storage.openshift.io/v1alpha1   true         VolumeReplication

[BrentRose]

I can't say for sure but I'll bet this is related to our rook/ceph implementation.

[mlbiam]

Yep, that would do it. OK, well we need to just get that disabled anyways. It's not needed anymore and just causes lots of heartburn. Thanks for the verification!

[BrentRose]

@mlbiam Thanks for your help getting my rolling on this!