[BUG] Connectors failing to start due to going to legacy Keycloak URL for Token

ChrisJBurns commented 1 year ago

Describe the issue:

Connectors pod is failing to start because it keeps trying to go to the following path for a token: POST /auth/realms/realms/camunda-platform/protocol/openid-connect/token. These logs are from our istio sidecar. In the connectors container we get An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found before it eventually crashes and restarts container.

Expected behavior:

I expect the connectors container to start without issue and not use the old /auth/ path. I have set the context to / because we are using keycloak v19.

How to reproduce: Using the Chart, here is my values file:

    global:
      image:
        registry: our.registry
      identity:
        auth:
          enabled: true
          publicIssuerUrl: "https://auth.test.our.domain/realms/camunda-platform"
          connectors:
            existingSecret: camunda-connectors-identity-secret
          operate:
            redirectUrl: https://camunda-operate.test.our.domain
            existingSecret: camunda-operate-identity-secret
          tasklist:
            existingSecret: camunda-tasklist-identity-secret
            redirectUrl: https://camunda-tasklist.test.our.domain
          optimize:
            existingSecret: camunda-optimize-identity-secret
            redirectUrl: https://camunda-optimize.test.our.domain
          zeebe:
            existingSecret: camunda-zeebe-identity-secret
        keycloak:
          internal: true
          legacy: false
          url:
            protocol: "http"
            host: "auth-idp.apps"
            port: 80
          contextPath: "/"
          realm: "realms/camunda-platform"
          # auth:
          #   adminUser: "admin"
          #   existingSecret: "camunda-keycloak"
          #   existingSecretKey: "admin-password"
    zeebe:
      image:
        repository: third-party/camunda/zeebe
        tag: 8.3.0 # {"$imagepolicy": "flux-system:camunda-zeebe:tag"}
    zeebe-gateway:
      image:
        repository: third-party/camunda/zeebe
        tag: 8.3.0  # {"$imagepolicy": "flux-system:camunda-zeebe:tag"}
    operate:
      image:
        repository: third-party/camunda/operate
        tag: 8.3.0 # {"$imagepolicy": "flux-system:camunda-operate:tag"}
    tasklist:
      image:
        repository: third-party/camunda/tasklist
        tag: 8.3.0 # {"$imagepolicy": "flux-system:camunda-tasklist:tag"}
    connectors:
      image:
        repository: third-party/camunda/connectors-bundle
        tag: 8.3.0 # {"$imagepolicy": "flux-system:camunda-connectors-bundle:tag"}
      env:
      - name: CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL
        value: "http://auth-idp.apps:80"
    optimize:
      image:
        repository: third-party/camunda/optimize
        tag: 3.11.0 # {"$imagepolicy": "flux-system:camunda-optimize:tag"}
      podLabels:
        sidecar.istio.io/inject: "false"
    identity:
      image:
        repository: third-party/camunda/identity
        tag: 8.3.0 # {"$imagepolicy": "flux-system:camunda-identity:tag"}
      keycloak:
        enabled: false
        httpRelativePath: "/"
      env:
        - name: IDENTITY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: camunda-keycloak
              key: admin-password
        - name: KEYCLOAK_REALM
          value: camunda-platform
        - name: KEYCLOAK_SETUP_REALM
          value: camunda-platform

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

Platform: AWS EKS
Helm CLI version: 3.11.2
Chart version:8.3.0

ChrisJBurns commented 1 year ago

I've also noticed that some environment variables aren't even being set inside of the container, despite them being on the Deployment. When I change for example the following values CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL to CAMUNDA_OPERATE_CLIENT_KEYCLOAK_URL ensuring all hyphens are underscored, the variable gets sourced into the container. Perhaps this needs a separate fix to go through all of the charts?

aabouzaid commented 1 year ago

@ChrisJBurns As mentioned in #950 the vars are loaded via K8s, but doesn't show if the user execs via shell, but it's loaded normally for the app (test it with env command directly and it will show).

So could you please try to set CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL? It should work.

ChrisJBurns commented 1 year ago

@aabouzaid Can confirm that the variable is being set (which is the internal k8s svc address for our Keycloak:

$ kubectl -n namespace  exec camunda-connectors-69f9fb75c7-wn7gg  -- env
...
CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL=http://auth-idp:80
...

However I still get the errors

2023-10-19T10:30:26.193Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:30:31.203Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:30:36.211Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:30:41.220Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:30:46.229Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:30:51.238Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:30:56.246Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:01.256Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:06.264Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:11.276Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:16.287Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:21.296Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:26.305Z  WARN 1 --- [           main] s.c.c.OperateClientProdAutoConfiguration : An attempt to connect to Operate failed: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:26.308Z  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'io.camunda.connector.runtime.inbound.lifecycle.InboundConnectorRestController': Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'inboundConnectorManager' defined in class path resource [io/camunda/connector/runtime/inbound/lifecycle/InboundConnectorLifecycleConfiguration.class]: Unsatisfied dependency expressed through method 'inboundConnectorManager' parameter 1: Error creating bean with name 'springInboundConnectorContextFactory' defined in class path resource [io/camunda/connector/runtime/inbound/InboundConnectorRuntimeConfiguration.class]: Unsatisfied dependency expressed through method 'springInboundConnectorContextFactory' parameter 4: Error creating bean with name 'springOperateClientAdapter' defined in class path resource [io/camunda/connector/runtime/inbound/operate/OperateClientConfiguration.class]: Unsatisfied dependency expressed through method 'springOperateClientAdapter' parameter 0: Error creating bean with name 'myOperateClient' defined in class path resource [io/camunda/connector/runtime/InboundConnectorsAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
2023-10-19T10:31:26.316Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2023-10-19T10:31:26.336Z  INFO 1 --- [           main] .s.b.a.l.ConditionEvaluationReportLogger : 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2023-10-19T10:31:26.349Z ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'io.camunda.connector.runtime.inbound.lifecycle.InboundConnectorRestController': Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'inboundConnectorManager' defined in class path resource [io/camunda/connector/runtime/inbound/lifecycle/InboundConnectorLifecycleConfiguration.class]: Unsatisfied dependency expressed through method 'inboundConnectorManager' parameter 1: Error creating bean with name 'springInboundConnectorContextFactory' defined in class path resource [io/camunda/connector/runtime/inbound/InboundConnectorRuntimeConfiguration.class]: Unsatisfied dependency expressed through method 'springInboundConnectorContextFactory' parameter 4: Error creating bean with name 'springOperateClientAdapter' defined in class path resource [io/camunda/connector/runtime/inbound/operate/OperateClientConfiguration.class]: Unsatisfied dependency expressed through method 'springOperateClientAdapter' parameter 0: Error creating bean with name 'myOperateClient' defined in class path resource [io/camunda/connector/runtime/InboundConnectorsAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:240)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)
    at io.camunda.connector.runtime.app.ConnectorRuntimeApplication.main(ConnectorRuntimeApplication.java:26)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'inboundConnectorManager' defined in class path resource [io/camunda/connector/runtime/inbound/lifecycle/InboundConnectorLifecycleConfiguration.class]: Unsatisfied dependency expressed through method 'inboundConnectorManager' parameter 1: Error creating bean with name 'springInboundConnectorContextFactory' defined in class path resource [io/camunda/connector/runtime/inbound/InboundConnectorRuntimeConfiguration.class]: Unsatisfied dependency expressed through method 'springInboundConnectorContextFactory' parameter 4: Error creating bean with name 'springOperateClientAdapter' defined in class path resource [io/camunda/connector/runtime/inbound/operate/OperateClientConfiguration.class]: Unsatisfied dependency expressed through method 'springOperateClientAdapter' parameter 0: Error creating bean with name 'myOperateClient' defined in class path resource [io/camunda/connector/runtime/InboundConnectorsAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:545)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:910)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
    ... 19 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'springInboundConnectorContextFactory' defined in class path resource [io/camunda/connector/runtime/inbound/InboundConnectorRuntimeConfiguration.class]: Unsatisfied dependency expressed through method 'springInboundConnectorContextFactory' parameter 4: Error creating bean with name 'springOperateClientAdapter' defined in class path resource [io/camunda/connector/runtime/inbound/operate/OperateClientConfiguration.class]: Unsatisfied dependency expressed through method 'springOperateClientAdapter' parameter 0: Error creating bean with name 'myOperateClient' defined in class path resource [io/camunda/connector/runtime/InboundConnectorsAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:545)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:910)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
    ... 33 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'springOperateClientAdapter' defined in class path resource [io/camunda/connector/runtime/inbound/operate/OperateClientConfiguration.class]: Unsatisfied dependency expressed through method 'springOperateClientAdapter' parameter 0: Error creating bean with name 'myOperateClient' defined in class path resource [io/camunda/connector/runtime/InboundConnectorsAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:801)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:545)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:910)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
    ... 47 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'myOperateClient' defined in class path resource [io/camunda/connector/runtime/InboundConnectorsAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:654)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:642)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:910)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788)
    ... 61 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'myOperateClient' threw exception with message: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:171)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650)
    ... 75 common frames omitted
Caused by: java.lang.RuntimeException: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at io.camunda.zeebe.spring.client.configuration.OperateClientProdAutoConfiguration.lambda$camundaOperateClient$0(OperateClientProdAutoConfiguration.java:54)
    at io.github.resilience4j.retry.Retry.lambda$decorateSupplier$4(Retry.java:211)
    at io.github.resilience4j.retry.Retry.executeSupplier(Retry.java:361)
    at io.camunda.zeebe.spring.client.configuration.OperateClientProdAutoConfiguration.camundaOperateClient(OperateClientProdAutoConfiguration.java:46)
    at io.camunda.connector.runtime.InboundConnectorsAutoConfiguration.myOperateClient(InboundConnectorsAutoConfiguration.java:50)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139)
    ... 76 common frames omitted
Caused by: io.camunda.operate.exception.OperateException: Error 404 obtaining access token : Not Found
    at io.camunda.operate.auth.SelfManagedAuthentication.authenticate(SelfManagedAuthentication.java:99)
    at io.camunda.operate.CamundaOperateClient$Builder.build(CamundaOperateClient.java:273)
    at io.camunda.zeebe.spring.client.configuration.OperateClientProdAutoConfiguration.lambda$camundaOperateClient$0(OperateClientProdAutoConfiguration.java:51)
    ... 85 common frames omitted

Stream closed EOF for camunda-connectors-69f9fb75c7-wn7gg (connectors)

aabouzaid commented 1 year ago

Looks like the context is set via another var. I will contact the Connectors team to debug that issue.

ChrisJBurns commented 1 year ago

Thanks @aabouzaid really appreciate the fast feedback!

lukas-beumer commented 1 year ago

@ChrisJBurns Can you provide the Kubernetes manifest which is used in your cluster? $ kubectl get deployment -n camunda camunda-platform-connectors -oyaml

ChrisJBurns commented 1 year ago

@lukas-beumer

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: camunda
    meta.helm.sh/release-namespace: camunda
  creationTimestamp: "2023-10-19T13:35:27Z"
  generation: 1
  labels:
    app: camunda-platform
    app.kubernetes.io/component: connectors
    app.kubernetes.io/instance: camunda
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: camunda-platform
    app.kubernetes.io/part-of: camunda-platform
    app.kubernetes.io/version: 8.3.0
    helm.sh/chart: camunda-platform-8.3.0
    helm.toolkit.fluxcd.io/name: camunda
    helm.toolkit.fluxcd.io/namespace: camunda
  name: camunda-connectors
  namespace: camunda
  resourceVersion: "88031721"
  uid: 112c17b6-f773-4d8f-9a15-d5017a900395
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: camunda-platform
      app.kubernetes.io/component: connectors
      app.kubernetes.io/instance: camunda
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: camunda-platform
      app.kubernetes.io/part-of: camunda-platform
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: camunda-platform
        app.kubernetes.io/component: connectors
        app.kubernetes.io/instance: camunda
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: camunda-platform
        app.kubernetes.io/part-of: camunda-platform
        app.kubernetes.io/version: 8.3.0
        helm.sh/chart: camunda-platform-8.3.0
    spec:
      containers:
      - env:
        - name: SERVER_PORT
          value: "8080"
        - name: CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL
          value: http://camunda-keycloak:80
        - name: CAMUNDA_OPERATE_CLIENT_CLIENT-ID
          value: connectors
        - name: CAMUNDA_OPERATE_CLIENT_CLIENT-SECRET
          valueFrom:
            secretKeyRef:
              key: connectors-secret
              name: camunda-connectors-identity-secret
        - name: CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM
          value: realms/camunda-platform
        - name: CAMUNDA_OPERATE_CLIENT_URL
          value: http://camunda-operate:80
        - name: ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS
          value: camunda-zeebe-gateway:26500
        - name: ZEEBE_CLIENT_SECURITY_PLAINTEXT
          value: "true"
        - name: ZEEBE_CLIENT_ID
          value: zeebe
        - name: ZEEBE_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              key: zeebe-secret
              name: camunda-zeebe-identity-secret
        - name: ZEEBE_AUTHORIZATION_SERVER_URL
          value: http://auth-idp.apps:80/realms/camunda-platform/protocol/openid-connect/token
        - name: ZEEBE_TOKEN_AUDIENCE
          value: zeebe-api
        - name: CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL
          value: http://auth-idp.apps:80
        image: camunda/connectors-bundle:8.3.0
        imagePullPolicy: IfNotPresent
        name: connectors
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /actuator/health/readiness
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: "1"
            memory: 1Gi
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1003
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1003
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp
status:
  conditions:
  - lastTransitionTime: "2023-10-19T13:35:27Z"
    lastUpdateTime: "2023-10-19T13:35:27Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2023-10-19T13:35:27Z"
    lastUpdateTime: "2023-10-19T13:35:28Z"
    message: ReplicaSet "camunda-connectors-7bc54d9cdc" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

lukas-beumer commented 1 year ago

- name: CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM
  value: realms/camunda-platform

Must be changed to "camunda-platform" only!

ChrisJBurns commented 1 year ago

@lukas-beumer I think this will cause a conflict.

I added realm: "realms/camunda-platform" because the default which is realm: "/realms/camunda-platform" conflicts when contextPath: "/" because it ends up being keycloak.ourdomain//realms which is invalid.

It does strip out the /realms/ bit, but for it to do this, it has to contain /realms/, but as described previously, if this is true, it conflicts and uses keycloak.ourdomain//realms

lukas-beumer commented 1 year ago

Ahhhh, okay. Sorry, we are using an external keycloak instance. The way the error sounds, I would guess that it has something to do with the URL.

ChrisJBurns commented 1 year ago

It's weird, the issue I believe is because for some reason, the call from connectors to get a token is adding the /auth/ bit into the URL e.g. keycloak.ourdomain/auth/realms/...

The 404 is happening because our Keycloak is v19, so the /auth/ bit isn't valid. All it needs to do is remove the /auth/ bit and it's fine. But as @aabouzaid stated above, there must be a variable somewhere being set that tells the code to add the /auth/ part.

lukas-beumer commented 1 year ago

@ChrisJBurns Do you mean this parameter? https://github.com/camunda/camunda-platform-helm/blob/main/charts/camunda-platform/values.yaml#L1831

ChrisJBurns commented 1 year ago

Yep, since raising this ticket, I've set this value now. I will change the issue description with my updated YAML.

aabouzaid commented 1 year ago

OK, it's hard coded in the client lib :grinning:

https://github.com/camunda-community-hub/camunda-operate-client-java/blob/fa38603174de3d5f18414d25386ccc5444cad52d/src/main/java/io/camunda/operate/auth/SelfManagedAuthentication.java#L71

So any solution mentioned will not work till it's a var created for it.

aabouzaid commented 1 year ago

The fix has been merged and it will be part of the next release :heavy_check_mark:

camunda / camunda-platform-helm

[BUG] Connectors failing to start due to going to legacy Keycloak URL for Token #978