camunda / camunda-platform-helm

Camunda Platform 8 Self-Managed Helm charts
https://docs.camunda.io/docs/self-managed/overview/
Apache License 2.0
74 stars 138 forks source link

[ISSUE] Identity crashes if fullURL is not provided since 9.3 #1442

Closed ManuelDittmar closed 8 months ago

ManuelDittmar commented 8 months ago

Describe the issue:

with the below values.yaml, identity fails to start. It only works if the fullURL is provided. This was not the case with 9.2.

global:
  identity:
    auth:

identity:
  enabled: true

optimize:
  enabled: true

zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 10Gi

zeebe-gateway:
  replicas: 1

connectors:
  enabled: true
  inbound:
    mode: disabled

elasticsearch:
  master:
    replicaCount: 1
    persistence:
      size: 15Gi

webModeler:
  enabled: true
  image:
    pullSecrets:
      - name: docker-registry
  restapi:
    mail:
      smtpHost: smtp.example.com
      smtpPort: 587
      smtpUser: user
      smtpPassword: secret
      fromAddress: no-reply@example.com
postgresql:
  enabled: true

Actual behavior:

Back-off restarting failed container identity in pod camunda-identity-846d96dfd-d8ffk_default(8bd8a2a7-60e7-4b6c-81bb-7bf72d86ffb6)

camunda-identity-846d96dfd-d8ffk 0/1 CrashLoopBackOff 3 (20s ago) 92s

Expected behavior:

Identity can be started successfully with default values.

How to reproduce:

  1. helm install camunda camunda/camunda-platform --version 9.2 -f values.yaml
  2. observe that identity is started successfully.
  3. helm install camunda camunda/camunda-platform --version 9.3 -f values.yaml
  4. observe that identity fails starting.

Logs:

Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )volume that contains injected data from multiple sources)
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/a.crt
 :: Spring Boot ::                (v3.1.9)
    DownwardAPI:             true
2024-03-13 09:59:06.103  INFO 1 --- [           main] i.c.i.Application                        : Starting Application using Java 17.0.10 with PID 1 (/app/identity.jar started by ? in /app)
2024-03-13 09:59:06.119  INFO 1 --- [           main] i.c.i.Application                        : The following 1 profile is active: "keycloak"
Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts
2024-03-13 09:59:09.450  WARN 1 --- [           main] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: io.camunda.identity.security.spring.filter.FilterExceptionHandler
2024-03-13 09:59:09.754  WARN 1 --- [           main] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: io.camunda.identity.config.IdentityCommon
2024-03-13 09:59:09.880  WARN 1 --- [           main] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: io.camunda.identity.impl.keycloak.config.record.KeycloakClient.5" already present on machine
2024-03-13 09:59:10.022  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'frontendController': Invocation of init method failed
2024-03-13 09:59:10.093 ERROR 1 --- [           main] o.s.b.SpringApplication                  : Application run faileda-identity-846d96dfd-d8ffk_default(8bd8a2a7-60e7-4b6c-81bb-7bf72d86ffb6)
 manu   donat-debug                                                                                                                                                                    in bash at 10:54:2org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'frontendController': Invocation of init method failed
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:222) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1760) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:596) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:518) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:949) ~[spring-context-6.0.17.jar!/:6.0.17]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:615) ~[spring-context-6.0.17.jar!/:6.0.17]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.1.9.jar!/:3.1.9]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:738) [spring-boot-3.1.9.jar!/:3.1.9]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:440) [spring-boot-3.1.9.jar!/:3.1.9]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) [spring-boot-3.1.9.jar!/:3.1.9]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) [spring-boot-3.1.9.jar!/:3.1.9]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) [spring-boot-3.1.9.jar!/:3.1.9]
        at io.camunda.identity.Application.main(Application.java:21) [classes!/:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [identity.jar:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) [identity.jar:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [identity.jar:?]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) [identity.jar:?]
Caused by: java.lang.IllegalArgumentException: Expected authority at index 7: http://
        at java.net.URI.create(Unknown Source) ~[?:?]
        at io.camunda.identity.config.IdentityCommon.basePath(IdentityCommon.java:62) ~[classes!/:?]
        at io.camunda.identity.frontend.controller.FrontendController.initialize(FrontendController.java:53) ~[classes!/:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.0.17.jar!/:6.0.17]
        ... 26 more
Caused by: java.net.URISyntaxException: Expected authority at index 7: http://
        at java.net.URI$Parser.fail(Unknown Source) ~[?:?]
        at java.net.URI$Parser.failExpecting(Unknown Source) ~[?:?]
        at java.net.URI$Parser.parseHierarchical(Unknown Source) ~[?:?]
        at java.net.URI$Parser.parse(Unknown Source) ~[?:?]
        at java.net.URI.<init>(Unknown Source) ~[?:?]
        at java.net.URI.create(Unknown Source) ~[?:?]
        at io.camunda.identity.config.IdentityCommon.basePath(IdentityCommon.java:62) ~[classes!/:?]
        at io.camunda.identity.frontend.controller.FrontendController.initialize(FrontendController.java:53) ~[classes!/:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.0.17.jar!/:6.0.17]
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.0.17.jar!/:6.0.17]
        ... 26 more

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

ManuelDittmar commented 8 months ago

works if fullURL is provided.

global:
  identity:
    auth:
      # Disable the Identity authentication for local development
      # it will fall back to basic-auth: demo/demo as default user
      enabled: true

# Disable identity as part of the Camunda core
identity:
  enabled: true
  fullURL: "http://identity:8080"

optimize:
  enabled: true

# Reduce for Zeebe and Gateway the configured replicas and with that the required resources
# to get it running locally
zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 10Gi

zeebe-gateway:
  replicas: 1

connectors:
  enabled: true
  inbound:
    mode: disabled

elasticsearch:
  master:
    replicaCount: 1
    # Request smaller persistent volumes.
    persistence:
      size: 15Gi

webModeler:
  enabled: true
  image:
    pullSecrets:
      # replace with the name of the previously created secret
      - name: docker-registry
  restapi:
    mail:
      smtpHost: smtp.example.com
      smtpPort: 587
      smtpUser: user
      smtpPassword: secret
      # email address to be displayed as sender of emails from Web Modeler
      fromAddress: no-reply@example.com
postgresql:
  enabled: true
felix-mueller commented 8 months ago

Saw the bug too in default config. Using fullurl in identity fixed it for me.

aabouzaid commented 8 months ago

Thanks for reporting that issue :+1:

@Ben-Sheppard if that's expected behavior from Identity, I will add a constrain in Helm to ensure that the user supply the fullURL value.

Ben-Sheppard commented 8 months ago

I have two thoughts here:

  1. This feels like a regression to me and potentially something that I've introduced when adding in the OIDC support, I would need to confirm this though
  2. I feel like there should be a fallback i.e. we can likely deduce the Identity URL from the information we have
Ben-Sheppard commented 8 months ago

Hmm seems like its this addition https://github.com/camunda/camunda-platform-helm/pull/1377/files#diff-be698369f3ab3270130fb11441ad7f467541c5e25d0ad445805fa0aa0c4251cfR32 - I can see why we added it (there was a requirement on our side to effectively have the internal Identity URL and the external URL but the helper method should rely on fallbacks if .Values.fullUrl is not present.

Look again at the error messages it seems that the catch block if identity.fullUrl is not set is causing the issue as it can default the $proto but not find the host etc? Could you confirm that @aabouzaid ?

aabouzaid commented 8 months ago

@Ben-Sheppard thanks for the info :+1:

The issue was coming from extra characters (newline and spaces) in the IDENTITY_URL + missing default value when no fullURL or Ingress is there.

It's been fixed in https://github.com/camunda/camunda-platform-helm/pull/1480 and https://github.com/camunda/camunda-platform-helm/pull/1481

Also with the new logic we have, the identity.fullURL is not mandatory anymore so I've updated the docs and removed it from the values of the tests.

renzpatriarca commented 5 months ago

@aabouzaid Has this been backported to HC 9.3.x? A customer using HC 9.3.3 encountered this issue and applied the workaround of adding identity.fullURL.

phani9613 commented 5 months ago

Hi @aabouzaid, please let us know if this fix has been backported to HC 9.3.X. we are using HC 9.3.3 as Renz mentioned. and we are awaiting your reply. Thanks

hamza-m-masood commented 5 months ago

From what I can see, this issue has not been backported to 9.3.x This fix was introduced at release 10.0.0

And the fix for the related issue was introduced at release 10.0.3

We will look to backport this fix