credentials cache creates malformed yaml

stephanpelikan commented 8 months ago

Describe the bug

The yaml-file created of the credentials cache ~/.camunda/credentials is malformed. A quote character is added in the last line:

---
zeebe-api:
  auth:
    credentials:
      accessToken: "eyJhbGc...9HiYHtwA"
      tokenType: "Bearer"
      expiry: "2024-01-23T08:31:40.1238419+01:00"
"

As there is nothing mentioned in the README.md about spring-zeebe is writing files, it took as a while to find the malformed file. However, removing the bad character works, but it comes back once the cache file is written again.

To Reproduce

We don't know exactly. Some team members suffer from this, others not. Maybe the configuration noted at the bottom helps you to get insights.

Expected behavior

Files written, especially consisting of credentials, have to be documented
No malformed files are written

Log/Stacktrace

Full Stacktrace

``` org.springframework.context.ApplicationContextException: Failed to start bean 'zeebeLifecycleEventProducer' at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:357) ~[spring-context-6.0.12.jar:6.0.12] at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na] at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:156) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:124) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:958) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:611) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.1.4.jar:3.1.4] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737) ~[spring-boot-3.1.4.jar:3.1.4] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) ~[spring-boot-3.1.4.jar:3.1.4] at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-3.1.4.jar:3.1.4] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309) ~[spring-boot-3.1.4.jar:3.1.4] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298) ~[spring-boot-3.1.4.jar:3.1.4] at com.myapp.launcher.api.ApiApplication.main(ApiApplication.java:34) ~[classes/:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na] at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:50) ~[spring-boot-devtools-3.1.4.jar:3.1.4] Caused by: io.camunda.zeebe.client.api.command.ClientStatusException: null at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.3.5.jar:8.3.5] at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.3.5.jar:8.3.5] at io.vanillabp.camunda8.deployment.Camunda8DeploymentAdapter.lambda$doDeployment$4(Camunda8DeploymentAdapter.java:168) ~[camunda8-spring-boot-adapter-1.2.0.jar:na] at java.base/java.util.Optional.map(Optional.java:260) ~[na:na] at io.vanillabp.camunda8.deployment.Camunda8DeploymentAdapter.doDeployment(Camunda8DeploymentAdapter.java:166) ~[camunda8-spring-boot-adapter-1.2.0.jar:na] at io.vanillabp.springboot.adapter.ModuleAwareBpmnDeployment.deployWorkflowModule(ModuleAwareBpmnDeployment.java:92) ~[spring-boot-support-1.0.7.jar:na] at io.vanillabp.springboot.adapter.ModuleAwareBpmnDeployment.deployWorkflowModule(ModuleAwareBpmnDeployment.java:55) ~[spring-boot-support-1.0.7.jar:na] at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[na:na] at io.vanillabp.springboot.adapter.ModuleAwareBpmnDeployment.deployAllWorkflowModules(ModuleAwareBpmnDeployment.java:43) ~[spring-boot-support-1.0.7.jar:na] at io.vanillabp.camunda8.deployment.Camunda8DeploymentAdapter.zeebeClientCreated(Camunda8DeploymentAdapter.java:87) ~[camunda8-spring-boot-adapter-1.2.0.jar:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-6.0.12.jar:6.0.12] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.0.12.jar:6.0.12] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.0.12.jar:6.0.12] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) ~[spring-aop-6.0.12.jar:6.0.12] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-6.0.12.jar:6.0.12] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391) ~[spring-tx-6.0.12.jar:6.0.12] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-6.0.12.jar:6.0.12] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.12.jar:6.0.12] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751) ~[spring-aop-6.0.12.jar:6.0.12] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) ~[spring-aop-6.0.12.jar:6.0.12] at io.vanillabp.camunda8.deployment.Camunda8DeploymentAdapter$$SpringCGLIB$$0.zeebeClientCreated() ~[camunda8-spring-boot-adapter-1.2.0.jar:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na] at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:174) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:437) ~[spring-context-6.0.12.jar:6.0.12] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.12.jar:6.0.12] at io.camunda.zeebe.spring.client.event.ZeebeLifecycleEventProducer.start(ZeebeLifecycleEventProducer.java:23) ~[spring-boot-starter-camunda-8.3.4.8.jar:8.3.4.8] at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:179) ~[spring-context-6.0.12.jar:6.0.12] ... 18 common frames omitted Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: CANCELLED at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[na:na] at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[na:na] at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.5.jar:8.3.5] ... 55 common frames omitted Caused by: io.grpc.StatusRuntimeException: CANCELLED at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.60.0.jar:1.60.0] at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.60.0.jar:1.60.0] at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.60.0.jar:1.60.0] at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.60.0.jar:1.60.0] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na] Caused by: com.fasterxml.jackson.databind.JsonMappingException: while scanning a quoted scalar in 'reader', line 8, column 1: " ^ found unexpected end of stream in 'reader', line 8, column 2: " ^ at [Source: (File); line: 7, column: 50] (through reference chain: java.util.LinkedHashMap["zeebe-api"]->io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache$OAuthCachedCredentials["auth"]) at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1853) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:572) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1409) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:623) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:449) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3657) ~[jackson-databind-2.15.2.jar:2.15.2] at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.readCache(OAuthCredentialsCache.java:61) ~[zeebe-client-java-8.3.5.jar:8.3.5] at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.computeIfMissingOrInvalid(OAuthCredentialsCache.java:87) ~[zeebe-client-java-8.3.5.jar:8.3.5] at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.applyCredentials(OAuthCredentialsProvider.java:84) ~[zeebe-client-java-8.3.5.jar:8.3.5] at io.camunda.zeebe.client.impl.ZeebeCallCredentials.lambda$applyRequestMetadata$0(ZeebeCallCredentials.java:48) ~[zeebe-client-java-8.3.5.jar:8.3.5] ... 3 common frames omitted Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning a quoted scalar in 'reader', line 8, column 1: " ^ found unexpected end of stream in 'reader', line 8, column 2: " ^ at [Source: (File); line: 7, column: 50] at com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException.from(MarkedYAMLException.java:28) ~[jackson-dataformat-yaml-2.15.2.jar:2.15.2] at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:433) ~[jackson-dataformat-yaml-2.15.2.jar:2.15.2] at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:1091) ~[jackson-core-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:321) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:623) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:449) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545) ~[jackson-databind-2.15.2.jar:2.15.2] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570) ~[jackson-databind-2.15.2.jar:2.15.2] ... 17 common frames omitted Caused by: org.yaml.snakeyaml.scanner.ScannerException: while scanning a quoted scalar in 'reader', line 8, column 1: " ^ found unexpected end of stream in 'reader', line 8, column 2: " ^ at org.yaml.snakeyaml.scanner.ScannerImpl.scanFlowScalarSpaces(ScannerImpl.java:1991) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.scanFlowScalar(ScannerImpl.java:1904) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.fetchFlowScalar(ScannerImpl.java:1082) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.fetchDouble(ScannerImpl.java:1064) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:452) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:662) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185) ~[snakeyaml-1.33.jar:na] at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:195) ~[snakeyaml-1.33.jar:na] at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429) ~[jackson-dataformat-yaml-2.15.2.jar:2.15.2] ... 25 common frames omitted ```

Environment:

OS: Windows
Deployment: Self Managed
Spring-Zeebe Version: 8.3.4.8
SpringBoot Version: 3.1.4
Configuration:

zeebe.client.enabled=false
zeebe.client.broker.gateway-address=d-camunda-zeebe.ads.lan
zeebe.client.security.plaintext=false
zeebe.client.security.cert-path=N:\\dev\\cert\\ADSRCA.crt
zeebe.client.connection-mode=ADDRESS
zeebe.client.id=zeebe
zeebe.client.secret=xxx
zeebe.authorization.server.url=https://d-camunda-keycloak.ads.lan/
zeebe.token.audience=zeebe-api
zeebe.client.broker.gatewayAddress=d-camunda-zeebe.ads.lan

zeebe.client.default-tenant-id=myapp

common.enabled=true
spring.security.oauth2.resourceserver.jwt.audiences=zeebe-api
common.client-id=zeebe
common.client-secret=xxx
common.keycloak.url=https://d-camunda-keycloak.ads.lan/
common.keycloak.realm=camunda-platform

jonathanlukas commented 7 months ago

Did you use zbctl on the same machine?

stephanpelikan commented 7 months ago

No. Only the spring-zeebe client.

shaarmann commented 7 months ago

I had some 'fun' looking further into this issue. I noticed the following:

Currently, the ZeebeClientConfiguration creates a new CredentialsProvider whenever getCredentialsProvider is called. In a self-managed setup using OAuth, the OAuthCredentialsProvider and the OAuthCredentialsCache is instantiated multiple times:

1x for instantiating the GatewayStub
2x for instantiating the ZeebeClient

While annotate with @ThreadSafe, the OAuthCredentialsCache uses synchronized methods and is only thread safe if there is only one respective object. Hence, the error reported in this issue may be caused by occasional concurrent writes to the credentials cache. However, I was not able to reproduce the error in my setup.

camunda-community-hub / spring-zeebe

credentials cache creates malformed yaml #609