Open vkorenev opened 3 years ago
Hi @vkorenev , reading your report puzzles me a little. Can you please check and confirm the Cloudflow versions of the components you are using:
kubectl describe <cloudflow-operator-pod>
and check the image tag)kubectl cloudflow version
)Cloudflow Sbt Plugin used to build the application ( kubectl get cloudflowapplications.cloudflow.lightbend.com -n <your namespace> -o yaml
and look for library_version
)
thanks in advance!
Hi @andreaTP, Cloudflow operator and sbt plugin are 2.0.21. However, the CLI version is 2.0.23. Should I downgrade the CLI?
I would suggest you to align and upgrade the versions to latest ( 2.0.25 ) and let us know if the problem persists
Describe the bug
Sometimes a streamlet that was working fine fails after the pod restarts. Further investigation discovered that all the configuration properties disappear from the secret that contains the config.
Example: A streamlet was working fine for some time. Then, after a pod was restarted by Kubernetes, it failed to start with the following exception:
Listing the configuration for this application shows that this property is present:
However, all the configuration properties disappeared from the secret which is mounted to
/etc/cloudflow-runner-secret
:When decoded, the properties are these:
But when this application is freshly deployed, the values are present there:
So at some point all the config values except
cloudflow.kafka.bootstrap-servers
are wiped out.To Reproduce
I have not found exact steps to reproduce this yet. But this happens periodically every 1-2 weeks.
Expected behavior
Streamlets should continue to run indefinitely with the last applied configuration.
Additional context
The deployed Cloudflow version is 2.0.21.