Closed priamai closed 2 years ago
Worth adding the beginning of the logs leading to that:
wait-for-it: waiting 60 seconds for kafka:9092
wait-for-it: kafka:9092 is available after 3 seconds
wait-for-it: waiting 60 seconds for elasticsearch:9200
wait-for-it: elasticsearch:9200 is available after 6 seconds
2022-04-25 09:21:08,210 INFO standalone io.kestra.cli.AbstractCommand Starting Kestra with environments [cli]
2022-04-25 09:21:08,863 INFO standalone io.kestra.cli.AbstractCommand Server Running: http://a6cffb1a0213:8080, Management server on port http://a6cffb1a0213:8081/health
2022-04-25 09:21:26,308 WARN standalone .r.e.AbstractElasticSearchRepository Replication incomplete, expected 2, got 1
LOTS OF THOSE REPEATED
2022-04-25 09:22:20,827 ERROR standalone-runner_1 .c.u.ThreadUncaughtExceptionHandlers Caught an exception in Thread[standalone-runner_1,5,main]. Shutting down.
org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:870)
at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:851)
at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:821)
at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:733)
at io.kestra.runner.kafka.services.KafkaStreamService$Stream.<init>(KafkaStreamService.java:122)
at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:110)
at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:69)
at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:61)
at io.kestra.runner.kafka.KafkaScheduler.init(KafkaScheduler.java:162)
at io.kestra.runner.kafka.KafkaScheduler.initStream(KafkaScheduler.java:170)
at io.kestra.runner.kafka.KafkaScheduler.run(KafkaScheduler.java:188)
at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:127)
at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:867)
... 14 common frames omitted
2022-04-25 09:22:22,846 ERROR standalone-runner_0 .c.u.ThreadUncaughtExceptionHandlers Caught an exception in Thread[standalone-runner_0,5,main]. Shutting down.
Can you give your environnement for running the compose please? (macos, linux, ...)
Seems that you don't write access on the host machine on /tmp/kestra-wd
.
Maybe rm -rf /tmp/kestra-wd
?
Ah good point, I totally forgot that it is a mounted folder. The /tmp/kestra-wd has root/root permissions which is odd considering I am running docker as non-sudo.
Info: Ubuntu 20.04 x64 bit User: local user with docker group Docker: run as local user without sudo
(same setup described here: https://docs.docker.com/engine/install/linux-postinstall/)
Could I mount it instead as a docker volume to avoid the permission issue or maybe is there a parameter to change the kestra user to non root?
We have unroot https://github.com/kestra-io/kestra/commit/ed05baae0b34e2e0c909c939ba4c051c82974d07 the docker image in order to be deployed on more environment. If you have started the docker the permission will be wrong. It's just a temporary directory, we can safely remove it and it will work
Which image tag should I use with the unroot version? This one: kestra/kestra:develop-full ?
Yes, this one is unroot
Okay I pulled that image from scratch but somehow the folder gets created again as root/root. I am beginning to wonder whether it is instead a docker issue?
Yes that was the case, by manually creating those folders via my local user (before starting the docker up), solved the problem!
You have reason ... I'm hitting this very old issue https://github.com/moby/moby/issues/2259 and there is no true workaround. So I decided to unswitch the docker-compose example to root to avoid the issue. Thanks for the reprt :+1:
Issue description
Hi there, I am running the standard docker-compose provided in the tutorial, but Kestra doesn't start. I am getting this error:
It seems to be related to Kafka but besides that I have no idea how to fix it.