search

kestra-io / kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
https://kestra.io
Apache License 2.0
7.57k stars 461 forks source link

running docker compose errors with Kafka #575

Closed priamai closed 2 years ago

priamai commented 2 years ago

Issue description

Hi there, I am running the standard docker-compose provided in the tutorial, but Kestra doesn't start. I am getting this error:

org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:870)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:851)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:821)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:733)
        at io.kestra.runner.kafka.services.KafkaStreamService$Stream.<init>(KafkaStreamService.java:122)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:110)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:69)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:61)
        at io.kestra.runner.kafka.KafkaScheduler.init(KafkaScheduler.java:162)
        at io.kestra.runner.kafka.KafkaScheduler.initStream(KafkaScheduler.java:170)
        at io.kestra.runner.kafka.KafkaScheduler.run(KafkaScheduler.java:188)
        at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

It seems to be related to Kafka but besides that I have no idea how to fix it.

priamai commented 2 years ago

Worth adding the beginning of the logs leading to that:

wait-for-it: waiting 60 seconds for kafka:9092
wait-for-it: kafka:9092 is available after 3 seconds
wait-for-it: waiting 60 seconds for elasticsearch:9200
wait-for-it: elasticsearch:9200 is available after 6 seconds
2022-04-25 09:21:08,210 INFO  standalone   io.kestra.cli.AbstractCommand Starting Kestra with environments [cli]
2022-04-25 09:21:08,863 INFO  standalone   io.kestra.cli.AbstractCommand Server Running: http://a6cffb1a0213:8080, Management server on port http://a6cffb1a0213:8081/health
2022-04-25 09:21:26,308 WARN  standalone   .r.e.AbstractElasticSearchRepository Replication incomplete, expected 2, got 1
LOTS OF THOSE REPEATED
2022-04-25 09:22:20,827 ERROR standalone-runner_1 .c.u.ThreadUncaughtExceptionHandlers Caught an exception in Thread[standalone-runner_1,5,main]. Shutting down.
org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:870)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:851)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:821)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:733)
        at io.kestra.runner.kafka.services.KafkaStreamService$Stream.<init>(KafkaStreamService.java:122)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:110)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:69)
        at io.kestra.runner.kafka.services.KafkaStreamService.of(KafkaStreamService.java:61)
        at io.kestra.runner.kafka.KafkaScheduler.init(KafkaScheduler.java:162)
        at io.kestra.runner.kafka.KafkaScheduler.initStream(KafkaScheduler.java:170)
        at io.kestra.runner.kafka.KafkaScheduler.run(KafkaScheduler.java:188)
        at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: base state directory [/tmp/kestra-wd/kafka-streams/] doesn't exist and couldn't be created
        at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:127)
        at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:867)
        ... 14 common frames omitted
2022-04-25 09:22:22,846 ERROR standalone-runner_0 .c.u.ThreadUncaughtExceptionHandlers Caught an exception in Thread[standalone-runner_0,5,main]. Shutting down.
tchiotludo commented 2 years ago

Can you give your environnement for running the compose please? (macos, linux, ...)

Seems that you don't write access on the host machine on /tmp/kestra-wd. Maybe rm -rf /tmp/kestra-wd?

priamai commented 2 years ago

Ah good point, I totally forgot that it is a mounted folder. The /tmp/kestra-wd has root/root permissions which is odd considering I am running docker as non-sudo.

priamai commented 2 years ago

Info: Ubuntu 20.04 x64 bit User: local user with docker group Docker: run as local user without sudo

(same setup described here: https://docs.docker.com/engine/install/linux-postinstall/)

Could I mount it instead as a docker volume to avoid the permission issue or maybe is there a parameter to change the kestra user to non root?

tchiotludo commented 2 years ago

We have unroot https://github.com/kestra-io/kestra/commit/ed05baae0b34e2e0c909c939ba4c051c82974d07 the docker image in order to be deployed on more environment. If you have started the docker the permission will be wrong. It's just a temporary directory, we can safely remove it and it will work

priamai commented 2 years ago

Which image tag should I use with the unroot version? This one: kestra/kestra:develop-full ?

tchiotludo commented 2 years ago

Yes, this one is unroot

priamai commented 2 years ago

Okay I pulled that image from scratch but somehow the folder gets created again as root/root. I am beginning to wonder whether it is instead a docker issue?

priamai commented 2 years ago

Yes that was the case, by manually creating those folders via my local user (before starting the docker up), solved the problem!

tchiotludo commented 2 years ago

You have reason ... I'm hitting this very old issue https://github.com/moby/moby/issues/2259 and there is no true workaround. So I decided to unswitch the docker-compose example to root to avoid the issue. Thanks for the reprt :+1: