broker, zookeeper, schema-registry, rest-proxy, connect-distributed fail periodicaly

[leopoldstrauch]

Every time i start the docker container with docker run --rm --net=host landoop/fast-data-dev some components crash periodicaly Any ideas?

sudo docker run --rm --net=host landoop/fast-data-dev
This is landoop’s fast-data-dev. Kafka 0.11.0.1, Confluent OSS 3.3.1.
You may visit http://localhost:3030 in about a minute.
2018-01-09 16:21:14,301 CRIT Supervisor running as root (no user in config file)
2018-01-09 16:21:14,301 INFO Included extra file "/etc/supervisord.d/01-fast-data.conf" during parsing
2018-01-09 16:21:14,301 INFO Included extra file "/etc/supervisord.d/99-supervisord-sample-data.conf" during parsing
2018-01-09 16:21:14,303 INFO supervisord started with pid 7
2018-01-09 16:21:15,306 INFO spawned: 'sample-data' with pid 93
2018-01-09 16:21:15,310 INFO spawned: 'zookeeper' with pid 94
2018-01-09 16:21:15,315 INFO spawned: 'caddy' with pid 95
2018-01-09 16:21:15,317 INFO spawned: 'broker' with pid 96
2018-01-09 16:21:15,319 INFO spawned: 'smoke-tests' with pid 98
2018-01-09 16:21:15,322 INFO spawned: 'connect-distributed' with pid 99
2018-01-09 16:21:15,325 INFO spawned: 'logs-to-kafka' with pid 101
2018-01-09 16:21:15,327 INFO spawned: 'schema-registry' with pid 106
2018-01-09 16:21:15,328 INFO spawned: 'rest-proxy' with pid 108
2018-01-09 16:21:16,430 INFO success: sample-data entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,430 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,430 INFO success: caddy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,430 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,431 INFO success: smoke-tests entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,431 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,431 INFO success: logs-to-kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,431 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:16,431 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:21,549 INFO exited: zookeeper (exit status 1; not expected)
2018-01-09 16:21:22,553 INFO spawned: 'zookeeper' with pid 172
2018-01-09 16:21:23,556 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:26,554 INFO exited: broker (exit status 1; not expected)
2018-01-09 16:21:27,557 INFO spawned: 'broker' with pid 223
2018-01-09 16:21:28,833 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:28,833 INFO exited: zookeeper (exit status 1; not expected)
2018-01-09 16:21:29,837 INFO spawned: 'zookeeper' with pid 225
2018-01-09 16:21:30,642 INFO exited: schema-registry (exit status 1; not expected)
2018-01-09 16:21:31,643 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:31,645 INFO spawned: 'schema-registry' with pid 305
2018-01-09 16:21:32,647 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:35,426 INFO exited: zookeeper (exit status 1; not expected)
2018-01-09 16:21:35,429 INFO spawned: 'zookeeper' with pid 334
2018-01-09 16:21:36,168 INFO exited: rest-proxy (exit status 1; not expected)
2018-01-09 16:21:36,245 INFO spawned: 'rest-proxy' with pid 362
2018-01-09 16:21:36,258 INFO exited: connect-distributed (exit status 1; not expected)
2018-01-09 16:21:37,260 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:37,263 INFO spawned: 'connect-distributed' with pid 364
2018-01-09 16:21:37,264 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:38,795 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:38,795 INFO exited: broker (exit status 1; not expected)
2018-01-09 16:21:39,798 INFO spawned: 'broker' with pid 366
2018-01-09 16:21:41,654 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-09 16:21:41,654 INFO exited: zookeeper (exit status 1; not expected)
2018-01-09 16:21:42,658 INFO spawned: 'zookeeper' with pid 391
2018-01-09 16:21:43,661 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
^C2018-01-09 16:21:45,055 WARN received SIGINT indicating exit request
2018-01-09 16:21:45,055 INFO waiting for sample-data, zookeeper, caddy, broker, smoke-tests, connect-distributed, logs-to-kafka, schema-registry, rest-proxy to die
2018-01-09 16:21:45,424 INFO stopped: rest-proxy (terminated by SIGTERM)
2018-01-09 16:21:45,746 INFO stopped: schema-registry (exit status 143)
2018-01-09 16:21:45,746 INFO stopped: logs-to-kafka (terminated by SIGTERM)
2018-01-09 16:21:45,747 INFO stopped: connect-distributed (terminated by SIGTERM)
2018-01-09 16:21:46,749 INFO stopped: smoke-tests (terminated by SIGTERM)
2018-01-09 16:21:47,071 INFO stopped: broker (exit status 143)
2018-01-09 16:21:47,072 INFO stopped: caddy (exit status 0)
2018-01-09 16:21:47,394 INFO stopped: zookeeper (exit status 143)
2018-01-09 16:21:47,394 INFO stopped: sample-data (terminated by SIGTERM)

[Antwnis]

I think this issue typically occurs when the docker instance does not have enough memory allocated to it. i.e. see image at http://lenses.stream/developers-guide/dev-environment/index.html#faq and try to allocate i.e. 4GB to your Kafka, ZK, Sh. Reg, Connect and other services

[leopoldstrauch]

I managed to started it by setting ADV_HOST and Ports manualy:

docker run --rm -it \
> -p 2181:2181 -p 3030:3030 -p 8081:8081 \
> -p 8082:8082 -p 8083:8083 -p 9092:9092 \
> -e ADV_HOST=127.0.0.1 \
> landoop/fast-data-dev

but i will check your link too thanks :)

[dustinfarris]

I was getting this too when trying to start the image in a docker-compose.yml file, but my colleagues were not. The only difference we could come up with is that I am macOS High Sierra, and they were not.

At any rate, I was able to solve by adding to my docker-compose for kafka:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

[maxencehenneron]

@leopoldstrauch 's suggestion fixed the issue on my side (docker for macos high sierra), I wonder why this is happening

[bclipp]

I also had the same issue, but when statistically setting the port (I'm using a mac) no issues.

[benjlai]

I had the same issue on my Mac after a bit of investigation it appears that "--net=host" which should use the hosts network directly doesn't work. But on closer reading "The host networking driver only works on Linux hosts". Because Mac Docker runs inside a virtual machine so when it binds to the "host" network adapter its binding to the virtual machine and not to your Mac's network adapter.
Full discussion here: https://forums.docker.com/t/should-docker-run-net-host-work/14215