Closed ferrarimarco closed 6 years ago
@StefanoSalsano currently zookeeper and kafka run on the same VM. Should I keep them in the same container?
if the effort is not big it's good to separate
but if it introduces issues we may need to address them...
The effort is practically the same. If they run in the same conteiner, we need some kind of process supervisor, like Spotify is doing, otherwise we can have two separate containers, like https://github.com/wurstmeister/kafka-docker.
In the two containers scenario we need at least docker-compose to start the stack (but since we need it anyway it's not a big deal).
UPDATE: zookeeper and kafka have been ported to Docker
cc @StefanoSalsano
@devbrand I have a couple of issues with flume
.
When I start the docker container I don't see the ports you mentioned available and listening, even if the processes are running.
If you want to try you need Docker (17.09.0-ce), Docker Compose (1.17.1) and InSpec (1.48.0).
Then you can run (after checking out the feat/GH-12-dockerization
git branch): ./test/test-docker-images.sh --docker-context-path=docker
(this command will build, start and test all the containers).
You can look into the container by running: docker exec -it scissorproject-flume /bin/bash
InSpec tests on the flume
container will fail.
Relevant InSpec output:
× Port 10000 should be listening
expected `Port 10000.listening?` to return true, got false
× Port 10000 processes should include "java"
expected [] to include "java"
✔ Port 10000 protocols should not include "udp"
× Port 10000 protocols should include "tcp"
expected [] to include "tcp"
× Port 10000 addresses should include "0.0.0.0"
expected [] to include "0.0.0.0"
× Port 10000 protocols should include "tcp6"
expected [] to include "tcp6"
× Port 10000 addresses should include "::"
expected [] to include "::"
× Port 15222 should be listening
expected `Port 15222.listening?` to return true, got false
× Port 15222 processes should include "java"
expected [] to include "java"
✔ Port 15222 protocols should not include "udp"
× Port 15222 protocols should include "tcp"
expected [] to include "tcp"
× Port 15222 addresses should include "0.0.0.0"
expected [] to include "0.0.0.0"
× Port 15222 protocols should include "tcp6"
expected [] to include "tcp6"
× Port 15222 addresses should include "::"
expected [] to include "::"
× Port 20000 should be listening
expected `Port 20000.listening?` to return true, got false
× Port 20000 processes should include "java"
expected [] to include "java"
✔ Port 20000 protocols should not include "udp"
× Port 20000 protocols should include "tcp"
expected [] to include "tcp"
× Port 20000 addresses should include "0.0.0.0"
expected [] to include "0.0.0.0"
× Port 20000 protocols should include "tcp6"
expected [] to include "tcp6"
× Port 20000 addresses should include "::"
expected [] to include "::"
× Port 44444 should be listening
expected `Port 44444.listening?` to return true, got false
× Port 44444 processes should include "java"
expected [] to include "java"
✔ Port 44444 protocols should not include "udp"
× Port 44444 protocols should include "tcp"
expected [] to include "tcp"
× Port 44444 addresses should include "0.0.0.0"
expected [] to include "0.0.0.0"
× Port 44444 protocols should include "tcp6"
expected [] to include "tcp6"
× Port 44444 addresses should include "::"
expected [] to include "::"
ps aux
output:
root@flume:/opt# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4512 800 ? Ss 11:22 0:00 /bin/sh /opt/start-flume.sh
root 6 0.3 0.6 5442504 100728 ? Sl 11:22 0:01 /usr/lib/jvm/default-java/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.man
root 8 0.3 0.6 5442504 99872 ? Sl 11:22 0:01 /usr/lib/jvm/default-java/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5446 -Dcom.sun.man
root 10 0.3 0.6 5442504 98012 ? Sl 11:22 0:01 /usr/lib/jvm/default-java/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5447 -Dcom.sun.man
root 12 0.2 0.2 6732444 42220 ? Sl 11:22 0:01 java -cp /opt/zookeeper-agent/conf:/opt/zookeeper-agent/lib/* -Dscissor.log.dir=/opt/zookeeper-agent/log at.srfg.ccl.zookeeper.ZookeeperBas
root 13 0.0 0.0 4416 696 ? S 11:22 0:00 tail -F /var/log/flume-ng/ingest.out /var/log/flume-ng/filter.out /var/log/flume-ng/output.out /opt/zookeeper-agent/log/scissor-zookeeper.l
root 1302 0.6 0.0 18252 3260 pts/0 Ss 11:30 0:00 /bin/bash
root 1312 0.0 0.0 34428 2756 pts/0 R+ 11:30 0:00 ps aux
netstat -aplno
output:
root@flume:/opt# netstat -aplno
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 0 0 0.0.0.0:5445 0.0.0.0:* LISTEN 6/java off (0.00/0/0)
tcp 0 0 0.0.0.0:38757 0.0.0.0:* LISTEN 8/java off (0.00/0/0)
tcp 0 0 0.0.0.0:5446 0.0.0.0:* LISTEN 8/java off (0.00/0/0)
tcp 0 0 0.0.0.0:5447 0.0.0.0:* LISTEN 10/java off (0.00/0/0)
tcp 0 0 0.0.0.0:35463 0.0.0.0:* LISTEN 10/java off (0.00/0/0)
tcp 0 0 0.0.0.0:33611 0.0.0.0:* LISTEN 10/java off (0.00/0/0)
tcp 0 0 0.0.0.0:33301 0.0.0.0:* LISTEN 6/java off (0.00/0/0)
tcp 0 0 0.0.0.0:43423 0.0.0.0:* LISTEN 6/java off (0.00/0/0)
tcp 0 0 0.0.0.0:41155 0.0.0.0:* LISTEN 8/java off (0.00/0/0)
tcp 0 0 127.0.0.11:44419 0.0.0.0:* LISTEN - off (0.00/0/0)
tcp 0 0 172.19.0.4:37734 172.19.0.2:2181 ESTABLISHED 12/java off (0.00/0/0)
udp 0 0 127.0.0.11:54111 0.0.0.0:* - off (0.00/0/0)
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ] STREAM CONNECTED 372765 6/java
unix 2 [ ] STREAM CONNECTED 370037 12/java
unix 2 [ ] STREAM CONNECTED 370085 10/java
unix 2 [ ] STREAM CONNECTED 372768 8/java
Output files are empty:
root@flume:/opt# cat /var/log/flume-ng/
filter.out ingest.out output.out
root@flume:/opt# cat /var/log/flume-ng/filter.out
root@flume:/opt# cat /var/log/flume-ng/ingest.out
root@flume:/opt# cat /var/log/flume-ng/output.out
root@flume:/opt#
UPDATE: I think I found the issue. morphline-lib
directory is not in the classpath...
root@flume:/opt# cat /opt/log/filter.log
12 Dec 2017 11:22:51,520 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:62) - Configuration provider starting
12 Dec 2017 11:22:51,540 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:134) - Reloading configuration file:/opt/flume-config/conf/filter/filter.conf
12 Dec 2017 11:22:51,574 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,575 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,575 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,578 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,578 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,578 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,578 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:k1
12 Dec 2017 11:22:51,579 INFO [conf-file-poller-0] (org.apache.flumClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more2:51,630 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [filter]
12 Dec 2017 11:22:51,631 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:147) - Creating channels
12 Dec 2017 11:22:51,651 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:42) - Creating instance of channel c1 type memory
12 Dec 2017 11:22:51,669 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:201) - Created channel c1
12 Dec 2017 11:22:51,671 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:41) - Creating instance of source r1, type avro
12 Dec 2017 11:22:51,745 ERROR [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:146) - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/kitesdk/morphline/api/Command
at org.apache.flume.sink.solr.morphline.MorphlineInterceptor.<init>(MorphlineInterceptor.java:56)
at org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder.build(MorphlineInterceptor.java:113)
at org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder.build(MorphlineInterceptor.java:104)
at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:113)
at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:82)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:348)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:101)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.kitesdk.morphline.api.Command
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
Additional points for improvement have been moved to the description of the commit
TODO
./test/test-docker-images.sh --docker-context-path=docker
)Components to Dockerize