ferrarimarco commented 6 years ago

TODO

[x] Run https://github.com/hadolint/hadolint during CI builds
[x] Run https://github.com/RedCoolBeans/dockerlint during CI builds
[x] List the components to dockerize
[x] Configure InSpec (see #13)
[x] Provide a docker-compose.yml to deploy as a service
[x] Set container hostnames
[x] Implement InSpec tests for containers
[x] Implement Dockerfiles for components to dockerize
[x] Update README (dependencies, how to run tests ./test/test-docker-images.sh --docker-context-path=docker)
[x] Add Docker to the list of deps

Components to Dockerize

[x] d-streamon-master
[x] d-streamon-slave
[x] datasource24
[x] event-correlator
[x] flume
[x] kafka-idmef-converter
[x] kafka-prelude-connector
[x] kafka
[x] zookeeper (it was on the kafka VM)
[x] logstash
[x] logstash24
[x] prelude-manager-oss
[x] prewikka-oss
[x] semantics

ferrarimarco commented 6 years ago

@StefanoSalsano currently zookeeper and kafka run on the same VM. Should I keep them in the same container?

StefanoSalsano commented 6 years ago

if the effort is not big it's good to separate

StefanoSalsano commented 6 years ago

but if it introduces issues we may need to address them...

ferrarimarco commented 6 years ago

The effort is practically the same. If they run in the same conteiner, we need some kind of process supervisor, like Spotify is doing, otherwise we can have two separate containers, like https://github.com/wurstmeister/kafka-docker.

In the two containers scenario we need at least docker-compose to start the stack (but since we need it anyway it's not a big deal).

ferrarimarco commented 6 years ago

UPDATE: zookeeper and kafka have been ported to Docker

cc @StefanoSalsano

ferrarimarco commented 6 years ago

@devbrand I have a couple of issues with flume.

When I start the docker container I don't see the ports you mentioned available and listening, even if the processes are running.

If you want to try you need Docker (17.09.0-ce), Docker Compose (1.17.1) and InSpec (1.48.0).

Then you can run (after checking out the feat/GH-12-dockerization git branch): ./test/test-docker-images.sh --docker-context-path=docker (this command will build, start and test all the containers).

You can look into the container by running: docker exec -it scissorproject-flume /bin/bash

InSpec tests on the flume container will fail.

Relevant InSpec output:

    ×  Port 10000 should be listening
     expected `Port 10000.listening?` to return true, got false
     ×  Port 10000 processes should include "java"
     expected [] to include "java"
     ✔  Port 10000 protocols should not include "udp"
     ×  Port 10000 protocols should include "tcp"
     expected [] to include "tcp"
     ×  Port 10000 addresses should include "0.0.0.0"
     expected [] to include "0.0.0.0"
     ×  Port 10000 protocols should include "tcp6"
     expected [] to include "tcp6"
     ×  Port 10000 addresses should include "::"
     expected [] to include "::"
     ×  Port 15222 should be listening
     expected `Port 15222.listening?` to return true, got false
     ×  Port 15222 processes should include "java"
     expected [] to include "java"
     ✔  Port 15222 protocols should not include "udp"
     ×  Port 15222 protocols should include "tcp"
     expected [] to include "tcp"
     ×  Port 15222 addresses should include "0.0.0.0"
     expected [] to include "0.0.0.0"
     ×  Port 15222 protocols should include "tcp6"
     expected [] to include "tcp6"
     ×  Port 15222 addresses should include "::"
     expected [] to include "::"
     ×  Port 20000 should be listening
     expected `Port 20000.listening?` to return true, got false
     ×  Port 20000 processes should include "java"
     expected [] to include "java"
     ✔  Port 20000 protocols should not include "udp"
     ×  Port 20000 protocols should include "tcp"
     expected [] to include "tcp"
     ×  Port 20000 addresses should include "0.0.0.0"
     expected [] to include "0.0.0.0"
     ×  Port 20000 protocols should include "tcp6"
     expected [] to include "tcp6"
     ×  Port 20000 addresses should include "::"
     expected [] to include "::"
     ×  Port 44444 should be listening
     expected `Port 44444.listening?` to return true, got false
     ×  Port 44444 processes should include "java"
     expected [] to include "java"
     ✔  Port 44444 protocols should not include "udp"
     ×  Port 44444 protocols should include "tcp"
     expected [] to include "tcp"
     ×  Port 44444 addresses should include "0.0.0.0"
     expected [] to include "0.0.0.0"
     ×  Port 44444 protocols should include "tcp6"
     expected [] to include "tcp6"
     ×  Port 44444 addresses should include "::"
     expected [] to include "::"

ps aux output:

root@flume:/opt# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4512   800 ?        Ss   11:22   0:00 /bin/sh /opt/start-flume.sh
root         6  0.3  0.6 5442504 100728 ?      Sl   11:22   0:01 /usr/lib/jvm/default-java/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.man
root         8  0.3  0.6 5442504 99872 ?       Sl   11:22   0:01 /usr/lib/jvm/default-java/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5446 -Dcom.sun.man
root        10  0.3  0.6 5442504 98012 ?       Sl   11:22   0:01 /usr/lib/jvm/default-java/bin/java -Xms100m -Xmx2000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5447 -Dcom.sun.man
root        12  0.2  0.2 6732444 42220 ?       Sl   11:22   0:01 java -cp /opt/zookeeper-agent/conf:/opt/zookeeper-agent/lib/* -Dscissor.log.dir=/opt/zookeeper-agent/log at.srfg.ccl.zookeeper.ZookeeperBas
root        13  0.0  0.0   4416   696 ?        S    11:22   0:00 tail -F /var/log/flume-ng/ingest.out /var/log/flume-ng/filter.out /var/log/flume-ng/output.out /opt/zookeeper-agent/log/scissor-zookeeper.l
root      1302  0.6  0.0  18252  3260 pts/0    Ss   11:30   0:00 /bin/bash
root      1312  0.0  0.0  34428  2756 pts/0    R+   11:30   0:00 ps aux

netstat -aplno output:

root@flume:/opt# netstat -aplno
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name Timer
tcp        0      0 0.0.0.0:5445            0.0.0.0:*               LISTEN      6/java           off (0.00/0/0)
tcp        0      0 0.0.0.0:38757           0.0.0.0:*               LISTEN      8/java           off (0.00/0/0)
tcp        0      0 0.0.0.0:5446            0.0.0.0:*               LISTEN      8/java           off (0.00/0/0)
tcp        0      0 0.0.0.0:5447            0.0.0.0:*               LISTEN      10/java          off (0.00/0/0)
tcp        0      0 0.0.0.0:35463           0.0.0.0:*               LISTEN      10/java          off (0.00/0/0)
tcp        0      0 0.0.0.0:33611           0.0.0.0:*               LISTEN      10/java          off (0.00/0/0)
tcp        0      0 0.0.0.0:33301           0.0.0.0:*               LISTEN      6/java           off (0.00/0/0)
tcp        0      0 0.0.0.0:43423           0.0.0.0:*               LISTEN      6/java           off (0.00/0/0)
tcp        0      0 0.0.0.0:41155           0.0.0.0:*               LISTEN      8/java           off (0.00/0/0)
tcp        0      0 127.0.0.11:44419        0.0.0.0:*               LISTEN      -                off (0.00/0/0)
tcp        0      0 172.19.0.4:37734        172.19.0.2:2181         ESTABLISHED 12/java          off (0.00/0/0)
udp        0      0 127.0.0.11:54111        0.0.0.0:*                           -                off (0.00/0/0)
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ]         STREAM     CONNECTED     372765   6/java              
unix  2      [ ]         STREAM     CONNECTED     370037   12/java             
unix  2      [ ]         STREAM     CONNECTED     370085   10/java             
unix  2      [ ]         STREAM     CONNECTED     372768   8/java

Output files are empty:

root@flume:/opt# cat /var/log/flume-ng/
filter.out  ingest.out  output.out  
root@flume:/opt# cat /var/log/flume-ng/filter.out 
root@flume:/opt# cat /var/log/flume-ng/ingest.out 
root@flume:/opt# cat /var/log/flume-ng/output.out 
root@flume:/opt#

UPDATE: I think I found the issue. morphline-lib directory is not in the classpath...

root@flume:/opt# cat /opt/log/filter.log 
12 Dec 2017 11:22:51,520 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:62)  - Configuration provider starting
12 Dec 2017 11:22:51,540 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:134)  - Reloading configuration file:/opt/flume-config/conf/filter/filter.conf
12 Dec 2017 11:22:51,574 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,575 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,575 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,578 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,578 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,578 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,578 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:k1
12 Dec 2017 11:22:51,579 INFO  [conf-file-poller-0] (org.apache.flumClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 16 more2:51,630 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [filter]
12 Dec 2017 11:22:51,631 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:147)  - Creating channels
12 Dec 2017 11:22:51,651 INFO  [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:42)  - Creating instance of channel c1 type memory
12 Dec 2017 11:22:51,669 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:201)  - Created channel c1
12 Dec 2017 11:22:51,671 INFO  [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:41)  - Creating instance of source r1, type avro
12 Dec 2017 11:22:51,745 ERROR [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:146)  - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/kitesdk/morphline/api/Command
    at org.apache.flume.sink.solr.morphline.MorphlineInterceptor.<init>(MorphlineInterceptor.java:56)
    at org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder.build(MorphlineInterceptor.java:113)
    at org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder.build(MorphlineInterceptor.java:104)
    at org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:113)
    at org.apache.flume.channel.ChannelProcessor.configure(ChannelProcessor.java:82)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:348)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:101)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.kitesdk.morphline.api.Command
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 16 more

ferrarimarco commented 6 years ago

Additional points for improvement have been moved to the description of the commit

scissor-project / open-scissor

Dockerize all the components #12

TODO

Components to Dockerize