create graylog-enterprise and graylog images

[jalogisch]

This PR will enable the build of two Graylog images.

graylog - what is the current way, only OSS plain vanilla Graylog graylog-enterprise - including all enterprise and integration plugins.

Before this gets merged the build rules on hub.docker needs to be adjusted.

Target of this to make it easier in docker based infrastructures to use the enterprise plugins

[malcyon]

@jalogisch I can get it to work locally if I run this:

docker run --name graylog --link mongo --link elasticsearch -p 9000:9000 -p 12201:12201 -p 1514:1514 -e GRAYLOG_HTTP_EXTERNAL_URI="http://172.17.0.4:9000/" -e GRAYLOG_HTTP_PUBLISH_URI="http://172.17.0.4:9000/" -d graylog-enterprise:latest

It also works with this docker compose file I threw together:

https://github.com/Graylog2/graylog-docker/blob/docker-compose/docker-compose.yml

[jalogisch]

Build, linter and integration test runs local without any issues. Need to find the reason why this fails now on travis.

Creating test_elasticsearch_1 ... done
Creating test_mongo_1         ... done
Creating test_graylog_1       ... done
wait for graylog port 9000
Waiting until Graylog has been started
HTTP/1.1 200 OK
X-UA-Compatible: IE=edge
X-Graylog-Node-ID: 32959d85-ce71-4d03-bbe5-fbc3503f8aa0
Content-Length: 1940
Content-Type: text/html

{"cluster_id":"03f957d8-3c67-4a5b-83e2-bc65dc07182e","node_id":"32959d85-ce71-4d03-bbe5-fbc3503f8aa0","version":"4.0.0-rc.2+49529be","tagline":"Manage your logs in the dark and have lasers going and make it look like you're from space!"}
inspect needed containers
 /test_graylog_1 has pid 1622            - healthy
 /test_elasticsearch_1 has pid 1226      - null
 /test_mongo_1 has pid 1247              - null

get cluster state with session
{"32959d85-ce71-4d03-bbe5-fbc3503f8aa0":{"facility":"graylog-server","codename":"Noir","node_id":"32959d85-ce71-4d03-bbe5-fbc3503f8aa0","cluster_id":"03f957d8-3c67-4a5b-83e2-bc65dc07182e","version":"4.0.0-rc.2+49529be","started_at":"2020-11-17T09:49:00.364Z","hostname":"200fa7c7cb5c","lifecycle":"running","lb_status":"alive","timezone":"Etc/UTC","operating_system":"Linux 5.4.39-linuxkit","is_processing":true}}
create permissions to create dashboards
successful

import input stream for plaintext
successful

import input stream for syslog
successful

send message to RAW input stream
send message to syslog input stream

check received messages
plaintext messages found: '1'
syslog messages found: '1'

Stopping test_graylog_1       ... done
Stopping test_elasticsearch_1 ... done
Stopping test_mongo_1         ... done
Removing test_graylog_1       ... done
Removing test_elasticsearch_1 ... done
Removing test_mongo_1         ... done
Removing network test_default

[malcyon]

@jalogisch You may need to resolve some conflicts, as we made some updates to the 4.0 branch for the new version and license.

[malcyon]

@jalogisch I added a Jenkins job in #134. It might help bypass the Travis CI issue.

[jalogisch]

Local Build is possible:

[+] Building 17.0s (21/21) FINISHED
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 4.96kB                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 34B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/openjdk:8-jre-slim-buster                                                                    1.9s
 => [internal] load metadata for docker.io/library/debian:buster-slim                                                                           1.9s
 => [internal] load build context                                                                                                               0.0s
 => => transferring context: 180B                                                                                                               0.0s
 => [stage-1 1/7] FROM docker.io/library/openjdk:8-jre-slim-buster@sha256:f7b69267a0028409a6a411b473a2bd66cc5bfe25850222ee166e520b09ee4a8c      0.0s
 => [graylog-downloader 1/8] FROM docker.io/library/debian:buster-slim@sha256:062bbd9a1a58c9c5b8fc9d83a206371127ef268cfcc65f1a01227c6faebdb212  0.0s
 => CACHED [graylog-downloader 2/8] WORKDIR /tmp                                                                                                0.0s
 => CACHED [graylog-downloader 3/8] RUN   apt-get update  > /dev/null &&   apt-get install --assume-yes     ca-certificates     curl > /dev/nu  0.0s
 => CACHED [graylog-downloader 4/8] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-4.0.1.tgz"     "https://pa  0.0s
 => CACHED [graylog-downloader 5/8] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-4.0.1.tgz.sha256.txt"       0.0s
 => CACHED [graylog-downloader 6/8] RUN   sha256sum --check "graylog-4.0.1.tgz.sha256.txt"                                                      0.0s
 => CACHED [graylog-downloader 7/8] RUN   mkdir /opt/graylog &&   tar --extract --gzip --file "/tmp/graylog-4.0.1.tgz" --strip-components=1 --  0.0s
 => CACHED [graylog-downloader 8/8] RUN   install     --directory     --mode=0750     /opt/graylog/data     /opt/graylog/data/journal     /opt  0.0s
 => CACHED [stage-1 2/7] COPY --from=graylog-downloader /opt/graylog /usr/share/graylog                                                         0.0s
 => CACHED [stage-1 3/7] COPY config /usr/share/graylog/data/config                                                                             0.0s
 => CACHED [stage-1 4/7] WORKDIR /usr/share/graylog                                                                                             0.0s
 => [stage-1 5/7] RUN   echo "export JAVA_HOME=/usr/local/openjdk-8"     > /etc/profile.d/graylog.sh &&   echo "export BUILD_DATE=2020-12-01T  13.6s
 => [stage-1 6/7] COPY docker-entrypoint.sh /                                                                                                   0.0s
 => [stage-1 7/7] COPY health_check.sh /                                                                                                        0.0s
 => exporting to image                                                                                                                          1.3s
 => => exporting layers                                                                                                                         1.3s
 => => writing image sha256:410b5628ca753245755d584ba6d7a6b19e5131a3c656d31c362d0086546e982d                                                    0.0s
 => => naming to docker.io/library/graylog                                                                                                      0.0s
[+] Building 55.0s (27/27) FINISHED
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 6.34kB                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 34B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/openjdk:8-jre-slim-buster                                                                    1.7s
 => [internal] load metadata for docker.io/library/debian:buster-slim                                                                           1.8s
 => [graylog-downloader 1/14] FROM docker.io/library/debian:buster-slim@sha256:062bbd9a1a58c9c5b8fc9d83a206371127ef268cfcc65f1a01227c6faebdb21  0.0s
 => [internal] load build context                                                                                                               0.0s
 => => transferring context: 180B                                                                                                               0.0s
 => [stage-1 1/7] FROM docker.io/library/openjdk:8-jre-slim-buster@sha256:f7b69267a0028409a6a411b473a2bd66cc5bfe25850222ee166e520b09ee4a8c      0.0s
 => CACHED [graylog-downloader 2/14] WORKDIR /tmp                                                                                               0.0s
 => [graylog-downloader 3/14] RUN   apt-get update  > /dev/null &&   apt-get install --assume-yes     ca-certificates     curl > /dev/null      8.2s
 => [graylog-downloader 4/14] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-4.0.1.tgz"     "https://packages  8.4s
 => [graylog-downloader 5/14] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-4.0.1.tgz.sha256.txt"     "https  1.2s
 => [graylog-downloader 6/14] RUN   sha256sum --check "graylog-4.0.1.tgz.sha256.txt"                                                            1.2s
 => [graylog-downloader 7/14] RUN   mkdir /opt/graylog &&   tar --extract --gzip --file "/tmp/graylog-4.0.1.tgz" --strip-components=1 --direct  2.0s
 => [graylog-downloader 8/14] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-enterprise-integrations-plugins-  2.7s
 => [graylog-downloader 9/14] RUN tar --extract --gzip --file "/tmp/graylog-enterprise-integrations-plugins-4.0.1.tgz" --strip-components=1 --  0.4s
 => [graylog-downloader 10/14] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-integrations-plugins-4.0.1.tgz"  3.0s
 => [graylog-downloader 11/14] RUN tar --extract --gzip --file "/tmp/graylog-integrations-plugins-4.0.1.tgz" --strip-components=1 --directory   0.6s
 => [graylog-downloader 12/14] RUN   curl     --silent     --location     --retry 3     --output "/tmp/graylog-enterprise-plugins-4.0.1.tgz"    5.1s
 => [graylog-downloader 13/14] RUN tar --extract --gzip --file "graylog-enterprise-plugins-4.0.1.tgz" --strip-components=1 --directory /opt/gr  1.9s
 => [graylog-downloader 14/14] RUN   install     --directory     --mode=0750     /opt/graylog/data     /opt/graylog/data/journal     /opt/gray  0.4s
 => CACHED [stage-1 2/7] COPY --from=graylog-downloader /opt/graylog /usr/share/graylog                                                         0.0s
 => CACHED [stage-1 3/7] COPY config /usr/share/graylog/data/config                                                                             0.0s
 => CACHED [stage-1 4/7] WORKDIR /usr/share/graylog                                                                                             0.0s
 => [stage-1 5/7] RUN   echo "export JAVA_HOME=/usr/local/openjdk-8"     > /etc/profile.d/graylog.sh &&   echo "export BUILD_DATE=2020-12-01T  14.1s
 => [stage-1 6/7] COPY docker-entrypoint.sh /                                                                                                   0.0s
 => [stage-1 7/7] COPY health_check.sh /                                                                                                        0.0s
 => exporting to image                                                                                                                          2.5s
 => => exporting layers                                                                                                                         2.5s
 => => writing image sha256:4a39d5582cfa89ba60d367a3ac1566edfee2273fd8e46d7c832f008bec150e78                                                    0.0s
 => => naming to docker.io/library/graylog-enterprise                                                                                           0.0s
[+] Building 24.1s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 1.86kB                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 34B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/openjdk:8-jre-slim-buster                                                                    1.8s
 => [internal] load build context                                                                                                               0.0s
 => => transferring context: 1.04kB                                                                                                             0.0s
 => CACHED [1/4] FROM docker.io/library/openjdk:8-jre-slim-buster@sha256:f7b69267a0028409a6a411b473a2bd66cc5bfe25850222ee166e520b09ee4a8c       0.0s
 => [2/4] RUN apt-get update &&     apt-get -y install --no-install-recommends apt-utils &&     apt-get -y upgrade &&     apt-get -y install -  8.9s
 => [3/4] RUN export GRAYLOG_FORWARDER_VERSION_SHORT=$(echo 3.3.9-20201119163247 | cut -d. -f1).$(echo 3.3.9-20201119163247 | cut -d. -f2) &&  12.1s
 => [4/4] COPY docker/forwarder/forwarder-entrypoint.sh /                                                                                       0.0s
 => exporting to image                                                                                                                          1.3s
 => => exporting layers                                                                                                                         1.3s
 => => writing image sha256:38c0efec5f0430b2d38e4d77228cd5ac298c0ec97da9432330ea32ec290f3e90                                                    0.0s
 => => naming to docker.io/library/graylog-forwarder                                                                                            0.0s

Need to ensure now that Jenkins does the job - @juju2112 that is your area, will you take care?

[jalogisch]

@juju2112 you might want to check the forwarder image and the hadolint messages

+ hadolint docker/oss/Dockerfile
+ hadolint docker/enterprise/Dockerfile
+ hadolint docker/forwarder/Dockerfile
docker/forwarder/Dockerfile:16 SC2155 Declare and assign separately to avoid masking return values.
docker/forwarder/Dockerfile:16 DL4006 Set the SHELL option -o pipefail before RUN with a pipe in it. If you are using /bin/sh in an alpine image or if your shell is symlinked to busybox then consider explicitly setting your SHELL to /bin/ash, or disable this check
make: *** [linter] Error 1