MISP / docker-misp

Automated Docker MISP container - Malware Information Sharing Platform and Threat Sharing
BSD 3-Clause "New" or "Revised" License
105 stars 31 forks source link

Docker image not starting #1

Closed billmurrin closed 6 years ago

billmurrin commented 6 years ago

I'm running the ./build.sh successfully on a Ubuntu box. I am able to run the container. mysql and redis will not start. In my configuration of build.sh, I've changed the FQDN to be 127.0.0.1. I can hit the website on 443, but get a page stating:

An Internal Error Has Occurred.
Error: An Internal Error Has Occurred.

When i pull up a bash in the docker image, I don't see mysql or redis running and see the following output in the supervisord output when the container starts:

2018-07-07 21:07:53,926 INFO supervisord started with pid 1
2018-07-07 21:07:54,928 INFO spawned: 'master' with pid 7
2018-07-07 21:07:54,930 INFO spawned: 'workers' with pid 8
2018-07-07 21:07:54,931 INFO spawned: 'syslog-ng' with pid 9
2018-07-07 21:07:54,932 INFO spawned: 'apache2' with pid 10
2018-07-07 21:07:54,939 INFO spawned: 'cron' with pid 11
2018-07-07 21:07:54,940 INFO spawned: 'misp-modules' with pid 14
2018-07-07 21:07:54,944 INFO spawned: 'mysqld_safe' with pid 17
2018-07-07 21:07:54,946 INFO spawned: 'redis-server' with pid 19
2018-07-07 21:07:54,964 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-07-07 21:07:54,965 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:07:55,425 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:07:55,964 INFO success: master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: workers entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:55,964 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-07-07 21:07:56,065 INFO spawned: 'redis-server' with pid 250
2018-07-07 21:07:56,070 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:07:56,071 INFO reaped unknown pid 255
2018-07-07 21:07:56,437 INFO spawned: 'mysqld_safe' with pid 281
2018-07-07 21:07:56,744 INFO exited: workers (exit status 0; expected)
2018-07-07 21:07:56,764 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:07:58,768 INFO spawned: 'mysqld_safe' with pid 445
2018-07-07 21:07:58,769 INFO spawned: 'redis-server' with pid 446
2018-07-07 21:07:58,775 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:07:58,776 INFO reaped unknown pid 457
2018-07-07 21:07:59,104 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:08:02,109 INFO spawned: 'mysqld_safe' with pid 604
2018-07-07 21:08:02,111 INFO spawned: 'redis-server' with pid 605
2018-07-07 21:08:02,116 INFO exited: redis-server (exit status 0; not expected)
2018-07-07 21:08:02,117 INFO gave up: redis-server entered FATAL state, too many start retries too quickly
2018-07-07 21:08:02,118 INFO reaped unknown pid 615
2018-07-07 21:08:02,493 INFO exited: mysqld_safe (exit status 0; not expected)
2018-07-07 21:08:03,495 INFO gave up: mysqld_safe entered FATAL state, too many start retries too quickly
ventz commented 6 years ago

@billmurrin I just cloned the repo and made your change (127.0.0.1) and built the image onto clean system (Ubuntu VM) and it works for me without any problems:

# mkdir -p /docker/misp-db
# git clone https://github.com/MISP/docker-misp.git
# cd docker-misp
# cat build.sh 
#!/bin/bash
docker rmi harvarditsecurity/misp
docker build \
    --rm=true --force-rm=true \
    --build-arg MYSQL_MISP_PASSWORD=ChangeThisDefaultPassworda9564ebc3289b7a14551baf8ad5ec60a \
    --build-arg POSTFIX_RELAY_HOST=localhost \
    --build-arg MISP_FQDN=127.0.0.1 \
    --build-arg MISP_EMAIL=admin@localhost \
    -t harvarditsecurity/misp container
# ./build.sh
(...too long...)
# docker images | grep misp
harvarditsecurity/misp                                    latest              47ca4c2fcff9        8 minutes ago       1.52GB
# docker run -it --rm -v /docker/misp-db:/var/lib/mysql harvarditsecurity/misp /init-db
Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
2018-07-07 21:57:05 140201963134080 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6) starting as process 41 ...
OK
Filling help tables...
2018-07-07 21:57:09 139698537606272 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6) starting as process 69 ...
OK
Creating OpenGIS required SP-s...
2018-07-07 21:57:12 140677243288704 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6) starting as process 98 ...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'
'/usr/bin/mysqladmin' -u root -h f268178a1c0d password 'new-password'

Alternatively you can run:
'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.

You can start the MariaDB daemon with:
cd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql'

You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/mysql-test' ; perl mysql-test-run.pl

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

180707 21:57:15 mysqld_safe Logging to syslog.
180707 21:57:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
# docker run -it -d -p 443:443 -p 80:80 -p 3306:3306 -v /docker/misp-db:/var/lib/mysql harvarditsecurity/misp
14fa6979d2f93035ff24d018f2f7ffe432ca3b3b2e2d42e3e0c40f477c3bec85

misp-localhost

Just for sanity, I confirmed this on my local system too and it worked there also.

I would suggest taking a look at your environment -- start by cleaning your env and deleting the image ("docker rmi harvarditsecurity/misp"). Then go with the binary first instead of a build ("docker pull harvarditsecurity/misp"). Go through the motions and make sure that runs for you without problems. Confirm the DB initialize successfully before you run the container. If that works, remove it again ("docker rmi harvarditsecurity/misp") and then as a next step clone the github repo and build the default one without any modifications, delete the mysql DB/dir, re-create it + init it, and run that one.

By default the container is set to bind to "localhost", which in this case for all purposes is the same as 127.0.0.1 The reason for this is because $MISP_FQDN is simply used in two places for ServerName, for the permanent redirect to https://$MISP_FQDN, and to configure the baseurl for MISP itself.

Also, check your local containers/services, and make sure nothing else is listening on port tcp/80, tcp/443, tcp/3306 before you run this container.

Another good local debug option is instead of running the container as a service (with -d), instead to run it with an interactive terminal console (with -it) and then manually running the services from the shell/console to see what is failing on your side.

billmurrin commented 6 years ago

OK, I totally did it wrong, I saw the 3 steps but skimmed it, saw the part that mentioned MAC, and skipped the rest of it thinking that it was for MAC. I never initialized the db and also wasn't adding the db volume to the image. DERP... Thanks for your help. It is working now.

ventz commented 6 years ago

@billmurrin Awesome - and good to know thanks. I'll try to add something to hopefully make that more clear.

cellango commented 5 years ago

Docker container works great without certs. Container on startup has the following error: 2019-01-15 22:04:13,791 INFO success: mysqld_safe entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-01-15 22:04:13,792 INFO spawned: 'redis-server' with pid 265 2019-01-15 22:04:13,799 INFO exited: redis-server (exit status 0; not expected) 2019-01-15 22:04:13,800 INFO reaped unknown pid 266 2019-01-15 22:04:14,596 INFO exited: workers (exit status 0; expected) 2019-01-15 22:04:16,599 INFO spawned: 'redis-server' with pid 303 2019-01-15 22:04:16,606 INFO exited: redis-server (exit status 0; not expected) 2019-01-15 22:04:16,607 INFO reaped unknown pid 304 Using this to start container:

!/bin/bash

docker run -it -d \ -p 443:443 \ -p 80:80 \ -p 3306:3306 \ -v /home/centos/docker/docker-misp/misp-db:/var/lib/mysql \ -v /home/centos/docker/docker-misp/certs:/etc/ssl/private \ harvarditsecurity/misp misp.crt and misp.key file in /home/centos/docker/docker-misp/certs. Also put the org master and intermediate in /etc/pki/ca-trust/source/anchors. Is there a way to debug what is going on?

ventz commented 5 years ago

@cellango There are 2 different things going on here.

The errors for redis are actually "fine" (but should be fixed yes) -- if you launch the container manually and run "supervisord", you will see that while it gives the errors, the process is running:

2019-01-16 01:17:11,978 CRIT Supervisor running as root (no user in config file)
2019-01-16 01:17:11,978 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2019-01-16 01:17:11,988 INFO RPC interface 'supervisor' initialized
2019-01-16 01:17:11,988 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-01-16 01:17:11,988 INFO supervisord started with pid 21
2019-01-16 01:17:12,990 INFO spawned: 'master' with pid 24
2019-01-16 01:17:12,993 INFO spawned: 'workers' with pid 25
2019-01-16 01:17:12,994 INFO spawned: 'syslog-ng' with pid 26
2019-01-16 01:17:12,996 INFO spawned: 'apache2' with pid 27
2019-01-16 01:17:12,998 INFO spawned: 'cron' with pid 28
2019-01-16 01:17:12,999 INFO spawned: 'misp-modules' with pid 29
2019-01-16 01:17:13,001 INFO spawned: 'mysqld_safe' with pid 30
2019-01-16 01:17:13,002 INFO spawned: 'redis-server' with pid 34
2019-01-16 01:17:13,013 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2019-01-16 01:17:13,013 INFO exited: redis-server (exit status 0; not expected)
2019-01-16 01:17:14,019 INFO success: master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:17:14,019 INFO success: workers entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:17:14,019 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:17:14,019 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:17:14,019 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:17:14,019 INFO success: mysqld_safe entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:17:14,021 INFO spawned: 'redis-server' with pid 280
2019-01-16 01:17:14,028 INFO exited: redis-server (exit status 0; not expected)
2019-01-16 01:17:14,814 INFO exited: workers (exit status 0; expected)
2019-01-16 01:17:16,822 INFO spawned: 'redis-server' with pid 319
2019-01-16 01:17:16,830 INFO exited: redis-server (exit status 0; not expected)

2019-01-16 01:17:19,839 INFO spawned: 'redis-server' with pid 321
2019-01-16 01:17:19,848 INFO exited: redis-server (exit status 0; not expected)
2019-01-16 01:17:20,849 INFO gave up: redis-server entered FATAL state, too many start retries too quickly

^Z
[2]+  Stopped                 supervisord
root@9a4c9319978e:/var/www/MISP# ps aux | grep redis
redis       40  0.1  0.1  48504  3788 ?        Ssl  01:17   0:00 /usr/bin/redis-server 0.0.0.0:6379

The reason for this is a bit stupid - redis changed their default config to daemonize, and the "fix" is as easy as setting up /etc/redis/redis.conf the daemonize no -- I will fix this upstream.

That fixes the redis problem (again, more log issue):

2019-01-16 01:26:15,442 CRIT Supervisor running as root (no user in config file)
2019-01-16 01:26:15,442 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2019-01-16 01:26:15,448 INFO RPC interface 'supervisor' initialized
2019-01-16 01:26:15,448 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-01-16 01:26:15,448 INFO supervisord started with pid 20
2019-01-16 01:26:16,451 INFO spawned: 'master' with pid 23
2019-01-16 01:26:16,454 INFO spawned: 'workers' with pid 24
2019-01-16 01:26:16,455 INFO spawned: 'syslog-ng' with pid 25
2019-01-16 01:26:16,457 INFO spawned: 'apache2' with pid 26
2019-01-16 01:26:16,459 INFO spawned: 'cron' with pid 27
2019-01-16 01:26:16,461 INFO spawned: 'misp-modules' with pid 28
2019-01-16 01:26:16,462 INFO spawned: 'mysqld_safe' with pid 31
2019-01-16 01:26:16,464 INFO spawned: 'redis-server' with pid 32
2019-01-16 01:26:16,495 INFO success: misp-modules entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2019-01-16 01:26:17,487 INFO success: master entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:17,487 INFO success: workers entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:17,487 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:17,488 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:17,488 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:17,488 INFO success: mysqld_safe entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:17,488 INFO success: redis-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-16 01:26:18,304 INFO exited: workers (exit status 0; expected)

For the SSL cert - I'll open a new issue in case it ends up being something else so that it is easily searchable in the future/helps others.

Thanks for mentioning the redis one by the way -- I've been meaning to fix that for a long time now! Just fixed it now, and will push a fixed copy later tonight.

ventz commented 5 years ago

@cellango Linking cert issue I opened so we can discuss there: https://github.com/MISP/docker-misp/issues/2