search

DCSO / MISP-dockerized

https://github.com/MISP/MISP
BSD 3-Clause "New" or "Revised" License
33 stars 11 forks source link

Fresh install can't create de database #90

Open gaglimax opened 4 years ago

gaglimax commented 4 years ago

Hi,

I am trying to perform a fresh install of the MISP.

Here is what I did (I change password with XXX and domain with localhost) : 

bash ./FOR_NEW_INSTALL.sh
Which version do you want to install:
[ 14 ] - 1.0.4
[ 16 ] - 1.1.1
[ 17 ] - 1.2.0
[ 18 ] - 1.3.0
[ 19 ] - 1.4.0

Please choose the version: 19
Selected version: 1.4.0...
[OK] Create 'backup' folder
[OK] Create  'config' folder
[OK] Create symlink 'current' for the folder 1.4.0
[OK] Create symlink 'current/backup'
[OK] Create symlink 'current/config'
make -C current install
make[1]: Entering directory `/tech/misp/MISP-dockerized/1.4.0'
 ###########    Checking Requirements   ###########
[OK] Docker is installed.               Output: Docker version 19.03.12, build 48a66213fe
[OK] Git is installed.                  Output: git version 1.8.3.1

[OK]   Check: https://misp.dcso.de
       Result: * Connected to proxy-b.xxx.fr (100.121.67.104) port 8080 (#0).
[OK]   Check: https://dockerhub.dcso.de/v2/
       Result: * Connected to proxy-b.xxx.fr (100.121.67.104) port 8080 (#0).
[OK]   Check: https://github.com/DCSO/misp-dockerized
       Result: * Connected to proxy-b.xxx.fr (100.121.67.114) port 8080 (#0).
[OK]   Check: https://github.com/misp/misp
       Result: * Connected to proxy-b.xxx.fr (100.121.67.114) port 8080 (#0).

[OK] User 'root' is root.
[OK] User 'root' has access to Docker daemon.

Create config and config/ssl directory...finished.
Create config/smime directory...finished.
Create config/pgp directory...finished.
[OK] Folder config exists.
[OK] Testfile in 'config' can be created.
[OK] Folder config/ssl exists.
[OK] Testfile in 'config/ssl' can be created.
[OK] Folder config/pgp exists.
[OK] Testfile in 'config/pgp' can be created.
[OK] Folder config/smime exists.
[OK] Testfile in 'config/smime' can be created.
[OK] Folder backup exists.
[OK] Testfile in 'backup' can be created.

[WARN] No SSL certificate found. We create a self-signed certificate in the volume.
     To change the SSL certificate and private key later:
     1. Please save your certificate in:      /tech/misp/MISP-dockerized/1.4.0/config/ssl/cert.pem
     2. Please save your private keyfile in:  /tech/misp/MISP-dockerized/1.4.0/config/ssl/key.pem
     3. do:                         make config-ssl

[WARN] No S/MIME certificate found.
     1. Please save your certificate in:  /tech/misp/MISP-dockerized/1.4.0/config/smime/cert.pem
     2. Please save your private key  in:  /tech/misp/MISP-dockerized/1.4.0/config/smime/key.pem
     3. Do:                        make config-smime

[WARN] No PGP key found.
     To replace the PGP public and private file later:
     1. Please save your public key in:      /tech/misp/MISP-dockerized/1.4.0/config/pgp/public.key
     2. Please save your private key in:  /tech/misp/MISP-dockerized/1.4.0/config/pgp/private.key
     3. Do:                         make config-pgp

End result:
[OK] No error is occured.

 ###########    Build Configuration     ###########
[build_config.sh] Check and import existing configuration...done
[build_config.sh] Check if all vars exists......done
[build_config.sh] Check Docker registry...
We have switched the container repository to the DCSO registry.
      If you want to use the public one from hub.docker.com,
      please change the parameter 'DOCKER_REGISTRY' at /tech/misp/MISP-dockerized/config/config.env and 'make install'

[build_config.sh] Check container version......done
[build_config.sh] Manual build...
[build_config.sh] Hostname (FQDN - example.org is not a valid FQDN) [DEFAULT: localhost]: localhost
[build_config.sh] Which MISP Network should we use [DEFAULT: 192.168.47.0/28]: 192.168.47.0/28
[build_config.sh] Which MISP Network BRIDGE Interface Name should we use [DEFAULT: mispbr0]: mispbr0
[build_config.sh] Do you want to use an external Database? [y/N] no
[build_config.sh] Set DB Host to docker default: localhost
[build_config.sh] Set DB Host Port to docker default: 3306
[build_config.sh] Which DB Root Password should we use for DB Connection [DEFAULT: generated]: XXXXX
[build_config.sh] Which DB Name should we use for DB Connection [DEFAULT: misp]: misp
[build_config.sh] Which DB User should we use for DB Connection [DEFAULT: misp]: misp
[build_config.sh] Which DB User Password should we use for DB Connection [DEFAULT: generated]: XXXXX
[build_config.sh] If you do a fresh installation, you should define a SALT. Is this SALT ok [DEFAULT: generated]: XXXXX
[build_config.sh] Which sender mailadress should MISP use [DEFAULT: admin@localhost]: admin@localhost
[build_config.sh] Which HTTP Serveradmin mailadress should we use [DEFAULT: admin@localhost]: admin@localhost
[build_config.sh] Should we use a proxy configuration? [y/N] y
[build_config.sh] Which proxy we should use for HTTPS connections (for example: http://proxy.example.com:8080) [DEFAULT: ]: proxy-b.xxx.fr:8080
[build_config.sh] Which proxy we should use for HTTP  connections (for example: http://proxy.example.com:8080) [DEFAULT: ]: proxy-b.xxx.fr:8080
[build_config.sh] For which site(s) we shouldn't use a Proxy (for example: localhost,127.0.0.0/8,docker-registry.somecorporation.com) [DEFAULT: 0.0.0.0]: localhost
[build_config.sh] Which mail domain we should use [DEFAULT: example.com]: example.com
[build_config.sh] Which relay host we should use [ IP or DNS]: mail.example.com
[build_config.sh] Which relay user we should use [DEFAULT: generated]: XXXX
[build_config.sh] Which relay user password we should use [DEFAULT: generated]: XXXX
[build_config.sh] Which sender address we should use [MAIL]:admin@localhost
[build_config.sh] Should we enable debugging options for a special peer? [y/N]: y
[build_config.sh] For which peer we need debug informations?: DEBUG_PEER
[build_config.sh] Would you start with S/MIME? [y/N] n
[build_config.sh] Would you start with PGP? [y/N] n
[build_config.sh] Would you enable Syslog logging? [y/N] n
[build_config.sh] How often you to start a cronjob? [ Dafault: 3600(s) | 0 means deactivated ]: 3600
[build_config.sh] Which user id do you want to use for the cron job execution? [ Default: 1 ]: 1
[build_config.sh] Set PHP variable memory_limit? [ Default: 512M ]: 512M
[build_config.sh] Set PHP variable max_execution_time? [ Default: 300 ]: 300
[build_config.sh] Set PHP variable post_max_size? [ Default: 50M ]: 50M
[build_config.sh] Set PHP variable upload_max_filesize? [ Default: 50M ]: 50M
[build_config.sh] Write configuration......done

[build_config.sh] To change the configuration, delete the corresponding line in:
/tech/misp/MISP-dockerized/config/config.env
 ###########    Pull Environment        ###########
docker run \
    --name misp-robot-init \
        --rm \
        --network="host" \
        -v /tech/misp/MISP-dockerized/1.4.0:/srv/MISP-dockerized \
        -v /tech/misp/MISP-dockerized/1.4.0/scripts:/srv/scripts:ro \
        -v ~/.docker:/root/.docker:ro \
        -v /var/run/docker.sock:/var/run/docker.sock:ro \
        dcso/misp-dockerized-robot:2.6 bash -c "docker-compose -f /srv/MISP-dockerized/docker-compose.yml -f /srv/MISP-dockerized/docker-compose.override.yml pull "
The HTTP_PROXY variable is not set. Defaulting to a blank string.
The HTTPS_PROXY variable is not set. Defaulting to a blank string.
The NO_PROXY variable is not set. Defaulting to a blank string.
Pulling misp-server  ... done
Pulling misp-modules ... done
Pulling misp-robot   ... done
Pulling misp-proxy   ... done
Pulling misp-redis   ... done
 ###########    Deploy Environment      ###########
docker run \
    --name misp-robot-init \
        --rm \
        --network="host" \
        -v /tech/misp/MISP-dockerized/1.4.0:/srv/MISP-dockerized \
        -v /tech/misp/MISP-dockerized/1.4.0/scripts:/srv/scripts:ro \
        -v ~/.docker:/root/.docker:ro \
        -v /var/run/docker.sock:/var/run/docker.sock:ro \
        dcso/misp-dockerized-robot:2.6 bash -c "docker-compose -f /srv/MISP-dockerized/docker-compose.yml -f /srv/MISP-dockerized/docker-compose.override.yml up -d "
The HTTP_PROXY variable is not set. Defaulting to a blank string.
The HTTPS_PROXY variable is not set. Defaulting to a blank string.
The NO_PROXY variable is not set. Defaulting to a blank string.
Creating volume "misp-dockerized_misp-vol-ssl" with default driver
Creating volume "misp-dockerized_misp-vol-pgp" with default driver
Creating volume "misp-dockerized_misp-vol-smime" with default driver
Creating volume "misp-dockerized_misp-vol-db-data" with default driver
Creating volume "misp-dockerized_misp-vol-redis-data" with default driver
Creating volume "misp-dockerized_misp-vol-server-apache2-config-sites-enabled" with default driver
Creating volume "misp-dockerized_misp-vol-server-MISP-app-Config" with default driver
Creating volume "misp-dockerized_misp-vol-server-MISP-cakeresque-config" with default driver
Creating volume "misp-dockerized_misp-vol-server-MISP-tmp" with default driver
Creating volume "misp-dockerized_misp-vol-server-MISP-attachments" with default driver
Creating volume "misp-dockerized_misp-vol-proxy-conf" with default driver
Creating misp-server  ... done
Creating misp-robot   ... done
Creating misp-modules ... done
Creating misp-redis   ... done
Creating misp-proxy   ... done

 ###########    MISP environment is ready       ###########
Please go to: localhost
Login credentials:
      Username: admin@admin.test
      Password: admin

Do not forget to change your SSL certificate with:    make change-ssl
Do not forget to change your S/MIME certificate with:    make change-smime
Do not forget to change your PGP certificate with:    make change-pgp'
 ##########################################################

make[1]: Leaving directory `/tech/misp/MISP-dockerized/1.4.0'

And here is the docker misp-server logs :

# docker logs misp-server -f
2020-09-01 13:15:42,370 CRIT Set uid to user 0
2020-09-01 13:15:42,373 INFO supervisord started with pid 7
2020-09-01 13:15:43,376 INFO spawned: 'postfix' with pid 10
2020-09-01 13:15:43,378 INFO spawned: 'db' with pid 11
2020-09-01 13:15:43,385 INFO spawned: 'workers' with pid 12
2020-09-01 13:15:43,386 INFO spawned: 'redis' with pid 14
2020-09-01 13:15:43,388 INFO spawned: 'apache2' with pid 15
2020-09-01 13:15:43,396 INFO spawned: 'cron' with pid 16
2020-09-01 13:15:43,399 INFO spawned: 'rsyslog' with pid 19
[ENTRYPOINT_MARIADB] check if the container was updated...
[ENTRYPOINT_MARIADB] it seems you have done an upgrade to version 1.4.0
[ENTRYPOINT_MARIADB] making sure db permissions are correct
[ENTRYPOINT_RSYSLOG] delete old MISP logs: rm -f /var/www/MISP/app/tmp/logs/empty
[ENTRYPOINT_APACHE] Check if analyze column should be added...
[ENTRYPOINT_APACHE] Change PHP values ...
-e [ENTRYPOINT_REDIS] ############### started REDIS with cmd: '--appendonly yes' #############
[ENTRYPOINT_APACHE] Check if PGP should be enabled....
[ENTRYPOINT_APACHE] PGP should not be activated.
[ENTRYPOINT_APACHE] Check if SMIME should be enabled...
[ENTRYPOINT_APACHE] S/MIME should not be activated.
[ENTRYPOINT_APACHE] Check if a cert is required...
46:C 01 Sep 13:15:43.464 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
46:C 01 Sep 13:15:43.464 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=46, just started
46:C 01 Sep 13:15:43.464 # Configuration loaded
[ENTRYPOINT_APACHE] 13:15:43 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
46:M 01 Sep 13:15:43.467 * Running mode=standalone, port=6379.
46:M 01 Sep 13:15:43.467 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
46:M 01 Sep 13:15:43.467 # Server initialized
46:M 01 Sep 13:15:43.467 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order
to retain the setting after a reboot. Redis must be restarted after THP is disabled.
46:M 01 Sep 13:15:43.467 * Ready to accept connections
[ENTRYPOINT_MARIADB] MYSQL bind address changed
[ENTRYPOINT_MARIADB] Write /etc/mysql/debian.cnf
Adding user `syslog' to group `www-data' ...
mkdir: cannot create directory '/srv/MISP-dockerized/current/config/.update': No such file or directory
2020-09-01 13:15:43,478 INFO exited: db (exit status 1; not expected)
Adding user syslog to group www-data
Done.
[ENTRYPOINT_RSYSLOG] Start rsyslogd
rsyslogd: error during parsing file /etc/rsyslog.d/rsyslog_custom.conf, on or before line 19: warnings occured in file '/etc/rsyslog.d/rsyslog_custom.conf' around line 19 [v8.32.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: unknown facility name "mispzmq" [v8.32.0]
rsyslogd: invalid character in selector line - ';template' expected [v8.32.0]
rsyslogd: error during parsing file /etc/rsyslog.d/rsyslog_custom.conf, on or before line 19: errors occured in file '/etc/rsyslog.d/rsyslog_custom.conf' around line 19 [v8.32.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/rsyslog_custom.conf, on or before line 22: warnings occured in file '/etc/rsyslog.d/rsyslog_custom.conf' around line 22 [v8.32.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: unknown priority name "" [v8.32.0]
rsyslogd: invalid character in selector line - ';template' expected [v8.32.0]
rsyslogd: error during parsing file /etc/rsyslog.d/rsyslog_custom.conf, on or before line 22: errors occured in file '/etc/rsyslog.d/rsyslog_custom.conf' around line 22 [v8.32.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.32.0 try http://www.rsyslog.com/e/2145 ]
rsyslogd: imfile: error with inotify API, ignoring file '/var/www/MISP/app/tmp/logs/error.log': No such file or directory [v8.32.0]
rsyslogd: imfile: error with inotify API, ignoring file '/var/www/MISP/app/tmp/logs/resque-worker-error.log': No such file or directory [v8.32.0]
rsyslogd: imfile: error with inotify API, ignoring file '/var/www/MISP/app/tmp/logs/resque-scheduler-error.log': No such file or directory [v8.32.0]
rsyslogd: imfile: error with inotify API, ignoring file '/var/www/MISP/app/tmp/logs/mispzmq.log': No such file or directory [v8.32.0]
rsyslogd: imfile: error with inotify API, ignoring file '/var/www/MISP/app/tmp/logs/mispzmq.error.log': No such file or directory [v8.32.0]
rsyslogd: file '/dev/stdout': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
2020-09-01 13:15:44,504 INFO success: postfix entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-01 13:15:44,505 INFO spawned: 'db' with pid 70
2020-09-01 13:15:44,506 INFO success: workers entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-01 13:15:44,506 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-01 13:15:44,506 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-01 13:15:44,506 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-01 13:15:44,506 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[ENTRYPOINT_MARIADB] check if the container was updated...
[ENTRYPOINT_MARIADB] it seems you have done an upgrade to version 1.4.0
[ENTRYPOINT_MARIADB] making sure db permissions are correct
[ENTRYPOINT_MARIADB] MYSQL bind address changed
[ENTRYPOINT_MARIADB] Write /etc/mysql/debian.cnf
mkdir: cannot create directory '/srv/MISP-dockerized/current/config/.update': No such file or directory
2020-09-01 13:15:44,535 INFO exited: db (exit status 1; not expected)
[ENTRYPOINT_APACHE] 13:15:45 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
2020-09-01 13:15:47,475 INFO spawned: 'db' with pid 145
[ENTRYPOINT_APACHE] 13:15:47 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_MARIADB] check if the container was updated...
[ENTRYPOINT_MARIADB] it seems you have done an upgrade to version 1.4.0
[ENTRYPOINT_MARIADB] making sure db permissions are correct
[ENTRYPOINT_MARIADB] MYSQL bind address changed
[ENTRYPOINT_MARIADB] Write /etc/mysql/debian.cnf
mkdir: cannot create directory '/srv/MISP-dockerized/current/config/.update': No such file or directory
2020-09-01 13:15:47,512 INFO exited: db (exit status 1; not expected)
[ENTRYPOINT_APACHE] 13:15:49 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
2020-09-01 13:15:51,482 INFO spawned: 'db' with pid 167
[ENTRYPOINT_APACHE] 13:15:51 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_MARIADB] check if the container was updated...
[ENTRYPOINT_MARIADB] it seems you have done an upgrade to version 1.4.0
[ENTRYPOINT_MARIADB] making sure db permissions are correct
[ENTRYPOINT_MARIADB] MYSQL bind address changed
[ENTRYPOINT_MARIADB] Write /etc/mysql/debian.cnf
mkdir: cannot create directory '/srv/MISP-dockerized/current/config/.update': No such file or directory
2020-09-01 13:15:51,522 INFO exited: db (exit status 1; not expected)
2020-09-01 13:15:52,523 INFO gave up: db entered FATAL state, too many start retries too quickly
[ENTRYPOINT_APACHE] 13:15:53 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:15:55 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:15:57 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:15:59 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:01 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:03 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:05 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:07 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:09 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:11 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:13 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:15 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:17 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:19 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:21 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:23 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:25 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:27 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:29 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:31 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:33 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:35 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.
[ENTRYPOINT_APACHE] 13:16:37 -  misp-proxy container create currently the certificate. misp-server wait until misp-proxy is finished.

[ENTRYPOINT_APACHE] Check if a dh file is required

[ENTRYPOINT_APACHE] Check if HTTPS MISP config should be enabled...
[ENTRYPOINT_APACHE] Check if HTTP MISP config should be disabled...
[ENTRYPOINT_APACHE] Check if Redis is ready...
[ENTRYPOINT_APACHE] Check if MySQL is ready...
[ENTRYPOINT_APACHE] ... wait until mariadb entrypoint has completly created the database
[ENTRYPOINT_APACHE] ... wait until mariadb entrypoint has completly created the database
[ENTRYPOINT_APACHE] ... wait until mariadb entrypoint has completly created the database
[ENTRYPOINT_APACHE] ... wait until mariadb entrypoint has completly created the database
[ENTRYPOINT_APACHE] ... wait until mariadb entrypoint has completly created the database
[ENTRYPOINT_APACHE] ... wait until mariadb entrypoint has completly created the database

The database is never created.

Thanks for any help.

talltechy commented 4 years ago

I also had this issue, Getting 502 Bad gateway error and logs showing no database

malvidin commented 4 years ago

The issue appears to be with the misp-server, not the misp-proxy. https://github.com/DCSO/MISP-dockerized-server/issues/50

Workaround after make install: docker exec misp-server mkdir -p /srv/MISP-dockerized/current/config/.update