MISP Server Configuration resets if container stop and restart

[francesco-ficarola]

Hello,

I set several custom parameters in the MISP server configuration. However, after restarting the container, all settings were lost and the default (initial) configuration appeared again.

Thanks, Francesco

[ventz]

@francesco-ficarola Make sure you are using persistent volumes with docker. All of the MISP settings are stored in the MySQL DB -- and that's on your persistent volume.

This is essentially the -v $docker-root/misp-db:/var/lib/mysql line in the DB initialization, and then every time you start your container.

Which produces (assuming the $docker-root is /docker) this:

[/docker/misp-db]
% ls
aria_log.00000001  certs    ib_logfile1  misp       mysql           tc.log
aria_log_control   ib_logfile0  ibdata1      multi-master.info  performance_schema

I just launched a new container with the current codebase, changed a few settings, and stopped it. After starting it back up, everything was there.

[francesco-ficarola]

@ventz Thank you for your quick response. I followed every step in the README. Indeed, I run:

# docker run -it --rm \
-v /docker/misp-db:/var/lib/mysql \
harvarditsecurity/misp /init-db

Then I executed:

# docker run -it -d \
-p 443:443 \
-p 8080:8080 \
-p 3306:3306 \
-v /docker/misp-db:/var/lib/mysql \
harvarditsecurity/misp

But if I change a variable in the server settings, after stopping the container as follow

# docker stop <id_container>

and start again the container using the command above, the custom setting is lost.

Contrary, events related to malware that I loaded are still there...

This is my /docker/misp-db directory:

root@xmisp-p1:/docker/misp-db# ll
total 176168
drwxr-xr-x 5 systemd-bus-proxy systemd-resolve     4096 Jan 25 08:57 ./
drwxr-xr-x 3 root              root                4096 Jan 21 15:50 ../
-rw-rw---- 1 systemd-bus-proxy systemd-resolve    16384 Jan 25 08:57 aria_log.00000001
-rw-rw---- 1 systemd-bus-proxy systemd-resolve       52 Jan 25 08:57 aria_log_control
-rw-r--r-- 1 systemd-bus-proxy systemd-resolve        0 Jan 21 15:51 .db_initialized
-rw-rw---- 1 systemd-bus-proxy systemd-resolve 79691776 Jan 25 08:57 ibdata1
-rw-rw---- 1 systemd-bus-proxy systemd-resolve 50331648 Jan 25 08:57 ib_logfile0
-rw-rw---- 1 systemd-bus-proxy systemd-resolve 50331648 Jan 21 15:50 ib_logfile1
drwx------ 2 systemd-bus-proxy systemd-resolve     4096 Jan 22 13:41 misp/
-rw-rw---- 1 systemd-bus-proxy systemd-resolve        0 Jan 21 15:51 multi-master.info
drwx------ 2 systemd-bus-proxy systemd-resolve     4096 Jan 21 15:50 mysql/
drwx------ 2 systemd-bus-proxy systemd-resolve     4096 Jan 21 15:50 performance_schema/

However, if I run docker volume ls no volume is listed.

[ventz]

@francesco-ficarola Assuming you are using Ubuntu as your base-os from the systemd-bus-proxy? What version of docker are you using?

Can you try docker rm -f $container_id(instead of stop) -- what happens when you re-launch it after that?

If you change the webUI password, does it stick? -- that is, is it only related to the server settings (the password I believe is "technically" a server setting).

This is very strange and I would love to be able to replicate it.

Would you mind outlining step by step on a clean setup (everything including mkdir for the folder, launch/init DB, start container - step by step what you do in the webUI, what setting you change, etc), so that we can debug? If you can even record a screen capture, that would be even better.

Thank you!

[francesco-ficarola]

@ventz Thank you for your support.

root@xmisp-p1:/docker/misp-db# docker --version
Docker version 18.09.1, build 4c52b90
root@xmisp-p1:/docker/misp-db# cat /etc/issue
Ubuntu 16.04.4 LTS \n \l

I tried to change my password as you suggested. That setting is permanent.

Anyway, here you are a step-by-step process with screenshots:

• Creating /docker/misp-db:

• Building: ./build.sh

• Initialize Database:

• Start the container:

• First access to MISP dashboard:

• Change a pair of settings:

• Stop the container:

• List /docker/misp-db:

• Start the container again:

• Settings restored to default:

[francesco-ficarola]

Hi @ventz, were you able to replicate the issue?

Edit: in the meantime I've upgraded to Ubuntu 18.04, so I'll try again.

[ventz]

@francesco-ficarola Sorry for the delay in reply -- we've been actually trying to replicate this problem, but it's working correctly for us here. I even tried on 2 completely clean systems, where on one I built the image, and on the other we pulled it as a binary -- on both it seems to save the settings.

Is it possible that your volume mount is not writable by docker somehow? (that is - it's the correct ownership, but the parent directory won't allow it to write)

There are essentially 2 places the information is stored: 1.) The DB 2.) /var/www/MISP/app/Config/config.php (for example, you should see a live and language around lines 34-35 in that file)

My assumption is that your DB/permissions are working correctly there, but your docker volumes are not. That would make sense as to why you can change the password and use the new one, but the DB info is not being loaded back into the config.php

[francesco-ficarola]

@ventz No problem! Thank you for your support.

I don't see any volume by typing docker volume ls. Is that normal?

Anyway, I don't find any string live or language within /var/www/MISP/app/Config/config.php

Thanks, Francesco

[francesco-ficarola]

Ok, if for instance I change the setting live to true, I can see that setting in /var/www/MISP/app/Config/config.php (but only after changing the setting, otherwise that does not exist, as showed previously).

Anyway, if I stop and start again the container, then that setting disappears from the file:

I noticed that when I start again the container, its ID (and hostname) is different from the previous one. Is that normal? It appears as a new instance of the image (like a virgin).

[ventz]

@francesco-ficarola I think I just picked up on what is happening from your last screenshot - it seems you are launching new containers with their own internal volumes.

Would you mind cleaning up the "left over" ones just to make sure that only 1 exist:

docker ps -a | awk {'print $1'} | xargs docker rm -f

After that, clean out the misp directories/db/remove all data, and start clean.

Init a new dir + db first. Then run the docker container (in daemon mode - per instructions). After you make the changes, try doing a docker stop $container-id Then try starting it right away docker start $container-id I bet at this point it will work.

After your last screenshot, I noticed the issue present in your first screenshot too, but at the time it didn't stand out that the container ID had changed -- you are not starting the same copy of the image, but actually launching a new container/with it's own volume system.

That said, now that brings up the big question - should the "web" volume be mapped externally, so that it can be "moved" outside of docker - not sure, that's a bigger "design" question.

[francesco-ficarola]

Excellent, thank you very much! As you noticed, that was my fault... I thought I had to start the container always using the command written in the README, instead of simply docker start $container-id. Thanks again, I think we can close this issue!

[ventz]

@francesco-ficarola No problem at all - I am glad it's working!

[b00lpy]

Hi, I know the Issue is closed, but I've got a question related with that. Is it possibile to mount another volume pointing at /var/www/MISP/app/Config/ in order to keep starting all the containers always with docker compose?

Thanks

[ventz]

@b00lpy The problem becomes that if you volume mount, it has to be the last operation within Dockerfile -- everything after the VOLUME declaration that's done to the volume is ignored, so that leaves the question of how to modify/set custom values for someone who is not bringing their own config.

Ex, things like this:

sudo -u www-data cp -a /var/www/MISP/app/Config/config.default.php /var/www/MISP/app/Config/config.php

sed -i -E "s/'salt'(\s+)=>\s''/'salt' => '`openssl rand -base64 32 | tr \'/\' \'0\'`'/" /var/www/MISP/app/Config/config.php ; \

sed -i -E "s/'baseurl'(\s+)=>\s''/'baseurl' => 'https:\/\/${MISP_FQDN}'/" /var/www/MISP/app/Config/config.php ; \

sed -i -e "s/email@address.com/${MISP_EMAIL}/" /var/www/MISP/app/Config/config.php ; \

Essentially, what's really needed is a change in the MISP project to have a "defaults" config, and an override file. I am actually pretty surprised that MISP hasn't done this already.

If you have an idea of how to solve this, let's do it.

[b00lpy]

@ventz I think you got the point of my question. MISP is a great tool but, speaking in terms of container environment, is still a little stucked with some constraints ( like this ). I think a good solution would be a docker image, provided by MISP, where all config files can be mounted as you suggest, and the only Entrypoint command should be the execution of start scripts for all MISP workers/services.

[SteveClement]

If anyone has good ideas etc, please add them here: https://github.com/MISP/MISP/issues/4727

[ventz]

@SteveClement Awesome - thanks!