search

harvard-itsecurity / docker-misp

Automated Docker MISP container - Malware Information Sharing Platform and Threat Sharing
BSD 3-Clause "New" or "Revised" License
175 stars 44 forks source link

GnuPG generation fail #36

Closed ghost closed 5 years ago

ghost commented 5 years ago

Hello! When building a docker image, on the very end I got this in the output (this is just a portion of it):

...
Build process completed successfully
Installing '/usr/lib/php/20170718/ssdeep.so'
install ok: channel://pecl.php.net/ssdeep-1.1.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=ssdeep.so" to php.ini
RNGDOPTIONS=--random-device /dev/urandom --rng-device /dev/urandom
HRNGDEVICE=/dev/urandom
Stopping Hardware RNG entropy gatherer daemon: (not running).
Starting Hardware RNG entropy gatherer daemon: rngd.
unable to adjust write_wakeup_threshold: Read-only file system
RNDADDENTROPY failed: Operation not permitted

gpg: keybox '/var/www/MISP/.gnupg/pubring.kbx' created
gpg: agent_genkey failed: Inappropriate ioctl for device
gpg: key generation failed: Inappropriate ioctl for device
gpg: WARNING: nothing exported
Stopping Hardware RNG entropy gatherer daemon: (not running).
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  udev
Use 'sudo apt autoremove' to remove it.
The following packages will be REMOVED:
  rng-tools*
0 upgraded, 0 newly installed, 1 to remove and 5 not upgraded.
...

I mean, docker image is built correctly. I can start (docker run) an image as well, but clicking "Download: GnuPG key" reveals blank web page (https://myfqdn/gpg.asc is blank).

ventz commented 5 years ago

Hi - it seems like your file system is in read-only mode:

unable to adjust write_wakeup_threshold: Read-only file system RNDADDENTROPY failed: Operation not permitted

I am assuming it could not write the gpg keys after it generated them.

ghost commented 5 years ago

Thanks for the answer. I've tried mount -o remount,rw / but with no success (I have Ubuntu Server 18.04.). I will manually generate keys and try to map them into the separate Docker volume.

ventz commented 5 years ago

@XizzoR Try to build it somewhere outside of / itself (for example, try creating "/docker/misp")

Here is what our build logs look like -- this is from the docker autobuild, but also, I just built it manually:

RNGDOPTIONS=--random-device /dev/urandom --rng-device /dev/urandom
HRNGDEVICE=/dev/urandom
Stopping Hardware RNG entropy gatherer daemon:
(not running).
Starting Hardware RNG entropy gatherer daemon:
rngd.
unable to adjust write_wakeup_threshold: Read-only file system
RNDADDENTROPY failed: Operation not permitted
gpg: keybox '/var/www/MISP/.gnupg/pubring.kbx
' created
gpg: agent_genkey failed: Inappropriate ioctl for device
gpg: key generation failed: Inappropriate ioctl for device
gpg:
WARNING: nothing exported
Stopping Hardware RNG entropy gatherer daemon:
(not running).
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
udev

Can you try catting /dev/urandom?

ghost commented 5 years ago

Thanks, but still no luck (making /docker/misp/ and building it from this dir, also building image from /home/username/). Catting /dev/urandom is not possible, since it has binary content, or am I missing something here?

ventz commented 5 years ago

@XizzoR What is your environment like (VMware ESX/AWS/Virtualbox/etc)

I have tried building this on a few systems here, and not seeing that issue.

For /dev/urandom - yea, just wanted to see if you have it/it's "generating" data. If you are using apparmor (ubuntu) or selinux (rhel/fc), it's possible that something is preventing the system from accessing /dev/urandom correctly. That said, for some reason docker things your volume is read-only. You can try to get a shell into the image after you deploy it and check if you can write to the "/" from within docker. Same to see if the docker container can "see" the data from /dev/urandom.

ghost commented 5 years ago

My environment is VMware ESXi. AFAIK, I am not using apparmor. And yes, I can write within "/" directory in misp docker image.

ventz commented 5 years ago

Can you test this on a personal system using VMware workstation/Fusion?

We cannot replicate this problem here -- have tried on a few cloud providers and personal systems.

ghost commented 5 years ago

Tried on VMware Fusion too. Same problem. Also, no luck with Ubuntu 16.04. I'm sorry, but the logs in your comment here looks exactly as mine, yet you said that you can't replicate this problem, or am I missing something?

ventz commented 5 years ago

Let me look into this carefully. I thought your files were empty? (on our side they were not).

For the logs - I may have pasted your output accidentally rather than the one from the build system. On our side I believe the message about the READONLY file system is not there. (I don't have access to the build system at the moment - will take a look later)

ventz commented 5 years ago

@XizzoR I think something has actually changed with GPG's batch automation between the last build (we keep packages versioned and "cached") and the "latest" available in the Ubuntu 18.04.

I think I also got to the bottom of the problem. Building a copy manually now and will let you know.

ventz commented 5 years ago

Yep - that's what it seems.

I'll push a new build right now, but look at this:

# cat /var/www/MISP/app/webroot/gpg.asc
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFyNQC4BEADIMd+nYwIZoX1MTBmM8ll3axa//aHGZZksJmAWY6Tm2eCA/LmQ
biuJfRrByBuMeU5bRYw23F8f+S/KPeqzDTG3zZjAv73+QSAxd0nq8tM+I4sDvIdB
DNvuwSkvlUA3WLWd2M28tJGctW0BLhy/DO1pYOozykkswXn1dCTatooDggdz38i7
...
b21lj4eY91EDetn7lEhFV/CURxnvzWG+EPAx/BEJjy5IQEI42iYKJAUtRkzaiot1
uJ+CiuYhjgXeWWW3RCaHRn2aT7UJnevz
=XNmP
-----END PGP PUBLIC KEY BLOCK-----

@XizzoR You should see change: https://github.com/harvard-itsecurity/docker-misp/commit/68b6b5a5be40a58f02a806b271a9b9012069c3d3

Attributed problem find to you in the change commit (also listed you on the README: https://github.com/harvard-itsecurity/docker-misp/blob/master/README.md)

Thank you for being persistent and following up on this!

ghost commented 5 years ago

Thank you! I'm glad that this is fixed now. Now, when I built an image, build logs looks like this:

Build process completed successfully
Installing '/usr/lib/php/20170718/ssdeep.so'
install ok: channel://pecl.php.net/ssdeep-1.1.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=ssdeep.so" to php.ini
HRNGDEVICE=/dev/urandom
Stopping Hardware RNG entropy gatherer daemon: (not running).
Starting Hardware RNG entropy gatherer daemon: rngd.
unable to adjust write_wakeup_threshold: Read-only file system
RNDADDENTROPY failed: Operation not permitted

gpg: keybox '/var/www/MISP/.gnupg/pubring.kbx' created
gpg: /var/www/MISP/.gnupg/trustdb.gpg: trustdb created
gpg: key A33CF808111214E5 marked as ultimately trusted
gpg: directory '/var/www/MISP/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/var/www/MISP/.gnupg/openpgp-revocs.d/75DC0F7637C587F66A64E94CA33CF808111214E5.rev'
Stopping Hardware RNG entropy gatherer daemon: (not running).
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
  udev
Use 'sudo apt autoremove' to remove it.
The following packages will be REMOVED:
  rng-tools*

It's still telling about "Read-only file system" (specifically, the command sudo rngd -f -r /dev/urandom is the problematic one), but gpg.asc is successfully created and it's finally accessible through web gui. I am closing this issue, since gpg key generation is not failing anymore. :)