[Debian] Murmur can only read certificate if it is file owner

[notinecrafter]

I tried to get murmur (or mumble-server, as the Debian packages so adamantly call it) to use my Let's Encrypt certificates. However, I continuously got the error: "Failed to read /etc/letsencrypt/live/example.com/fullchain.pem".

The Mumble wiki suggests that I have to give mumble-server execute permission in the /etc/letsencrypt/live and /etc/letsencrypt/archive directories. I did this using setfacl, but no change occurred.

I then tried to make a new group called ssl (for exactly this purpose), made mumble-server a member of this group, and set ownership of both directories to root:ssl, but to no avail.

Only when I made mumble-server the owner of (a copy of) the files, as suggested in issue 1949, it managed to successfully read the certificates. However, in a production environment with other things running having mumble be the owner of your certificate files is obviously not entirely desirable.

@janschejbal 's solution to issue 1949, having mumble run with priority, does not solve this issue on my Debian 9 (Stretch) system.

UPDATE: @janschejbal's solution didn't work because dpkg-reconfigure didn't actually change /etc/default/mumble-server. Manually changing /etc/default/mumble-server did solve the issue.

I'm leaving this issue open because it's still unexpected behaviour to say the least, and because of this other issue I encountered with dpkg-reconfigure. If you, as a developer, think this needs to be a separate issue I suggest you make it one.

[mkrautz]

I suppose this would be better suited as a Debian bug?

cc @crknadle

[crknadle]

@mkrautz thanks for the heads-up on this. I just tested 'dpkg-reconfigure mumble-server' with mumble-server 1.2.18-1. The second Debconf quesiton is what sets MURMUR_USE_CAPABILITIES in /etc/default/mumble-server -- it appears to work correctly, but isn't obvious when this is being set because of the wording used right now, which is: "Mumble-server (murmurd) can use higher priority process and network priority to ensure low latency audio forwarding even on highly loaded servers. Allow mumble-server to use higher priority?"

I expect that the file permissions issue concerning reading the SSL cert(s) isn't Debian-specific, though it's worth understanding what's going on there either way. This is going to become more of an issue soon because Let's Encrypt is going to start supporting wildcard SSL certs next month, as wildcard certs will need to be sharable between a number of daemons. https://letsencrypt.org/2017/07/06/wildcard-certificates-coming-jan-2018.html

[somniumAeternam]

Since the last update on openSUSE Tumbleweed for the mumble-server 1.2.19-4, I have the same issue that I cannot read the certificates of Lets Encrypt. These certificates are working for all other services (webserver, prosody etc.) but not for mumble-server anymore. On openSUSE Tumbleweed there is no option "MURMUR_USE_CAPABILITIES". And even if I start as root and increase the permissions (rwx), the the server fails to read the certificates. What can I do now? I'm forced to use a self-signed certificate.

[andrewgdunn]

I've just been bit by this as well in debian

[john-penntech]

Having the same issue in Ubuntu 18.04.1 LTS

[RenHWall]

I just migrated my mumble server to a new system and ran into this problem again. Old system: Ubuntu 16.04, mumble-server 1.2.12-1 New system: Ubuntu 18.04, mumble-server 1.3.0-1 On both mumble cannot load certificates unless allow mumble server to use higher priority has been set with dpkg-reconfigure. This is regardless of what permissions the mumble-server user has on the folders containing the keys. IMO this is not just a bug on Debian, but a bug on Ubuntu as well. There just happens to be an easier workaround on Ubuntu.

[bobonov]

I can confirm same issue on ubuntu server 20.04 and mumble server 1.3.0 It fails to load certbot certificate

[Krzmbrzl]

@bobonov could you please list the permissions for the certificate file? And could you also check that all directories in the path of the certificate file are accessible (have the x flag for the respective user/group).

As which user (and in what group) is murmur running? I assume it's a user separate from your default user account?

[bobonov]

Sorry for the late answer: certificate are letsencrypt one and are generated with certbot. Since I use nginx, at installation time I used the following option

certbot --nginx

certificate are working properly in nginx.

in mumble-server.ini I configured the following:

sslCert=/etc/letsencrypt/live/my.domain/fullchain.pem sslKey=/etc/letsencrypt/live/my.domain/privkey.pem

It start as root, (service at boot time), then it switch to default user uname=mumble-server ps -axu | grep mumble mumble-+ 1024 0.0 0.1 154784 10316 ? Sl mag02 0:03 /usr/sbin/murmurd -ini /etc/mumble-server.ini

Here follow the ls -la /etc/letsencrypt/live/my.domain/

lrwxrwxrwx 1 root root   43 apr 30 14:37 cert.pem -> ../../archive/my.domain/cert3.pem
lrwxrwxrwx 1 root root   44 apr 30 14:37 chain.pem -> ../../archive/my.domain/chain3.pem
lrwxrwxrwx 1 root root   48 apr 30 14:37 fullchain.pem -> ../../archive/my.domain/fullchain3.pem
lrwxrwxrwx 1 root root   46 apr 30 14:37 privkey.pem -> ../../archive/my.domain/privkey3.pem

These are symlink to the files which permission are:

ls -la /etc/letsencrypt/archive/my.domain/

-rw-r--r-- 1 root root 1923 apr 30 14:37 cert3.pem
-rw-r--r-- 1 root root 1647 apr 30 14:37 chain3.pem
-rw-r--r-- 1 root root 3570 apr 30 14:37 fullchain3.pem
-rw-r--r-- 1 root root 1704 apr 30 14:37 privkey3.pem

Files currently in use are the *3.pem, certbot always add the new certificate incrementally at renew times and symlink to the latest one.

Probably it should read the certificate before switching the user.

[Krzmbrzl]

@bobonov Thanks for the input. I assume you are also getting the "Failed to read ..." error message in the log?

To me your file permissions look correct, but on the other hand our code also looks correct: https://github.com/mumble-voip/mumble/blob/5be2bb965faa5e0b6c691d8dc3a6f2692355c97e/src/murmur/Meta.cpp#L477-L484 Maybe the opening of read-only files doesn't work well in Qt making this a Qt bug?

Are you using the static server binary or are you using a dynamically linked (I think if you installed it from the repos, it should be dynamic). In the latter case what's the output of qmake --version?

[streaps]

Private keys shouldn't be readable by everyone. Some applications refuse to use certs that have read permissions for others. I don't think this is the problem here, but with 644 permissions you could also make mumble-server (or murmur) the owner.

[RenHWall]

@streaps The private keys are not really world readable since the parent folder has restricted permissions, this is the default behavior of certbot. Here is a more complete example from my Ubuntu 18.04 server.

~$ sudo ls -l /etc/letsencrypt/
total 40
drwx------ 4 root root     4096 Dec 22 02:09 accounts
drwx------ 3 root root     4096 Dec 22 02:08 archive
-rw-r--r-- 1 root root      121 Feb 10  2019 cli.ini
drwxr-xr-x 2 root root     4096 Apr 20 08:10 csr
drwx------ 2 root root     4096 Apr 20 08:10 keys
drwxr-x--- 3 root ssl-cert 4096 Dec 22 02:08 live
-rw-r--r-- 1 root root     1143 Dec 22 02:08 options-ssl-nginx.conf
drwxr-xr-x 2 root root     4096 Apr 20 08:10 renewal
drwxr-xr-x 5 root root     4096 Dec 22 02:08 renewal-hooks
-rw-r--r-- 1 root root      424 Dec 22 02:08 ssl-dhparams.pem

~$ sudo ls -l /etc/letsencrypt/live/
total 8
drwxr-xr-x 2 root root 4096 Apr 20 08:10 my.domain.name
-rw-r--r-- 1 root root  740 Dec 22 02:08 README

~$ sudo ls -l /etc/letsencrypt/live/my.domain.name
total 4
lrwxrwxrwx 1 root root  34 Apr 20 08:10 cert.pem -> ../../archive/my.domain.name/cert3.pem
lrwxrwxrwx 1 root root  35 Apr 20 08:10 chain.pem -> ../../archive/my.domain.name/chain3.pem
lrwxrwxrwx 1 root root  39 Apr 20 08:10 fullchain.pem -> ../../archive/my.domain.name/fullchain3.pem
lrwxrwxrwx 1 root root  37 Apr 20 08:10 privkey.pem -> ../../archive/my.domain.name/privkey3.pem
-rw-r--r-- 1 root root 692 Dec 22 02:08 README

The mumble-server user is a member of the ssl-cert group and so has permissions to read the folder, and as such should be able to load the certs. Edit: I forgot how symlink permissions work.

[streaps]

@RenHWall

In your example the /etc/letsencrypt/archive directory has "drwx------" permissions and can only be accessed by root.

The files in /etc/letsencrypt/live/my.domain.name are only symlinks that point to the archive directory where the cert files are.

Try to make archive directory and the files inside readable for the ssl-cert group.

[RenHWall]

Thanks to @streaps for pointing out my misunderstanding symlink permissions mumble is now correctly reading the certificates without starting as root. Since new files are placed in /etc/letsencrypt/archive/my.domain.name/ every time the certificate is updated rather than updating the existing files the permissions of the new certificate needs to be changed as well. To accomplish this I have edited /etc/letsencrypt/renewal-hooks/deploy/mumble-restart.sh to include the following.

#!/bin/sh
#Update permissions on new certificates so the ssl-cert group (mumble-server) can read them.
chmod 640 /etc/letsencrypt/archive/my.domain.name/privkey*
chgrp -R ssl-cert /etc/letsencrypt/archive/

# Actually restart mumble
systemctl restart mumble-server.service

[Krzmbrzl]

@RenHWall that's great to hear. Also thanks for sharing your solution :)

@bobonov any chance you are experiencing the same problem? :thinking:

[TredwellGit]

On Ubuntu with Certbot you can:

sudo chmod 755 /etc/letsencrypt/{archive,live}/
sudo chgrp mumble-server -- /etc/letsencrypt/archive/mumble.example.net/privkey*.pem
sudo chmod 640 -- /etc/letsencrypt/archive/mumble.example.net/privkey*.pem

In /etc/mumble-server.ini set:

sslCert=/etc/letsencrypt/live/mumble.example.net/fullchain.pem
sslKey=/etc/letsencrypt/live/mumble.example.net/privkey.pem

In /etc/letsencrypt/renewal-hooks/deploy/mumble.example.net:

#!/usr/bin/env sh
exec pkill -USR1 -F /run/mumble-server/mumble-server.pid

This fixes the overly restrictive default Certbot permissions scheme, uses live reloading of certificates so there is no need to restart the server, and does not require the Mumble server to start as root. The group permissions from the previous privkey file will be used on new renewed privkey files created by Certbot.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information. With only the information that is currently in the issue, we don't have enough information to take action.

Please reach out if you have or find the answers we need so that we can investigate further (or if you feel like this issue shouldn't be closed for another reason).

[neobrain]

I ran into a similar issue on openSUSE, where I couldn't even get murmur to read the privkey.pem when (for testing only) running murmur as root and chmodding the files to 777, so there was another layer to the problem here. I've been able to solve the problem and since this report pops up in websearch queries, I figured I'd share my (very distribution-specific) solution.

(As a general word of caution, note that you normally do not want to randomly chown or chmod sensitive files like private ssl keys, unless you know the risk.)

It turns out openSUSE applies an AppArmor profile (/etc/apparmor.d/usr.sbin.murmurd) to murmur that enables access to the SSL certificates. Unfortunately, it does not grant access to the private keys. Further, the service file launches murmurd as the user mumble-server instead of launching it as root and having murmur change to the lesser-privileged user itself, hence also failing to access the SSL files due to RWX flags.

To solve this, three things need to be done:

1. In /etc/apparmor.d/usr.sbin.murmurd, insert the line #include <abstractions/ssl_keys> below #include <abstractions/ssl_certs>.
2. In the same file, insert the two lines capability setgid, and capability setuid, after capability sys_resource,
3. In /usr/lib/systemd/system/mumble-server.service, change both the User and Group lines to root.

With those changes, Murmur is working just fine with my Let's Encrypt certificate! (btw if anyone wants to report those problems to the openSUSE package maintainers - feel free to since I probably won't get to doing so myself. I've spent way too much time debugging this already 😅)

(CC @somniumAeternam, although you probably found a different solution by now :) )

[Lagg]

I got tired of encountering this issue every time I do an update so instead I took advantage of the kinda-sorta new SIGUSR1 support that soft-reloads cert settings. My crontab presently looks like:

0 */12 * * * certbot renew --quiet && ~/murmur-cert-renew.sh

murmur-cert-renew.sh:

#!/bin/sh

MURMUR_CERTDIR="/var/lib/murmur"
CERTDIR="/etc/letsencrypt/live/lagg.me"

MURMUR_CERT="$MURMUR_CERTDIR/cert.pem"
MURMUR_KEY="$MURMUR_CERTDIR/key.pem"

CERT="$CERTDIR/fullchain.pem"
KEY="$CERTDIR/privkey.pem"

cp $CERT $MURMUR_CERT
cp $KEY $MURMUR_KEY

pkill -USR1 murmurd

The certs are otherwise expected perm flags and works fine with murmurd running as its own uid/gid. Kind of annoying to have sensitive data duplicated in this way but it's probably more predictable than playing musical chairs with runtime uids. Hope it helps someone and wasn't too obvious.

[oli-ver]

Thanks to @streaps for pointing out my misunderstanding symlink permissions mumble is now correctly reading the certificates without starting as root. Since new files are placed in /etc/letsencrypt/archive/my.domain.name/ every time the certificate is updated rather than updating the existing files the permissions of the new certificate needs to be changed as well. To accomplish this I have edited /etc/letsencrypt/renewal-hooks/deploy/mumble-restart.sh to include the following.

#!/bin/sh
#Update permissions on new certificates so the ssl-cert group (mumble-server) can read them.
chmod 640 /etc/letsencrypt/archive/my.domain.name/privkey*
chgrp -R ssl-cert /etc/letsencrypt/archive/

# Actually restart mumble
systemctl restart mumble-server.service

Thanks, this helped very much. I was struggling with the idea to change the letsencrypt directory settings for Mumble though and rather decided to copy the certificates of the domain I need to the mumble folder.

#!/bin/sh
#Make certificate available for Mumble
cp /etc/letsencrypt/live/my.domain.name/privkey.pem /var/lib/mumble-server/
cp /etc/letsencrypt/live/my.domain.name/fullchain.pem /var/lib/mumble-server/
chown -R mumble-server:mumble-server /var/lib/mumble-server/

# Restart mumble
systemctl restart mumble-server.service

And then in /etc/mumble-server.ini:

sslCert=/var/lib/mumble-server/fullchain.pem
sslKey=/var/lib/mumble-server/privkey.pem