Closed r4dh4l closed 2 years ago
Krzmbrzl
commented
2 years ago Have you ever started the new server while it could not access the Let's Encrypt cert and thus generated a self-signed one instead during startup? In that case I think it could happen that it remembers this certificate and keeps using that... That's just a rather wild guess though as I am not too familiar with that part of the code base.
Krzmbrzl
commented
2 years ago A quick test you could perform: Rename your database so that the server creates a fresh one. If it still doesn't pick up the LE cert, then there's a config issue at hand. Otherwise the self-signed cert is stored somewhere in the DB and the server apparently prefers to use that.
pa5tlife
commented
2 years ago I'm having this problem too I'm not sure where or what to write over for the database.
Can anyone help me with a permission issue I'm having on the server with mumble-server application?
If you go here you can see the detail of the solution that mumbles gives for this, but I dont know how to do it and I dont know if changing the permissions is a good idea. https://wiki.mumble.info/wiki/Obtaining_a_Let's_Encrypt_Murmur_Certificate
<C>[date time] Failed to read /etc/letsencrypt/live/mumble.example.org/cert.pem
I tried to reconfigure the mumble-server with the correct path to the fullchain.pem and to the private key, but then I get this error:
2022-05-26 15:47:23.733 MetaParams: Failed to read /etc/letsencrypt/live/domain/fullchain.pem 2022-05-26 15:47:23.733 MetaParams: Failed to load SSL settings. See previous errors.
I even requested a new certificate for my subdomain in order to try to avoid this because that worked on another system (ubuntu jammy with certbot from snap).
Thanks so much for your time
pa5tlife
commented
2 years ago Also when I run this command: sudo murmurd -v -fg
I get this output:
<W>2022-05-26 16:49:32.072 1 => Server: TCP Listen on [IP]:64738 failed: The bound address is already in use
<W>2022-05-26 16:49:32.073 1 => Server: TCP Listen on IP:64738 failed: The bound address is already in useWhat does that mean : "failed: The bound address is already in use" ?
pa5tlife
commented
2 years ago For anyone who is working on solving this issue, please see some help I got at the Let's Encrypt Community forum: https://community.letsencrypt.org/t/moving-to-new-server-copying-everything-by-hand-but-sites-unavailable/178408/6
They suggested that I create a deploy hook to:
The process was a bit confusing. In the end I have the same result. It cant read the cert.pem or fullchain.pem I tried both.
I really shouldnt have moved to Debian 11, but I wanted to so bad. I'm going to have to de-list from the mumble public servers list until I figure out how to get the SSL certificate working and showing my server name in the green on the client again.
Krzmbrzl
commented
2 years ago What does that mean : "failed: The bound address is already in use" ?
Well it essentially means what it says: The respective ports are already being used by a different application so the server can't bind to them. Most likely this means that you have another instance of a Mumble server running on your system that is still blocking these ports.
A failure to read the cert is usually caused by wrong permissions on the certificate itself or on one of its parent-directories. What user do you start the server as and what are the exact permissions for the certificate and all of its parent directories?
r4dh4l
commented
2 years ago @Krzmbrzl
Have you ever started the new server while it could not access the Let's Encrypt cert and thus generated a self-signed one instead during startup? In that case I think it could happen that it remembers this certificate and keeps using that... That's just a rather wild guess though as I am not too familiar with that part of the code base.
I don't think so. Even not during the Debian upgrade process (because I've stopped the mumble service before upgrading).
A quick test you could perform: Rename your database so that the server creates a fresh one. If it still doesn't pick up the LE cert, then there's a config issue at hand. Otherwise the self-signed cert is stored somewhere in the DB and the server apparently prefers to use that.
Thank you. Done:
mumble-server@vm-mumble:~$ rm mumble-server.sqlite_bkp2022-05-27
mumble-server@vm-mumble:~$ mv mumble-server.sqlite mumble-server.sqlite_bkp2022-05-27
mumble-server@vm-mumble:~$ ls -la
total 120
drwxr-x--- 3 mumble-server mumble-server 4096 May 27 10:12 .
drwxr-xr-x 48 root root 4096 May 24 21:21 ..
drwxrwx--- 2 mumble-server mumble-server 4096 May 27 10:07 .config
-rw-r----- 1 mumble-server mumble-server 110592 May 27 10:08 mumble-server.sqlite_bkp2022-05-27
mumble-server@vm-mumble:~$
root@vm-mumble:~# murmurd -v -fg
<X>2022-05-27 10:12:41.273 SSL: OpenSSL version is 'OpenSSL 1.1.1n 15 Mar 2022'
<W>2022-05-27 10:12:41.274 Initializing settings from /etc/mumble-server.ini (basepath /etc)
<W>2022-05-27 10:12:42.151 MetaParams: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA"
<C>2022-05-27 10:12:42.154 Successfully switched to uid 118
<W>2022-05-27 10:12:42.213 ServerDB: Opened SQLite database /var/lib/mumble-server/mumble-server.sqlite
<W>2022-05-27 10:12:42.213 ServerDB: Using SQLite's default rollback journal.
<W>2022-05-27 10:12:43.036 Performed initial PBKDF2 benchmark. Will use 8000 iterations as default
<W>2022-05-27 10:12:43.037 Generating new tables...
<W>2022-05-27 10:12:43.152 Resource limits were 0 0
<W>2022-05-27 10:12:43.152 Successfully dropped capabilities
<W>2022-05-27 10:12:43.262 Murmur 1.3.4 (1.3.4-1) running on X11: Debian GNU/Linux 11 (bullseye): Booting servers
<W>2022-05-27 10:12:43.437 1 => Password for 'SuperUser' set to '*****'
<W>2022-05-27 10:12:43.854 1 => Server listening on 0.0.0.0:64738
<W>2022-05-27 10:12:44.818 1 => Generating new server certificate.
<W>2022-05-27 10:12:45.262 1 => Announcing server via bonjour
<W>2022-05-27 10:12:45.334 1 => Not registering server as public
...
mumble-server@vm-mumble:~$ ls -la
total 228
drwxr-x--- 3 mumble-server mumble-server 4096 May 27 10:14 .
drwxr-xr-x 48 root root 4096 May 24 21:21 ..
drwxrwx--- 2 mumble-server mumble-server 4096 May 27 10:07 .config
-rw-r----- 1 mumble-server mumble-server 110592 May 27 10:14 mumble-server.sqlite
-rw-r----- 1 mumble-server mumble-server 110592 May 27 10:08 mumble-server.sqlite_bkp2022-05-27
mumble-server@vm-mumble:~$ Generating new server certificate. seems to indicate that is /etc/mumble-server.ini ignored, but why?
root@vm-mumble:~# ls -la /etc/mumble-server.ini
-rw-r----- 1 root mumble-server 17202 May 23 23:38 /etc/mumble-server.ini
root@vm-mumble:~#I've noticed there is a new option in 1.3.4:
; If your certificate is signed by an authority that uses a sub-signed or
; "intermediate" certificate, you probably need to bundle it with your
; certificate in order to get Murmur to accept it. You can either concatenate
; the two certificates into one file, or you can put it in a file by itself and
; put the path to that PEM-file in sslCA.
;sslCA=I've set it to
; If your certificate is signed by an authority that uses a sub-signed or
; "intermediate" certificate, you probably need to bundle it with your
; certificate in order to get Murmur to accept it. You can either concatenate
; the two certificates into one file, or you can put it in a file by itself and
; put the path to that PEM-file in sslCA.
;sslCA=
sslCA=/etc/letsencrypt/live/my.ddns.net/chain.pemstopped the server, deleted the databse, started the server but unfortunately still get Generating new server certificate and a certification warning in the client.
Krzmbrzl
commented
2 years ago Hm... Quite frankly I don't have any concrete ideas. I could only try to help you debug the issue further if you were willing and capable of compiling Mumble yourself and making a few small adjustments to its source code (add some logging statements).
@davidebeatrici do you have any idea?
davidebeatrici
commented
2 years ago I would check what the server is actually accessing using strace.
stale[bot]
commented
2 years ago This support-issue has been automatically marked as stale because it has not had recent activity. If no further activity occurs, the issue will be automatically closed as we'll assume your problem to be fixed.
ppfeufer
commented
2 years ago I faced a similar issue, where the wrong certificate was loaded. I had to delete these two rows from the murmur_config DB table. (Obviously, make a backup before doing so ...)
r4dh4l
commented
2 years ago @Krzmbrzl @davidebeatrici Thank you for your answer. Because I need the Mumble service I made a rollback of my Mumble-VM. Unfortunately the snapshot I made before uprading to Debian 11 had some kind of problem so the rollback made the VM unusable. The whole upgrade project was a desaster.
@ppfeufer If I manage to repair my Mumble-VM I will try your suggestion, thank you.
r4dh4l
commented
2 years ago After repairing my Mumble-VM I tried what @ppfeufer mentioned in https://github.com/mumble-voip/mumble/issues/5702#issuecomment-1152874838:
# systemctl stop mumble-server.service
# apt install sqlite3
# su - mumble-server -s /bin/bash
$ cp -p /var/lib/mumble-server/mumble-server.sqlite /var/lib/mumble-server/mumble-server.sqlite_bkp2022-07-06
$ sqlite3 mumble-server.sqlite
$ sqlite> .tables
acl channel_links group_members servers users
bans channels groups slog
channel_info config meta user_info
sqlite> So unfortunately there is no murmur_config db or did I understand something wrong?
ppfeufer
commented
2 years ago config in your case I guess ...
r4dh4l
commented
2 years ago Sorry, so obvious... however after
sqlite> .tables
acl channel_links group_members servers users
bans channels groups slog
channel_info config meta user_info
sqlite> SELECT * FROM config;
1|certificate|-----BEGIN CERTIFICATE-----
MIIDSzCCAjOgAwIBAgIBATANBgkqhkiG9w0BAQUFADAuMSwwKgYDVQQDDCNNdXJt
dXIgQXV0b2dlbmVyYXRlZCBDZXJ0aWZpY2F0ZSB2MjAeFw0xODA1MjEwMDEwMDda
Fw0zODA1MTYwMDEwMDdaMC4xLDAqBgNVBAMMI011cm11ciBBdXRvZ2VuZXJhdGVk
IENlcnRpZmljYXRlIHYyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
4IOyhSjVYwXqeAUz8qFQsY55g2T05mF6bH3JN8r+k2bqexbWm4Xc68Q/pzUqgvFF
dskXaJdU1n23mt+eXyuSw9bwfW5810UPxIo4t07j87AdOSUJ10QpUq/QZKhzEv1W
Sop6grncL54gljjnUjKUPVd7YQcwQY/v2rsWWeRAjdfIbE9QXC/Si0YP9alHAwPI
Tg0y/dbdx+NrZxBGvICxleq7GtOACI7SElcJM5zD/FaBFALG+OPLQHLL6vwAUn9f
cVUveIu+T84Beo7AKIqTWGOv5s14KSRnyoEOAkRb3jr9WTu8KjX64nz3FUTgX1u4
pBl4SF0naEp71Pgz9BqeBQIDAQABo3QwcjAMBgNVHRMBAf8EAjAAMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU6zuSXrzYWCkD0K3bd411
Vuo9hKQwJAYJYIZIAYb4QgENBBcWFUdlbmVyYXRlZCBmcm9tIG11cm11cjANBgkq
hkiG9w0BAQUFAAOCAQEA3L06xeMNvjxFPLO927wxo7wIwV5xYT5Xhfu4Jc84vAeV
eF2ehwufha2VhUwltMX9vIHpGUx1d+yySa3/Vc1VTeA/i1SOnGigwKz0Aw7KessN
rm0CCD0UFdi7MfA0zJ2MMwAaCDCgcYrDQFvhnekuYn9dbMT7BSiQ92wIXxhD5Qbp
hmtR6P086t1NPvG42/NCMoS3UC52v/+pccKN4hEQne6IrJnZ8UmToUn8DaNZOIFY
ydHg46z5LXoA4CsiRbz/R0KgXRrJZFtfx7GTD7qXWi24NTRKtLzf7puUfGRdnr7h
JCz96PwFax1yg4Ut5dxDbkkdn7YFdsM1lXXoj4ddsw==
-----END CERTIFICATE-----
1|key|-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA4IOyhSjVYwXqeAUz8qFQsY55g2T05mF6bH3JN8r+k2bqexbW
...
/OgcsN5OuhVM8iN1ySCb2gKRXeQ8gwFT+iSj2Kh+aSB3aKRDmo0YsmE=
-----END RSA PRIVATE KEY-----
sqlite> DELETE FROM config;
sqlite> SELECT * FROM config;
sqlite> .quit
# rebootI still get the the cer warning by the client which says using "Murmur Autogenerated Certificate v2". Did I forget something?
Edit: With the following command it is possible to delete the database certificate etntries without accessing the database:
murmurd -wipesslMake sure the mumble-server can read the certificate files. If it can't, it will probably generate its own again ...
Krzmbrzl
commented
2 years ago Indeed, that's the case
r4dh4l
commented
2 years ago But the user for the mumble-server has full access to the LE certificates (with same access rights as under Debian 10):
mumble-server@vm-mumble:~$ ls -la /etc/letsencrypt/live/my.ddns.net/
total 12
drwxrwx--- 2 root letsencrypt 4096 Mar 24 23:46 .
drwxrwx--- 3 root letsencrypt 4096 Nov 4 2019 ..
lrwxrwxrwx 1 root letsencrypt 38 Mar 24 23:45 cert.pem -> ../../archive/my.ddns.net/cert15.pem
lrwxrwxrwx 1 root letsencrypt 39 Mar 24 23:45 chain.pem -> ../../archive/my.ddns.net/chain15.pem
lrwxrwxrwx 1 root letsencrypt 43 Mar 24 23:45 fullchain.pem -> ../../archive/my.ddns.net/fullchain15.pem
lrwxrwxrwx 1 root letsencrypt 41 Mar 24 23:45 privkey.pem -> ../../archive/my.ddns.net/privkey15.pem
-rwxrwx--- 1 root letsencrypt 692 Nov 4 2019 README
mumble-server@vm-mumble:~$ getent group letsencrypt
letsencrypt:x:1002:sync-le,mumble-server
mumble-server@vm-mumble:~$ cat /etc/letsencrypt/live/my.ddns.net/
cert.pem chain.pem fullchain.pem privkey.pem README
mumble-server@vm-mumble:~$ cat /etc/letsencrypt/live/my.ddns.net/cert.pem
-----BEGIN CERTIFICATE-----
MIIFIDCCBAigAwIBAgISBEVeX6pId7ilgblDt5FlRmY8MA0GCSqGSIb3DQEBCwUA
...
5S6I4hZmA4t0gyk4IjGJJ+63Bok=
-----END CERTIFICATE-----
mumble-server@vm-mumble:~$ cat /etc/letsencrypt/live/my.ddns.net/chain.pem
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
...
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
...
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
mumble-server@vm-mumble:~$ cat /etc/letsencrypt/live/my.ddns.net/fullchain.pem
-----BEGIN CERTIFICATE-----
MIIFIDCCBAigAwIBAgISBEVeX6pId7ilgblDt5FlRmY8MA0GCSqGSIb3DQEBCwUA
...
5S6I4hZmA4t0gyk4IjGJJ+63Bok=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
...
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
...
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
mumble-server@vm-mumble:~$ cat /etc/letsencrypt/live/my.ddns.net/fullchain.pem
-----BEGIN CERTIFICATE-----
MIIFIDCCBAigAwIBAgISBEVeX6pId7ilgblDt5FlRmY8MA0GCSqGSIb3DQEBCwUA
...
5S6I4hZmA4t0gyk4IjGJJ+63Bok=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
...
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
...
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
mumble-server@vm-mumble:~$ cat /etc/letsencrypt/live/my.ddns.net/privkey.pem
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
mumble-server@vm-mumble:~$ So there is some change in the way mumble-server handles the certificates, isn't it?
Krzmbrzl
commented
2 years ago Not that I would be aware of
r4dh4l
commented
2 years ago I just tried a lot of other things:
MURMUR_USE_CAPABILITIES=0 to MURMUR_USE_CAPABILITIES=1 in /etc/default/mumble-server) -> no successSo maybe my SSL config in /etc/mumble-server.ini is wrong. Could someone confirm which is the right one?
sslCert=/etc/letsencrypt/live/example.com/cert.pem
sslKey=/etc/letsencrypt/live/example.com/privkey.pem
sslCA=/etc/letsencrypt/live/example.com/fullchain.pem(https://www.reddit.com/r/mumble/comments/3x55al/lets_encrypt_free_sslcertificate_for_your/)
or
sslCert=/etc/letsencrypt/live/example.com/fullchain.pem
sslKey=/etc/letsencrypt/live/example.com/privkey.pem
sslCA=/etc/letsencrypt/live/example.com/cert.pem(https://wiki.mumble.info/wiki/Obtaining_a_Let%27s_Encrypt_Murmur_Certificate)
?
ppfeufer
commented
2 years ago As far as I know, sslCA doesn't need to be set for LetsEncrypt certificates.
sslCert => fullchain.pemsslKey => privkey.pemsslCA would be the Certificate Authority, in this case, LetEncrypt itself (If I understand it correctly), so you can't really set it
r4dh4l
commented
2 years ago Okay, thank. Even if I only set
sslCert=/etc/letsencrypt/live/example.com/fullchain.pem
sslKey=/etc/letsencrypt/live/example.com/privkey.pemand delete the certificate entries in the database after stopping mumble-server and restarting it die Mumble Client complaints about self-signed certificates. I just noticed that the Mumble Client says not only
SSL Verification failed: The certificate is self-signed, and untrusted
but also
SSL Verification failed: The host name did not match any of the valid hosts for this certificate
Maybe this is the problem? The Lets Encrypt certificate I'm using for multiple virtual machines is, let's say, my.ddns.net but the hostname of the VM running the mumble server is vm-mumble. Is that really a problem?
ppfeufer
commented
2 years ago The Lets Encrypt certificate I'm using for multiple virtual machines is, let's say, my.ddns.net but the hostname of the VM running the mumble server is vm-mumble. Is that really a problem?
Yes, the host for the cert has to match the mumble's host. I mean, that's one of the ideas behind an SSL certificate, to verify the host ...
r4dh4l
commented
2 years ago Yes, the host for the cert has to match the mumble's host. I mean, that's one of the ideas behind an SSL certificate, to verify the host ...
I just checked the FQDN of my virtual machine:
# hostname -f
vm-mumble.my.ddns.netmy.ddns.net is the domain my LE certificate were created with. The ddns.net service does not allow to register third level domains (vm-mumble in my case) so the certbot command I used for creating my Lets Encrypt certificates was:
certbot certonly --webroot -w /var/www/certbot-webroot -d my.ddns.netUnder Debian10 the created certificates obviously worked as wildcard certificates so again: If I make a rollback to the Debian 10 snapshot of the Mumble server there wouldn't be a certificate warning in the Mumble client. Why this happens now? Even if hostname verification is one idea behind an SSL certificate I doubt this verification is something introduced with Debian 11, is it?
ppfeufer
commented
2 years ago I guess the right command would have been:
certbot certonly --webroot -w /var/www/certbot-webroot -d *.my.ddns.netThank you, I just tried your suggestion but get the error
Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA. You may need to use an authenticator plugin that can do challenges over DNS.
According to https://systemrequest.net/index.php/308/ I have to control the whole domain (my.ddns.net in my case) or provide all subdomains individually in the certbot request otherwise. If I try
certbot certonly --webroot -w /var/www/certbot-webroot -d my.ddns.net -d vm-mumble.my.ddns.netcertbot replies
Failed authorization procedure. vm-mumble.my.ddns.net (http-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for vm-mumble.my.ddns.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for vm-mumble.my.ddns.net - check that a DNS record exists for this domain
As commented above ddns.net doesn't allow me to set third level domains (vm-mumble.my.ddns.net). I use several virtual machines behind my.ddns.net and delegate requests for several services (data cloud, messenger etc.) using a virtual machine as reverse proxy. Mumble server is the only service that is directly accessed by a port forwarding (64738).
So it means that I can only use the mumble server with the LE certs under Debian 10 (for whatever reason) or self-signed certificates under Debian 11?
r4dh4l
commented
2 years ago Found the problem: It was totally not related to certificates or SSL settings but caused by a wrong semicolon in the welcometext section of the mumble config where I had something like this:
welcometext="<br />
<br />Welcome to ...
<br />...
<br />Etherpad needed?
;<br />-> <a href="https://pad.example.org/" style="color:green">pad.example.org</a>"
<br />-> <a href="https://pad.example2.org/" style="color:green">pad.example2.org</a>"Because of ;<br /> Mumble ignores the whole mumble-server.ini at all (and does not complain about it). Is this a bug to report separately?
davidebeatrici
commented
2 years ago Yes.
ppfeufer
commented
2 years ago ;<br />-> <a href="https://pad.example.org/" style="color:green">pad.example.org</a>"
<br />-> <a href="https://pad.example2.org/" style="color:green">pad.example2.org</a>"My money would be more on the " at the end of the first line, which probably makes the second line a syntax error in the config file ...
r4dh4l
commented
2 years ago
TLTR
No certificate problem but a config problem caused by a semicolon: https://github.com/mumble-voip/mumble/issues/5702#issuecomment-1186622287
The issue
After upgrading my Mumble VM from Debian 10 to Debian 11 the LetsEncrypt certificate is not used anymore so mumble clients report usage of a self-signed certificate.
Mumble server version:
I think my
/etc/mumble-server.iniis correct:The mumble-server user can access the LE directories:
(user
mumble-serveris part of the groupletsencrypt)Starting the mumble-server reports nothing unusual:
Same for the mumble client:
Any ideas why my server is not using the LE certificate?
Mumble version
1.3.4
Mumble component
Server
OS
Linux
Additional information
No response