search

mumble-voip / mumble

Mumble is an open-source, low-latency, high quality voice chat software.
https://www.mumble.info
Other
6.28k stars 1.11k forks source link

Packet loss with udp and ipv6 #4592

Closed valentin-claras closed 3 years ago

valentin-claras commented 3 years ago

Describe the bug

When using ipv6, voip fails quickly after udp mode is enable

Steps to Reproduce

In the mumble client I have 3 connection defined to the same server:

  1. using a domain name ⚠️

    long connection time, then I experienced some packet loss at some point.

  2. using an ipv4 address ✔️

    everything works fine.

  3. using an ipv6 adress ❌

    long connection time, sometime failing. Udp is disabled at start. When (if) udp mode is enabled again, then voip doesn't work.

Expected behavior

Each one of them should work the same.

Additional context

I must say that I'm lost on what's going on with ipv6 😕 . I'm by no mean a sysadmin, so I might surely be missing things, but I'm trying my best 😅.

In the mumble client, when I enable Force TCP mode in the network parameters, then everythin works fine. But I cannot expect from my users to know nor do this.

Here are all the information that might be relevant to this case.

Host

I setup host=0.0.0.0 :: in order to make sure both ipv4 and ipv6 where listened. Before this, ipv4 was not showed as listened by netcat, but was still working fine. I guess setting host= wouldn't change anything 🤷 

$ netstat -tulp | grep murmurd
tcp        0      0 0.0.0.0:64738           0.0.0.0:*               LISTEN      27881/murmurd
tcp6       0      0 [::]:64738              [::]:*                  LISTEN      27881/murmurd
udp        0      0 0.0.0.0:64738           0.0.0.0:*                           27881/murmurd
udp6       0      0 [::]:64738              [::]:*                              27881/murmurd

When setting host=0.0.0.0 only, people using the domain name as a method to connect still experience the long connection time, but does not experience the packet loss afterwards.

IPV4

When using a connection that does not allow ipv6, everything works fine using the domain name configuration.

Firewall

As far as I can tell there is no firewall on the server. Ufw is not installed and iptables list no rules

$ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Certificate

I have a let's encrypt certificate setup on the same domain name (and server) and used by another service, but I did not referenced it in the config file. I doubt this has any link to this issue, but still trying to give any information I have!

mumble-server.ini ; Murmur configuration file. ; ; General notes: ; * Settings in this file are default settings and many of them can be overridden ; with virtual server specific configuration via the Ice or DBus interface. ; * Due to the way this configuration file is read some rules have to be ; followed when specifying variable values (as in variable = value): ; * Make sure to quote the value when using commas in strings or passwords. ; NOT variable = super,secret BUT variable = "super,secret" ; * Make sure to escape special characters like '\' or '"' correctly ; NOT variable = """ BUT variable = "\"" ; NOT regex = \w* BUT regex = \\w* ; Path to database. If blank, will search for ; murmur.sqlite in default locations or create it if not found. database=/var/lib/mumble-server/mumble-server.sqlite ; Murmur defaults to using SQLite with its default rollback journal. ; In some situations, using SQLite's write-ahead log (WAL) can be ; advantageous. ; If you encounter slowdowns when moving between channels and similar ; operations, enabling the SQLite write-ahead log might help. ; ; To use SQLite's write-ahead log, set sqlite_wal to one of the following ; values: ; ; 0 - Use SQLite's default rollback journal. ; 1 - Use write-ahead log with synchronous=NORMAL. ; If Murmur crashes, the database will be in a consistent state, but ; the most recent changes might be lost if the operating system did ; not write them to disk yet. This option can improve Murmur's ; interactivity on busy servers, or servers with slow storage. ; 2 - Use write-ahead log with synchronous=FULL. ; All database writes are synchronized to disk when they are made. ; If Murmur crashes, the database will be include all completed writes. ;sqlite_wal=0 ; If you wish to use something other than SQLite, you'll need to set the name ; of the database above, and also uncomment the below. ; Sticking with SQLite is strongly recommended, as it's the most well tested ; and by far the fastest solution. ; ;dbDriver=QMYSQL ;dbUsername= ;dbPassword= ;dbHost= ;dbPort= ;dbPrefix=murmur_ ;dbOpts= ; Murmur defaults to not using D-Bus. If you wish to use dbus, which is one of the ; RPC methods available in Murmur, please specify so here. ; ;dbus=system ; Alternate D-Bus service name. Only use if you are running distinct ; murmurd processes connected to the same D-Bus daemon. ;dbusservice=net.sourceforge.mumble.murmur ; If you want to use ZeroC Ice to communicate with Murmur, you need ; to specify the endpoint to use. Since there is no authentication ; with ICE, you should only use it if you trust all the users who have ; shell access to your machine. ; Please see the ICE documentation on how to specify endpoints. #ice="tcp -h 127.0.0.1 -p 6502" ; Ice primarily uses local sockets. This means anyone who has a ; user account on your machine can connect to the Ice services. ; You can set a plaintext "secret" on the Ice connection, and ; any script attempting to access must then have this secret ; (as context with name "secret"). ; Access is split in read (look only) and write (modify) ; operations. Write access always includes read access, ; unless read is explicitly denied (see note below). ; ; Note that if this is uncommented and with empty content, ; access will be denied. ;icesecretread= icesecretwrite= ; If you want to expose Murmur's experimental gRPC API, you ; need to specify an address to bind on. ; Note: not all builds of Murmur support gRPC. If gRPC is not ; available, Murmur will warn you in its log output. ;grpc="127.0.0.1:50051" ; Specifying both a certificate and key file below will cause gRPC to use ; secured, TLS connections. ;grpccert="" ;grpckey="" ; How many login attempts do we tolerate from one IP ; inside a given timeframe before we ban the connection? ; Note that this is global (shared between all virtual servers), and that ; it counts both successfull and unsuccessfull connection attempts. ; Set either Attempts or Timeframe to 0 to disable. ;autobanAttempts = 10 ;autobanTimeframe = 120 ;autobanTime = 300 ; Specifies the file Murmur should log to. By default, Murmur ; logs to the file 'murmur.log'. If you leave this field blank ; on Unix-like systems, Murmur will force itself into foreground ; mode which logs to the console. logfile=/var/log/mumble-server/mumble-server.log ; If set, Murmur will write its process ID to this file ; when running in daemon mode (when the -fg flag is not ; specified on the command line). Only available on ; Unix-like systems. pidfile=/run/mumble-server/mumble-server.pid ; The below will be used as defaults for new configured servers. ; If you're just running one server (the default), it's easier to ; configure it here than through D-Bus or Ice. ; ; Welcome message sent to clients when they connect. ; If the welcome message is set to an empty string, ; no welcome message will be sent to clients. ;welcometext="
Welcome to this server running Murmur.
Enjoy your stay!
" welcometext="
Ahoy and welcome to the ****** mumble server!
" ; Port to bind TCP and UDP sockets to. port=64738 ; Specific IP or hostname to bind to. ; If this is left blank (default), Murmur will bind to all available addresses. ;host= host=0.0.0.0 :: ; Password to join server. ;serverpassword= serverpassword=******* ; Maximum bandwidth (in bits per second) clients are allowed ; to send speech at. bandwidth=72000 ; Maximum number of concurrent clients allowed. users=100 ; Per-user rate limiting ; ; These two settings allow to configure the per-user rate limiter for some ; command messages sent from the client to the server. The messageburst setting ; specifies an amount of messages which are allowed in short bursts. The ; messagelimit setting specifies the number of messages per second allowed over ; a longer period. If a user hits the rate limit, his packages are then ignored ; for some time. Both of these settings have a minimum of 1 as setting either to ; 0 could render the server unusable. messageburst=5 messagelimit=1 ; Respond to UDP ping packets. ; ; Setting to true exposes the current user count, the maximum user count, and ; the server's maximum bandwidth per client to unauthenticated users. In the ; Mumble client, this information is shown in the Connect dialog. allowping=true ; Amount of users with Opus support needed to force Opus usage, in percent. ; 0 = Always enable Opus, 100 = enable Opus if it's supported by all clients. ;opusthreshold=100 ; Maximum depth of channel nesting. Note that some databases like MySQL using ; InnoDB will fail when operating on deeply nested channels. ;channelnestinglimit=10 ; Maximum number of channels per server. 0 for unlimited. Note that an ; excessive number of channels will impact server performance ;channelcountlimit=1000 ; Regular expression used to validate channel names. ; (Note that you have to escape backslashes with \ ) ;channelname=[ \\-=\\w\\#\\[\\]\\{\\}\\(\\)\\@\\|]+ ; Regular expression used to validate user names. ; (Note that you have to escape backslashes with \ ) ;username=[-=\\w\\[\\]\\{\\}\\(\\)\\@\\|\\.]+ ; Maximum length of text messages in characters. 0 for no limit. ;textmessagelength=5000 ; Maximum length of text messages in characters, with image data. 0 for no limit. ;imagemessagelength=131072 ; Allow clients to use HTML in messages, user comments and channel descriptions? ;allowhtml=true ; Murmur retains the per-server log entries in an internal database which ; allows it to be accessed over D-Bus/ICE. ; How many days should such entries be kept? ; Set to 0 to keep forever, or -1 to disable logging to the DB. ;logdays=31 ; To enable public server registration, the serverpassword must be blank, and ; this must all be filled out. ; The password here is used to create a registry for the server name; subsequent ; updates will need the same password. Don't lose your password. ; The URL is your own website, and only set the registerHostname for static IP ; addresses. ; Only uncomment the 'registerName' parameter if you wish to give your "Root" channel a custom name. ; ;registerName=Mumble Server ;registerPassword=secret ;registerUrl=http://www.mumble.info/ ;registerHostname= registerName="******" ; If this option is enabled, the server will announce its presence via the ; bonjour service discovery protocol. To change the name announced by bonjour ; adjust the registerName variable. ; See http://developer.apple.com/networking/bonjour/index.html for more information ; about bonjour. ;bonjour=True ; If you have a proper SSL certificate, you can provide the filenames here. ; Otherwise, Murmur will create its own certificate automatically. ;sslCert= ;sslKey= ; The sslDHParams option allows you to specify a PEM-encoded file with ; Diffie-Hellman parameters, which will be used as the default Diffie- ; Hellman parameters for all virtual servers. ; ; Instead of pointing sslDHParams to a file, you can also use the option ; to specify a named set of Diffie-Hellman parameters for Murmur to use. ; Murmur comes bundled with the Diffie-Hellman parameters from RFC 7919. ; These parameters are available by using the following names: ; ; @ffdhe2048, @ffdhe3072, @ffdhe4096, @ffdhe6144, @ffdhe8192 ; ; By default, Murmur uses @ffdhe2048. ;sslDHParams=@ffdhe2048 ; The sslCiphers option chooses the cipher suites to make available for use ; in SSL/TLS. This option is server-wide, and cannot be set on a ; per-virtual-server basis. ; ; This option is specified using OpenSSL cipher list notation (see ; https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT). ; ; It is recommended that you try your cipher string using 'openssl ciphers ' ; before setting it here, to get a feel for which cipher suites you will get. ; ; After setting this option, it is recommend that you inspect your Murmur log ; to ensure that Murmur is using the cipher suites that you expected it to. ; ; Note: Changing this option may impact the backwards compatibility of your ; Murmur server, and can remove the ability for older Mumble clients to be able ; to connect to it. ;sslCiphers=EECDH+AESGCM:EDH+aRSA+AESGCM:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA ; If Murmur is started as root, which user should it switch to? ; This option is ignored if Murmur isn't started with root privileges. uname=mumble-server ; If this options is enabled, only clients which have a certificate are allowed ; to connect. ;certrequired=False ; If enabled, clients are sent information about the servers version and operating ; system. ;sendversion=True ; This sets password hash storage to legacy mode (1.2.4 and before) ; (Note that setting this to true is insecure and should not be used unless absolutely necessary) ;legacyPasswordHash=false ; By default a strong amount of PBKDF2 iterations are chosen automatically. If >0 this setting ; overrides the automatic benchmark and forces a specific number of iterations. ; (Note that you should only change this value if you know what you are doing) ;kdfIterations=-1 ; You can configure any of the configuration options for Ice here. We recommend ; leave the defaults as they are. ; Please note that this section has to be last in the configuration file. ; [Ice] Ice.Warn.UnknownProperties=1 Ice.MessageSizeMax=65536 =
Krzmbrzl commented 3 years ago

Have you tried verifying that your server properly supports UDP connections via IPv6 with external tools? (E.g. https://www.thegeekdiary.com/how-to-test-porttcp-udp-connectivity-from-a-linux-server/)

And what is the version of the used server and client?

valentin-claras commented 3 years ago

At first I felt stupid because connection on either tcp and udp was refused with ipv6 😰, but that was because I temporarily switch back to host=0.0.0.0 only 😆

With listening enable on both, via host=0.0.0.0 ::, I can successfully connect through ipv6 on both tcp and udp to the server (tested from the server itself, and on local computer).

I'm using default debian 10 installation for the server, which means 1.3.0 ; and latest client: 1.3.3.

Krzmbrzl commented 3 years ago

Could you perhaps share the address of the server, so I could have a look myself? That'd probably be easier than doing "remote debugging" via text messages :sweat_smile:

TredwellGit commented 3 years ago

I have used Mumble via IPv6 for years without issue. Most weird IPv6 problems are because all of ICMPv6 was blocked erroneously in an attempt to block just ping. Also, do not use netstat or any other net-tools; they were obsoleted by iproute2 a decade ago.

valentin-claras commented 3 years ago

Could you perhaps share the address of the server, so I could have a look myself? That'd probably be easier than doing "remote debugging" via text messages

Sure! I've disabled the password requirement. The address is zaedyus-irae.com, with default port used.

Krzmbrzl commented 3 years ago

Okay that's interesting. I tried connecting to your server using the 3 methods you have described. Using the domain name directly is no issue at all. It connects instantly and UDP seems to stay intact (though I didn't linger for too long on your server xD). Same applies for connecting via IPv4.

When attempting to use IPv6 though it says that the network is unreachable.

I obtained the IP addresses like this:

$ host zaedyus-irae.com
zaedyus-irae.com has address 164.132.229.46
zaedyus-irae.com has IPv6 address 2001:41d0:401:3200::627

and ping yields the same result for the IPv6 address:

$ ping 2001:41d0:401:3200::627
ping: connect: Network is unreachable

So currently it seems that your server is not reachable through IPv6 at all (or has a wrong IPv6 address configured) :thinking:

EDIT: I just noticed that I generally can't ping any IPv6 address from my machine, so there's currently some sort of problem on my side. I'll have to investigate :eyes:

Johni0702 commented 3 years ago

There is definitely something odd going on with IPv6 routing/filtering here.

I've tried from three machines (one local, two servers from one provider), all of which can connect via IPv6+TCP (tested with openssl s_client -connect [2001:41d0:401:3200::627]:64738) but for UPD only one of them (named s below) gets through while the other two get no reply (tested using this script modified to do IPv6 by replacing AF_INET with AF_INET6).

Traceroute shows the same result: only that one server gets through.

UPD over IPv4 also works as expected for all of them.

UDP queries ``` $ python2 ./mumble-ping.py 2001:41d0:401:3200::627 64738 1606140907:NaN:NaN $ ssh s "python2 ./mumble-ping.py 2001:41d0:401:3200::627 64738" recvd 24 bytes Version 1.3.0, 1/100 Users, 12.4ms, 72kbit/s $ ssh s1 "python2 ./mumble-ping.py 2001:41d0:401:3200::627 64738" 1606140211:NaN:NaN ```
Traceroute ``` [user@ssh-hub ~]$ mtr -r -c10 2001:41d0:401:3200::627 Start: 2020-11-23T15:08:10+0100 HOST: home Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway 10.0% 10 0.2 0.5 0.2 2.7 0.8 2.|-- fd09:24ef:4179::a89:4 10.0% 10 0.4 0.4 0.3 0.5 0.1 3.|-- 2003:0:8900:c800::1 10.0% 10 5.5 7.5 4.1 27.8 7.7 4.|-- 2003:0:8900:c800::1 10.0% 10 4.0 11.1 3.6 64.4 20.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- 2001:41d0:aaaa:100::5 10.0% 10 26.1 25.6 15.7 34.2 5.7 9.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 10.|-- be101.sbg-g1-nc5.fr.eu 30.0% 10 12.6 13.1 12.6 14.5 0.7 11.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 $ ssh s "mtr -r -c10 2001:41d0:401:3200::627" Start: 2020-11-23T15:08:38+0100 HOST: s Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a03:4000:27::3 0.0% 10 0.8 1.1 0.6 2.5 0.5 2.|-- 2a00:11c0:47:3::20 0.0% 10 0.5 1.1 0.5 2.5 0.7 3.|-- 2a00:11c0:47:1:47::141 0.0% 10 3.8 3.9 3.7 4.1 0.1 4.|-- decix2.routers.ovh.net 30.0% 10 11.3 5.7 4.6 11.3 2.5 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- 2001:41d0:aaaa:100::5 0.0% 10 17.4 18.8 12.7 28.0 4.1 7.|-- 2001:41d0:aaaa:100::4 60.0% 10 5.0 4.9 4.6 5.2 0.2 8.|-- be101.sbg-g1-nc5.fr.eu 90.0% 10 187.4 187.4 187.4 187.4 0.0 9.|-- vl100.sbg-d1-a75.fr.eu 0.0% 10 7.0 6.9 6.6 7.1 0.2 10.|-- be5.sbg-z2g2-a75.fr.eu 0.0% 10 7.1 7.0 6.7 8.1 0.4 11.|-- po7.sbg-z2c2-a70.fr.eu 0.0% 10 6.8 6.8 6.6 6.9 0.1 12.|-- 2001:41d0:401:c2::25 0.0% 10 7.0 6.7 6.5 7.1 0.2 13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 14.|-- 2001:41d0:401:3200::627 0.0% 10 7.0 7.0 6.8 7.1 0.1 $ ssh s1 "mtr -r -c10 2001:41d0:401:3200::627" Start: 2020-11-23T14:55:52+0100 HOST: s1 Loss% Snt Last Avg Best Wrst StDev 1.|-- 2a03:4000:3b::2 0.0% 10 0.7 5.4 0.3 49.9 15.6 2.|-- 2a03:4000:ff00:1:: 0.0% 10 0.3 0.4 0.3 0.8 0.2 3.|-- 2a00:11c0:47:3::20 0.0% 10 0.4 0.6 0.4 1.0 0.2 4.|-- 2a00:11c0:47:1:47::141 0.0% 10 3.6 4.0 3.5 5.7 0.7 5.|-- decix2.routers.ovh.net 10.0% 10 5.3 7.2 4.4 13.5 3.6 6.|-- 2001:41d0::2579 70.0% 10 4.4 4.5 4.4 4.8 0.3 7.|-- 2001:41d0:aaaa:100::3 0.0% 10 27.3 18.1 11.6 27.3 6.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 9.|-- be101.sbg-g1-nc5.fr.eu 60.0% 10 11.9 9.6 7.5 11.9 2.2 10.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 ```
valentin-claras commented 3 years ago

Just letting you know: I've disabled the ipv6 redirection in my domain name configuration, resolving all my issues, and opened a ticket with my server provider. I highly doubt this issue comes from mumble, so I'm closing it :)

Krzmbrzl commented 3 years ago

Just now that I finally managed to get IPv6 to work on my machine xDD

But yeah, given that I know of several people who are using Mumble with IPv6 with seemingly no issues, I also think that the issue is probably specific to your network (setup) :+1: