Unable to connect Matrix Synapse voice calls via Coturn

[xthursdayx]

Hi, I'm using this docker image as part of the matrix-docker-ansible-deploy stack and am having difficulty getting it working for voice (or video) calls. This docker container is running in an Ubuntu Server VM along with Synapse. When a (locally registered) user calls from outside of the local network to another user on the same local network as the VM the call rings but never fully connects after being answered. This also happens if the user on the LAN calls the external user. However, if both users are on the same LAN (the same LAN as the Coturn/Synapse VM) then both voice and video calls work.

My network is behind a pfSense firewall, but I have the ports 3478(tcp/udp), 5349 (tcp/udp), and 49152-49172 (udp) forwarded to the VM directly and outbound traffic from the VM is set to use static ports.

This is my turnserver.conf

use-auth-secret
static-auth-secret=<secret-pass>
realm=my.domain

listening-port=3478
tls-listening-port=5349
min-port=49152
max-port=49172
external-ip=<my-external-ip>

log-file=stdout
pidfile=/var/tmp/turnserver.pid
userdb=/var/tmp/turnserver.db

no-cli

cert=/matrix/ssl/config/live/my.domain/fullchain.pem
pkey=/matrix/ssl/config/live/my.domain/privkey.pem

prod
no-tcp-relay

user-quota=12
total-quota=1200

denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=192.168.0.0-192.168.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
allowed-peer-ip=10.0.0.1

In my Synapse homserver.yaml Coturn is set to allow guests (at the moment at least), the shared-auth-secret is set, and the turn-uris are:

- turns:my.domain?transport=udp
- turns:my.domain?transport=tcp
- turn:my.domain?transport=udp
- turn:my.domain?transport=tcp

I've been trying to figure this out for a couple of weeks now but haven't had any luck even figuring out where the problem is originating. When I run journalctl -fu matrix-coturn the logs don't show anything except for the creation of the turn servers and connection to the SQlite database.

Any ideas about how I can continue trouble shooting this, or what might be going wrong?

[tyranron]

@xthursdayx debugging this part is always tricky and some sort of luck. Some tips that helped us (we're dealing with Kurento):

1. Running Docker image in host network.
2. We have tried using turns: but without success, so we just switched it off as our WebRTC traffic is encrypted already. We also use stun:. Here is an example:

   {
      "ice_servers":[
         {
            "urls":[
               "stun:my.domain:9934"
            ]
         },
         {
            "urls":[
               "turn:my.domain:9934",
               "turn:my.domain:9934?transport=tcp"
            ],
            "username":"<static-user>",
            "credential":"<static-path>"
         }
      ]
   }

3. We don't restrict with min-port/max-port as that worked much worse in our case (we didn't figure out why).
4. Verbose logging CLI flags:

   -v, --verbose 'Moderate' verbose mode.
   -V, --Verbose 'Extra' verbose mode, very annoying and not recommended.

[xthursdayx]

Thanks for the advice @tyranron

Are you using 9934 in place of the default 3478?

Also, did you find no way to use a static-auth-secret instead of of a static user/pw combo?

[tyranron]

@xthursdayx

Are you using 9934 in place of the default 3478?

Yes, but even when we're using default one, we specify port explicitly. Actually, this is just a part of our Helm chart templating.

Also, did you find no way to use a static-auth-secret instead of of a static user/pw combo?

No, we don't need it in our case, actually. Firstly, we had used static user/pass, but then switched to dynamic credentials backed by Redis (media server generates required credentials, puts them into Redis, forms ICE servers and returns them to user, then user connects and Coturn finds required credentials in Redis).

[xthursdayx]

Okay, I'll keep trying things.

After running turnutils_uclient I realized that my static-auth-secret may have been causing problems because it contained a symbol. I switched it to an alphanumeric password and now I'm getting these messages in my coturn log:

522: check_stun_auth: Cannot find credentials of user <1573310683:@zandrsn:my.domain>
Nov 08 15:29:18 matrix matrix-coturn[30928]: 522: ERROR: check_stun_auth: Cannot find credentials of user <1573310683:@zandrsn:my.domain>
Nov 08 15:29:18 matrix matrix-coturn[30928]: 522: check_stun_auth: Cannot find credentials of user <1573310683:@zandrsn:my.domain>

This seems to indicate that there is a problem with Matrix and Coturn exchanging the user credentials. From what I understand, when you use a static-auth-secret the user name should be a combo of "timestamp:userid", but this doesn't seem to be happening (if I'm interpreting this message correctly).

[tyranron]

@xthursdayx hmm... I'm missing the lt-cred-mech in your configuration, which is vital for WebRTC, did you try it?

[xthursdayx]

I had it in there, but according to the Coturn documentation the use of static-auth-secret negates the need to include lt-cred-mech in the conf. Here is info from the default turnserver.conf example:

Be aware that use-auth-secret overrides some parts of lt-cred-mech.
The use-auth-secret feature depends internally on lt-cred-mech, so if you set
this option then it automatically enables lt-cred-mech internally
as if you had enabled both.

Note that you can use only one auth mechanism at the same time! This is because, 
both mechanisms conduct username and password validation in different ways.

Use either lt-cred-mech or use-auth-secret in the conf
to avoid any confusion.

[xthursdayx]

So, I was able to get voice and video calls working with my Matrix/Riot VM by running this coturn docker image on the host machine on the host network (instead of a bridge, br0, or whatever), as you suggested. I then added the correct turn uris to my matrix homeserver.yaml:

matrix_synapse_turn_uris: |
  {{
    [
      'turns:my.domain?transport=udp',
      'turns:my.domain?transport=tcp',
      'turn:my.domain?transport=udp',
      'turn:my.domain?transport=tcp'
    ]
  }}

My turnserver.conf looks like this:

listening-port=3478
tls-listening-port=5349

external-ip=<my_external_ip>

min-port=49152
max-port=49172

use-auth-secret
static-auth-secret=<share-secret>

userdb=/var/lib/coturn/turndb

realm=my.domain

user-quota=12
total-quota=1200

no-tcp-relay

cert=/etc/ssl/fullchain.pem
pkey=/etc/ssl/privkey.pem

stdout

denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=192.168.0.0-192.168.255.255
denied-peer-ip=172.16.0.0-172.31.255.255

allowed-peer-ip=10.0.0.1

pidfile="/var/run/turnserver.pid"

no-cli

and /etc/ssl is mapped to the location of my LetsEncrypt certificates and keys on the host machine.

The base web domain for matrix is hosted on the host web server and connections to matrix are accepted via an NGINX reverse proxy from port 443 WAN to port 8008 on the VM. Port 8448 is forwarded directly to the VM for federation and ports 3478, 5349, and 49152:49172 are forwarded to the web server where this Coturn docker container is running and accepting connections on those ports.

[Sudheerkumar]

Hi @xthursdayx are you running it on a bear metal kubernetes or any of the PAAS offerings like AKS?

1. We are using AKS to host out CoTurn server. We enabled hostNetwork:true but still unable to establish connection with the server. Are we missing something ?

2. We kept the clsuter piece aside as we are unable to proceed with the AKS, Now when we are running the coturn docker image on a azure ubuntu VM and tried static-auth-secret and use turnutils_uclient it always gives 401 unauthorised,(we are following the TURN REST API standard to get the username and password) we are currently hardcoding the secret used to encrypt pwd instead of getting from the DB. Are we missing something?

[xthursdayx]

@Sudheerkumar I'm running the container as a normal docker container not kubernetes or AKS. Sorry not to be able to help more...