Unable to connect to the voice call: timed out waiting for peer connection

davide-prade commented 1 year ago

Summary

Cannot start/join to call

Environment Information

Device Name: HUAWEI Mate 20 lite
OS Version: EMUI 12.0.0
Mattermost App Version: 2.4.0 (Build 6000465)
Mattermost Server Version: 7.9.2 (self-hosted)

Steps to reproduce

Open Mattermost app
Start call

Expected behavior

Calls started/joined successfully.

Observed behavior (that appears unintentional)

Cannot start/join the voice call. The following error appears: "Unable to connect to the voice call: timed out waiting for peer connection". Screenshot_20230516_121130

Possible fixes

No fix found.

amyblais commented 1 year ago

@streamer45 @cpoile Is this something we want to look at fixing today before v2.4 is officially released?

cpoile commented 1 year ago

@amyblais It's likely not related to the mobile release specifically, so no need for it to impact.

Hi @davide-prade , this error is seen when the mobile calls client cannot reach the Calls plugin or rtcd service, so it's a networking issue. You would need to debug the connection. For example: Are you able to connect to a call using the webapp or desktop? If so, is the mobile device on the exact same wifi as the webapp/desktop that works?

davide-prade commented 1 year ago

Hi @cpoile, voice calls work correctly with desktop app end/or web browser. I have tried the mobile app also on the same wifi connection and the error is the same.

cpoile commented 1 year ago

That's strange then, because it usually means that the peer connection cannot complete (usually networking, like udp 8443 is not reachable by the mobile app). Are you trying to connect using an account that is already connected to the call?

davide-prade commented 1 year ago

Yes, I have tried also to join a call opened by another user. The UDP port 8443 is open and reachable and other users have no problem to use webapp and/or desktop app.

cpoile commented 1 year ago

Yah, you definitely can't join a call that you're already in, so that would never work. Can you try starting a call from the mobile app? And just to make sure, you're not on cellular data, you're on the same wifi that the working webapp uses?

davide-prade commented 1 year ago

Yes, I have also tried to start a call from mobile app. I tried on the cellular data and also on the same wifi network. The error is the same.

cpoile commented 1 year ago

Are you able to check on a different Android or Apple phone?
Can you give details of the networking setup for your MM server? (firewalls, load balancers, etc.)

davide-prade commented 1 year ago

Are you able to check on a different Android or Apple phone?

Today I tried again with a colleague of mine at work with desktop app and he cannot initiate a call, while I can. Last days it was the other way around. On the same wifi network I can establish a call, but only from the webapp and/or desktop app. At the moment I have no other mobile devices on which to try.

2. Can you give details of the networking setup for your MM server? (firewalls, load balancers, etc.)

The version of Mattemost is the one included in the Docker image of GitLab CE.

version: '3.7'

services:
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: '.........'
    container_name: gitlab-ce
    environment:
      GITLAB_OMNIBUS_CONFIG: |
#       .........
        mattermost_external_url 'https://mattermost.anydata.it'
#       .........
    ports:
#       .........
      - '8443:8443/udp'
#       .........
    volumes:
#       .........
    networks:
      - 'gitlab'

#.........

networks:
  gitlab:
    name: gitlab-network

The ISP firewall has opened the 8443/UDP port and there is an ISP DNAT that redirects 8443/UDP traffic from public IP to internal VM IP. There are no load balancers.

cpoile commented 1 year ago

Thank you @davide-prade, I've created a ticket to look into this, but it will likely be next week. Thank you for your patience while we try to get to the bottom of things.

streamer45 commented 1 year ago

@davide-prade Could you share your server config section for Calls?

davide-prade commented 1 year ago

@streamer45 here is the configuration section:

"com.mattermost.calls": {
    "allowscreensharing": true,
    "defaultenabled": true,
    "enablerecordings": false,
    "icehostoverride": "mattermost.anydata.it",
    "iceserversconfigs": "[{\"urls\":[\"stun:stun.global.calls.mattermost.com:3478\"]}]",
    "jobserviceurl": null,
    "maxcallparticipants": 0,
    "maxrecordingduration": 60,
    "rtcdserviceurl": null,
    "serversideturn": false,
    "turncredentialsexpirationminutes": 1440,
    "turnstaticauthsecret": "",
    "udpserveraddress": "0.0.0.0",
    "udpserverport": 8443
}

streamer45 commented 1 year ago

Thanks @davide-prade. What MM Calls version are you running?

A couple of things I would try are:

Leave udpserveraddress empty.
- There's an issue in the latest plugin version that doesn't properly recognize 0.0.0.0. Leaving it blank will have the same behaviour though, listening on all available interfaces.
Temporarily clear the iceserversconfigs and check connectivity works from other clients as well (e.g. browsers).
- This is mostly to remove the potential complexity introduced by STUN generated candidates and force the connection to go through the configured udp port instead.

Also note that any of the above changes will need you to restart the plugin in order to take effect.

davide-prade commented 1 year ago

@streamer45 the MM calls plugin version is 0.15.1.

I cleared the two settings that you indicate me, but the error is the same from the mobile. I disabled and re-enabled the calls plugin: is it enough? From browsers seems to work like before.

streamer45 commented 1 year ago

Yes, that's enough. So, if both your mobile client and browser you tested on are connected to the same network it should work, unless of course there's something on the mobile side preventing connectivity, a VPN app could do this or some system setting that restricts UDP maybe.

To debug this further we may need a bit more coordination, would you be able to join our community server at https://community.mattermost.com/? You can find me there as @claudio.costa.

davide-prade commented 1 year ago

@streamer45 I have tried from mobile network and also on the same wifi connection: the error is the same. We also tried a few days ago with an Apple phone and the call ended within seconds.

I will contact you in the community chat.

bguerin commented 1 year ago

Same problem here Mattermost server 8.0.1, self hosted Mattermost client 5.4.0

My RTC port (8443) is well opened and forwarded. Mattermost server see something if I trust its logs From tcpdump on my server, I see that the server is trying to respond to my internal LAN IP address (my server is hosted on a virtual private server on the cloud), which of course will NOT work

streamer45 commented 1 year ago

@bguerin That's usually normal. Several candidates get advertised to clients, some of which are local and won't work. The important one to have clients connect from the outer internet is the main host candidate that's served through the RTC port.

Could you share your current config for the plugin?

bguerin commented 1 year ago

@bguerin That's usually normal. Several candidates get advertised to clients, some of which are local and won't work. The important one to have clients connect from the outer internet is the main host candidate that's served through the RTC port.

Right, but what I'm describing is my cloud server sending packets to my 10.0.0.0/16 private, internal, LAN IP address of my laptop running the desktop app, and sending packets ONLY to that IP. And I mean the ip.dst field, not something in the RTC layer. And obviously, a 10.0.0.0/16 IP address cannot be routed on the Internet

Could you share your current config for the plugin?

 "com.mattermost.calls": {
                "allowenablecalls": false,
                "allowscreensharing": true,
                "defaultenabled": true,
                "enableipv6": false,
                "enablerecordings": false,
                "enableringing": true,
                "enablesimulcast": true,
                "icehostoverride": "chat.mydomain.tld",
                "iceservers": "stun:stun.global.calls.mattermost.com:3478",
                "iceserversconfigs": "[{\"urls\":[\"stun:stun.global.calls.mattermost.com:3478\"]}]",
                "jobserviceurl": null,
                "maxcallparticipants": 8,
                "maxrecordingduration": 60,
                "recordingquality": "medium",
                "rtcdserviceurl": null,
                "serversideturn": false,
                "tcpserveraddress": "",
                "tcpserverport": 8443,
                "turncredentialsexpirationminutes": 1440,
                "turnstaticauthsecret": "",
                "udpserveraddress": "",
                "udpserverport": 8443
}

streamer45 commented 1 year ago

@bguerin Server will try to send packets to whatever IP the client is sharing, including local ones. Of course it won't ever receive back from those which is fine and expected.

The problem seems to be that connectivity through chat.mydomain.tld:8443 is not really working as it should because that's the path packets should normally take if the client is outside the local network. You said the port is open and correctly forwarded. Would you be able to verify that packets can flow through correctly with some simple checks using netcat?

Example:

On the MM instance, running:

nc -l -u 0.0.0.0 8443

Make sure to temporarily disable the plugin or it will likely fail to listen on the same port.

On your client (i.e. where the Desktop app runs):

nc -u -v chat.mydomain.tld 8443

That should connect successfully.

bguerin commented 1 year ago

@streamer45 test already done, I can netcat to my server, on UDP port 8443

Connection from my client, on my home network (so NATed on Internet) to my cloud server is working The reverse, cloud server to my client, is not

I known that a client may advertise several addresses. The weird thing here is that my server only responds to my private home IP, I never see packets sent to my public IP My client keep trying connecting, and my server keep responding to a black hole

I was on mattermost server version 7.10.5 and it works fine. Then upgrading to 7.9.2 then 8.0.0 and stopped working, without changing anything else than the docker image to run

streamer45 commented 1 year ago

@bguerin Does the netcat check work inside docker container as well? We had a few cases of people struggling to connect with docker setups due to some weird configurations on the internal network.

I think the best way to debug further would be for you to reach out to me directly on our community server so that we can more easily share information (logs/configs and such).

streamer45 commented 1 year ago

I am closing this issue as the original reporter has deemed it fixed. The problem was that some devices/networks were unable to connect through UDP (likely due to client-side firewall restrictions). Enabling TCP candidates (a new functionality introduced in recent version) made it possible for these devices to connect without resorting to a more complex setup involving a TURN server.

Feel free to open a separate issue if you encounter similar problems. Closing for now.

heyvaldemar commented 1 year ago

I've successfully resolved the issue. The culprit was in the Traefik configuration.

For calls to function correctly, the following lines in the Traefik configuration are essential:

- "--entryPoints.calls.address=:8443/udp"

- "8443:8443/udp"

      - "traefik.udp.routers.mm-call-rtr.service=mm-call-svc"
      - "traefik.udp.routers.mm-call-rtr.entrypoints=calls"
      - "traefik.udp.services.mm-call-svc.loadBalancer.server.port=8443"
      - "traefik.http.middlewares.compresstraefik.compress=true"

For a comprehensive look, you can view my entire Docker configuration here.

mattermost / mattermost-plugin-calls