ICE fails when external IP does not match host's egress IP

[hkparker]

Description:

I have a somewhat interesting network setup. I am trying to run jitsi, per the docker installation instructions, on a VM in a private network. I have another VPS that is running nginx and proxies everything back to my VM over a wireguard tunnel. That's what my DNS points at. My VM, however, does not egress traffic back out through this VPS, it uses another ISP. I believe this is causing issues with ICE. Here's a crude diagram:

I've set my DOCKER_HOST_ADDRESS to that of the VPS (see docker-jitsi-meet#171). Everything seems to spin up without a hitch, however a few seconds after another participant joins a meeting, the frontend crashes and retries, like this:

With the following errors in the console

2020-05-27T04:13:31.838Z [modules/RTC/BridgeChannel.js] <l._send>:  Bridge Channel send: no opened channel.
...
2020-05-27T04:13:38.748Z [conference.js] <ee._onConferenceFailed>:  CONFERENCE FAILED: conference.iceFailed

and no errors from the containers.

I've confirmed everything is reachable over the network (all the ports in the documentation correctly go to the containers via my DNS, proxied through the VPS). I'm starting to get some confidence that this is related to having a different egress IP than my ingress IP. I'm no ICE expert, so I hope that makes sense.

I could try tunneling the traffic out that VPS as well, but I'd rather not be forced to. Happy to produce any requested logs.

Steps to reproduce:

1. Be in this crazy network setup I'm in
2. Install jisi as described in the docker deploy guide
3. Try to get 2 or more people in a meeting

I realize that's not very helpful and I'm sorry.

Expected behavior:

Meeting should not crash.

Actual behavior:

Described above.

Server information:

• Jitsi Meet version: latest versions of the containers (jitsi/base:stable-4627)
• Operating System: Arch Linux (docker)

Client information:

• Browser / app version: Chromium 80.0.3987.87
• Operating System: Arch Linux

[saghul]

You should set DOCKER_HOST_ADDRESS to your egress address in that case I think. Do you accept incoming UDP traffic there or is it purely egress? If it's the latter I think you might be out of luck.

[hkparker]

It's purely egress. I have the ability to open ports there but that IP is dynamic so I shouldn't be setting it in configuration anywhere. Alright, I was sorta expecting this to be impossible, let me try routing traffic from the VM out of the VPS and report back tonight, just to confirm that's actually what's causing this. Thanks for the quick response.

[hkparker]

Alright, with a couple source based routing rules, and NAT on my VPS, I was able to get my VM to egress out the VPS. No change in behavior though, as soon as another person joins the room I can only see them for a few seconds before everything crashes as before.

EDIT: this is without changing my DOCKER_HOST_ADDRESS, it still points to the VPS which is now the ingress and egress point for jitsi. Perhaps this wasn't my issue after all?

[saghul]

Did you wipe the config volume if you changed that value?

[hkparker]

Yep, I rm -r'd the config directory that gets volumed in after changing anything in .env

[hkparker]

Here's another clue from the console:

2020-05-28T05:10:40.870Z [modules/xmpp/JingleSessionPC.js] removeRemoteStream - OK,SDPs:  s {media: Array(3), raw: "v=0↵o=- 6427763265599790915 5 IN IP4 127.0.0.1↵s…trickle↵a=sctpmap:5000 webrtc-datachannel 1024↵", session: "v=0↵o=- 6427763265599790915 5 IN IP4 127.0.0.1↵s…5-03102453b271↵a=group:BUNDLE audio video data↵"} s {media: Array(3), raw: "v=0↵o=- 6427763265599790915 6 IN IP4 127.0.0.1↵s…trickle↵a=sctpmap:5000 webrtc-datachannel 1024↵", session: "v=0↵o=- 6427763265599790915 6 IN IP4 127.0.0.1↵s…5-03102453b271↵a=group:BUNDLE audio video data↵"}

Not sure why there's a 127.0.0.1 in there, or if that's normal.

[hkparker]

Looking at the console some more, it seems jitsi thinks the P2P channel is open, then tries a ping, and it fails and the P2P state is false before the hangup.

[saghul]

Please paste the full log somewhere for inspection.

[Bes79]

i have same issue but not in docker. I install jitsi from source by this https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-manual

[hkparker]

Alright, full logs from docker-compose and the browser console attached, hopefully this is helpful. compose-logs.txt console-logs.txt

I tried downgrading all the containers to the previous release tag (stable-4548-1), and now two people can join the call without crashing (audio and video work). However, I still see the following errors regularly printed to the javascript console, and a third person I had join the meeting was unable to see me and I was unable to see them.

2020-05-28T21:20:10.999Z [JitsiConference.js] <h.sendMessage>:  Failed to send E2E ping request or response. undefined
2020-05-28T21:20:12.560Z [modules/RTC/BridgeChannel.js] <l._send>:  Bridge Channel send: no opened channel.

Here are the traces from those logs: e2e-ping-trace.txt no-open-channel.txt

Seems to me that the underlying issue with ICE is still present in my setup on the older version, but that it doesn't cause a crash there (just calls don't work with 3+ people).

[hkparker]

Have you had a chance to take a look at this in any more detail? I'll start messing with older versions, looking at network logs, and seeing what else I can try to change to learn more.

[Janevski]

I have the same isue. And watching the logs and console logs, looks like the very same.

One deployment works, the other doesn't. Same firewall rules, same docker jitsi config, same version tags (copy paste), same nginx config. I've been banging my head against the wall this whole day. My future plans involve banging my head agains the wall within the next few months with this. Then i'd need another wall, being a head strong person. May God help us all, but i doubt even God knows what's wrong with this.

PS: And it's not a 'crazy network' they're just Hetzner Cloud VPS.

[iamkhalidbashir]

@Janevski LOL. I am facing the same issue

[DeviaVir]

Not sure if it helps you, but I've had some success with more than 2 users (switching to TCP/UDP) by setting the following JVB env vars:

  extraEnvs:
    PUBLIC_URL: https://some-url
    JVB_TCP_HARVESTER_DISABLED: "false"
    JVB_TCP_PORT: "30003"
    JVB_TCP_MAPPED_PORT: "30003"

and making sure that TCP route is actually routable from outside.

This is next to DOCKER_HOST_ADDRESS set to a small nginx box that just forwards UDP and TCP to the correct endpoints.

Oh, also, restarting prosody after every single change.

Note: this also assumes you use TCP port 30003.

[iamkhalidbashir]

My issue was that I opened the TCP port and did not open UDP instead