search

AlexxIT / go2rtc

Ultimate camera streaming application with support RTSP, RTMP, HTTP-FLV, WebRTC, MSE, HLS, MP4, MJPEG, HomeKit, FFmpeg, etc.
https://github.com/AlexxIT/Blog
MIT License
4.29k stars 352 forks source link

Issue with network topology involving VPN and 2 sites #333

Open J-Prince opened 1 year ago

J-Prince commented 1 year ago

Hello,

I am having issues with WebRTC protocol in my particular network topology involving 2 sites connected via VPN.

My setup:

Site1 Docker server running Frigate 0.12 rc2

2023-04-04 12:11:33.629540773  [INFO] Preparing go2rtc config...
2023-04-04 12:11:33.719847311  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2023-04-04 12:11:33.733325568  [INFO] Starting go2rtc...
2023-04-04 12:11:33.801569519  12:11:33.800 INF go2rtc version 1.2.0 linux/amd64
2023-04-04 12:11:33.801572791  12:11:33.801 INF [api] listen addr=:1984
2023-04-04 12:11:33.801576488  12:11:33.801 INF [rtsp] listen addr=:8554
2023-04-04 12:11:33.801578018  12:11:33.801 INF [srtp] listen addr=:8443
2023-04-04 12:11:33.801579341  12:11:33.801 INF [webrtc] listen addr=:8555
2023-04-04 12:11:43.630126325  [INFO] Starting go2rtc healthcheck service...

Site1 is connected to Site2 via Wireguard VPN.

On Site2, unable to start a webrtc session in Safari, iPhone.

tcpdump shows the data flowing freely, in both directions, and non-stop on UDP port 8555 between the Frigate server in Site1 and the device in Site2. However, the video never loads which is quite puzzling.

There is no NAT to deal with, all IPs are reachable directly between Site1 and Site2.

I have the following setup in Frigate:

go2rtc:
  streams:
    Cam1:
      - rtsp://user:pass@10.10.10.101:554/Streaming/Channels/101
    Cam2:
      - rtsp://user:pass@10.10.10.102:554/Streaming/Channels/101
  webrtc:
    candidates:
      - 172.17.0.2:8555  # docker image internal IP
      - 10.10.10.10:8555  # docker host

Would it be possible to tell go2rtc to NOT try to use any STUN or TURN server, since the connection should work internally. Maybe adding "local" subnets in the go2rtc configuration?

Even without Frigate in the equation, for example using the "stream.html WebRTC stream" link from the go2rtc page, same result.

Thank you!

sovapatr commented 1 year ago

I'm having the same issue. Using Frigate. Site to Site VPN using Wireguard on the routers. In Wireshark I see a constant stream of UDP traffic coming from the server on port 8555. But the video player just remains at loading.

AlexxIT commented 1 year ago

What viewers (html) have your tried? What about Frigate 12 release?

sovapatr commented 1 year ago

I'm using Frigate 0.12.0-DA3E197. Trying to access the stream from the /live/webrtc/links.html?src=backyard page. I've tried in both Chrome and Firefox if that is what you mean for viewers. stream.html?src=backyard&mode=mse works while at the second site. But mode=webrtc only seems to work when my laptop is on the same network as the server.

gedl commented 1 year ago

Current version: 1.5.0

Hey,

I have this problem too.

All my networking is internal (ie, the viewers and go2rtc are on different sites connected by wireguard at their respective routers). The moment I fire a stream (I'm using the webrtc-camera card), I see dozens of public ip-to-public ip UDP requests between the two routers (from the router behind which go2rtc is to the router behind which the viewer is), and this triggers the destination router's port scan alarm and ultimately blocks the originating router's ip.

Now, I can either a) whitelist the originating router's IP (which is a PITA, because it's dynamic and has I'd have to script out something) or b) disable port scanning altogether, but since both go2rtc and the viewer's host are directly reachable (through their private IPs) and I have no intention to make the cameras available on the internet, I am looking for a way to, somehow, instruct go2rtc to just use private IPs and therefore avoid the whole STUN situation altogether - because it's ultimately not necessary.

I've added 

webrtc:
  listen: ":8555"
  candidates:
   - 192.168.38.209:8555

to the go2rtc.yml config file, but that didn't alter this behaviour. I fully admit I don't understand these config params deeply (or the protocols in question, for that matter) but I was wondering if there's a way to keep everything flowing through the private IPs.

Thanks

AlexxIT commented 1 year ago

Basic docker users (also Frigate docker users) shouldn't have any problems with LAN connections. And shouldn't add any local IP to go2rtc config. They should use docker network host option. And they will be fine.

Frigate addon users should proxy WebRTC port and set local candidate IP manually. Because they can't use network host.

gedl commented 1 year ago

Hey AlexxIT

I'm running the go2rtc server in an LCX container in a proxmox host, on bridged mode.

Do you have any recommendations to stop go2rtc from going the STUN route ?

AlexxIT commented 1 year ago

@gedl do you have problems with local or with external connections?

gedl commented 1 year ago

Hi @AlexxIT

I don't expose the cameras externally at all - there is no domain name (or route, for that matter) into the go2rtc box. The STUN kicks in for internal to internal network requests, like so:

192.168.37.7 (browser) -> 192.168.37.1 / 192.168.138.1 (router 1 Lan/Wg) ---> internet ---> 192.168.38.1 / 192.168.138.2 (router 2 Lan/wg) -> 192.168.38.5 (proxmox) -> 192.168.38.115 (go2rtc).

Right when I open the webrtc card in homeassistant, the go2rtc fires a lot of UDP packets to router 2's public IP, instead of simply using the readily available private network path.

The network hosts on either side of the internet have open-bar access to each others, on all ports.

AlexxIT commented 1 year ago

WebRTC (in the browser and in the go2rtc) will try all possible ways to connect. Because they don't know which is the best route.

gedl commented 1 year ago

I understand that. I guess what @J-Prince and I are asking is if there's a way to instruct either party (but particularly go2rtc) to use a specific route, or to disable STUN altogether and skip straight to the next alternative (until eventually settles on using the internal, direct route between the hosts).

AlexxIT commented 1 year ago

You can disable STUN in the go2rtc yaml. Just use empty ice servers list.

gedl commented 1 year ago

Maybe I'm after a red herring with the STUN remark, but with 

webrtc:
  ice_servers: []

I still see a blast of UDP connections from one router's public IP (the one behind which go2rtc runs) to the other router's public IP (behind which the browser is). These two hosts are directly reachable through private IPs, and my goal was to make them do so rather than trying to punch UDP holes on the each other's firewalls.

This is what tcpdump shows when the page loads

23:09:13.280426 IP 18x.58.xxx.xx3.15543 > 2xx.187.xxx.xx0.53805: UDP, length 100
23:09:13.280813 IP 18x.58.xxx.xx3.15543 > 2xx.187.xxx.xx0.61214: UDP, length 100
23:09:13.281087 IP 18x.58.xxx.xx3.15543 > 2xx.187.xxx.xx0.53805: UDP, length 100
23:09:13.281302 IP 18x.58.xxx.xx3.15543 > 2xx.187.xxx.xx0.61214: UDP, length 100
23:09:13.291919 IP 18x.58.xxx.xx3.15543 > 2xx.187.xxx.xx0.53805: UDP, length 100
23:09:13.292146 IP 18x.58.xxx.xx3.15543 > 2xx.187.xxx.xx0.61214: UDP, length 100
AlexxIT commented 1 year ago

Browser still has ice settings. You can be changed only with JS code. No any settings for that. But it shouldn't be a problem.

gedl commented 1 year ago

But the IP blasting the requests is go2rtc's router, not the browser's.

I agree there shouldn't be any problem in the general case. In my particular case this is identified as a port scan (about 50 connections within 2 seconds).

Totally understand if this is a niche situation that doesn't warrant a generally available setting to avoid, but since it is go2rtc's side blasting the browser's side, I was hoping there would be a setting there for it just not to.

AlexxIT commented 1 year ago

You can enable trace log level for webrtc module and check all local and remote candidates there

gleanlux commented 1 year ago

I have the same problem. Has anyone found a solution? @gedl @J-Prince @sovapatr The two subnets are directly connected with wireguard, freely passable, yet webrtc cannot pass through.

I created an EOIP tunnel, the client got the ip address directly from the server subnet, but still it was not able to establish the connection. I haven't got around to tracing my connections yet.

J-Prince commented 1 year ago

I have the same problem. Has anyone found a solution? @gedl @J-Prince @sovapatr The two subnets are directly connected with wireguard, freely passable, yet webrtc cannot pass through.

Unfortunately no. I have to resort to use jsmpeg for Frigate live view, which less than ideal.

gleanlux commented 1 year ago

I started testing the problem from many directions and came to the following conclusions:

The most interesting observation:

Could there be any connection between WEBRTC and the jump points? Because we can clearly state that we are not talking about an IP fault here. @AlexxIT Please help us to investigate the error so we can find a solution. Any way I can help you to solve it.

AlexxIT commented 1 year ago

I think it's hard task for really hi level network engineers to build solid local routing between multiple local subnets. There is no magic here. Two clients are exchanging IP addresses. And both are trying to reach each other. They can or can't do it. Based on your internal routing.

barzag commented 6 days ago

@gedl Did you find a solution to your problem?