tsightler / ring-mqtt

Ring devices to MQTT Bridge
MIT License
569 stars 103 forks source link

Bug: Intermittent stream playback with HA and go2rtc #653

Closed Scope666 closed 1 year ago

Scope666 commented 1 year ago

_ring-mqtt_logs.txt.zip

Describe the Bug

One of my 4 Ring cameras will let me view the live stream with the HA / go2rtc / RTSPtoWebRTC combo, the rest won't. Ring-mqtt works perfectly without go2rtc / RTSPtoWebRTC. If I open the go2rtc web UI and click on Links, all of the MSE streams playback fine, but most of the time the WebRTC ones won't, which is what I think HA is trying to use. In the Ring app none of my cameras have the the option to enable "Legacy Video" as mentioned in your FAQ, I think they're all too old for H265. The one camera that seems to work most of the time (at least for Live) is the newest one I have, a Floodlight Cam Wired Plus... it's newer because it was a warranty replacement for the original Floodlight Cam.

Steps to Reproduce

This combo has worked solid for months, only recently started having issues.

Expected Behavior

Streams consistently play back. (live and event)

Log Output

Attached .zip at top

Screenshots

Go2RTC log

Config File

{
    "mqtt_url": "mqtt://192.168.1.200:1883",
    "mqtt_options": "",
    "livestream_user": "",
    "livestream_pass": "",
    "disarm_code": "",
    "enable_cameras": true,
    "enable_modes": true,
    "enable_panic": false,
    "hass_topic": "homeassistant/status",
    "ring_topic": "ring",
    "location_ids": [
        ""
    ]
}

go2rtc.yaml:
------------
log:
  level: debug
api:
  listen: ''
srtp:
  listen: ''
rtsp:
  listen: ':8554'
  default_query: video=h264&audio=aac&audio=opus
webrtc:
  listen: ''
streams:
  e04f43a453ea_live: exec:/app/ring-mqtt-latest/scripts/start-stream.sh e04f43a453ea live ring/dr4yes-fjo6-0/camera/e04f43a453ea {output}
  e04f43a453ea_event: exec:/app/ring-mqtt-latest/scripts/start-stream.sh e04f43a453ea event ring/dr4yes-fjo6-0/camera/e04f43a453ea {output}
  44ead8fd33c0_live: exec:/app/ring-mqtt-latest/scripts/start-stream.sh 44ead8fd33c0 live ring/dr4yes-fjo6-0/camera/44ead8fd33c0 {output}
  44ead8fd33c0_event: exec:/app/ring-mqtt-latest/scripts/start-stream.sh 44ead8fd33c0 event ring/dr4yes-fjo6-0/camera/44ead8fd33c0 {output}
  9c7613b476f6_live: exec:/app/ring-mqtt-latest/scripts/start-stream.sh 9c7613b476f6 live ring/dr4yes-fjo6-0/camera/9c7613b476f6 {output}
  9c7613b476f6_event: exec:/app/ring-mqtt-latest/scripts/start-stream.sh 9c7613b476f6 event ring/dr4yes-fjo6-0/camera/9c7613b476f6 {output}
  44ead8254f04_live: exec:/app/ring-mqtt-latest/scripts/start-stream.sh 44ead8254f04 live ring/dr4yes-fjo6-0/camera/44ead8254f04 {output}
  44ead8254f04_event: exec:/app/ring-mqtt-latest/scripts/start-stream.sh 44ead8254f04 event ring/dr4yes-fjo6-0/camera/44ead8254f04 {output}

Install Type

Unraid Docker

Version

v5.4.0

Operating System

Unraid

Architecture

x86_64

Machine Details

12th Gen Intel® Core™ i7-12700 @ 2079 MHz

tsightler commented 1 year ago

I'm not really sure what you expect me to do here as you state that "Ring-mqtt works perfectly without go2rtc / RTSPtoWebRTC" and also "If I open the go2rtc web UI and click on Links, all of the MSE streams playback fine..." which indicates that ring-mqtt is working fine as playback wouldn't happen regardless of the frontend protocol without this being the case. If you are having problems with the WebRTC connection that is between your browser and go2rtc, there's not much I can do about it. You might try a different browser.

I will offer some notes based on your logs, although the logs do not appear to line up with the debug output you provided above from go2rtc and they are super hard to read because it appears you tried to test many overlapping items also while still in the first 3 minutes with discovery data happening as well.

One thing I noted is that your first test was performed within 2 seconds of the addon starting, this stream will certainly fail (and it did) because there's no way the worker threads for the camera streams would have started by then. You can see the 5 second timeout (RTSP default) and then immediate retry in the logs. The retry was successful, launching the stream ~7 seconds after the initial request which timed out. This stream ended ~38 seconds later due to EOF (i.e. client disconnect). Looks totally normal.

The second stream started fine, took ~2.6 seconds, although there was a TCP reset error in the communications with go2rtc (not sure why, networking perhaps, or maybe the stream was terminated/timed out there). I personally suggest not starting any tests until 20-30 seconds after ring-mqtt has sent discovery data and put devices online, anything before that point is almost certain to fail or be unstable. I also see an event stream that appears to work fine.

So far nothing really special, although there is a weird anomoly at 03:02:57.435 where there is a client connection for the event stream, but then at 03:02:57.550 there is another client for the same stream. Not sure why there are two RTSP consumers connecting at nearly the same instant, but this should be OK, it was just weird. The stream seems to start fine, then at 03:03:16 one of these two clients disconnects, but the other remains connected (at least there is no disconnect in the logs). This event stream continues playing until it reaches the end of the recording, which happens at 03:05:03. As there is still a client connected go2rtc immediately attempts to restart the stream from the beginning.

This doesn't actually work correctly, because of a timing issue (a race condition really). Basically, go2rtc attempts to restart the producer script, but ring-mqtt hasn't detected that the previous stream has ended yet. When go2rtc starts the producer script it sees the message from ring-mqtt that the prior stream has ended and thinks there was an error so it immediately exits. This is really no big deal as go2rtc just immediately restarts the producer script again, which works the second time, however at 03:05:05 the second client disconnects, and the event stream is properly closed. So, a weird string of events, and a little bit of a behavior that could be improved, but nothing too concerning, honestly some (most?) frontend clients won't survive the stream restart anyway because it's a hard loop from the end of the previous video to the start of a new one.

10 seconds later the client reconnects to the event stream, again, everything works as expected, the stream plays back for just over 2 minutes, hits the end, and the entire process repeats.

I'm not really seeing anything wrong here.

Scope666 commented 1 year ago

I appreciate you looking at the logs. I've been running this combo for many months and it's always been great. I only recently started having this issue. Perhaps its changes in the way Ring is sending the streams, or it could be a go2rtc bug. It's just strange how the one camera works most of the time, but the other 3 just hang. I'm guessing HA is trying to use the WebRTC stream and that only seems to work on the one camera. If I go into the go2rtc UI I can play back pretty much any live or event stream as long as it's MSE. I wonder if there's a way I can tell HA to use MSE instead for now...

Also, if you want me to capture any other logs or anything, please let me know. Again, I know this probably isn't a ring-mqtt issue, but I didn't know who to ask since I learned this trick from you. (HA + ring-mqtt + RTSPtoWebRTC + go2rtc)

tsightler commented 1 year ago

It's not likely to be in the way Ring is sending the streams because that would impact all client viewing, regardless of WebRTC vs MSE. I've noticed no changes in how Ring streams video or handles recording (other than the H.265 stuff, which they appear to have stopped or at leasted figured out how to allow negotiation, which wasn't that hard really), nor would I expect any since they mostly just stream it exactly as the camera records it (transcoded videos might switch, but I haven't seen that so far, my guess is the cost of transcoding H.265 from H.264 is too high).

Have you tried other browsers (which browser are you using)?

Also, I notice your setup is a little unusual. You have go2rtc in a container, but it seems it's pointing to the same IP address for the RTSP stream, even though that has to be in the ring-mqtt container. Are you running go2rtc on the host network or are you exposing the 8554 port from the ring-mqtt container to the host network?

Have you tried just connecting to the RTSP streams from ring-mqtt with VLC? If so, that's the best way.

By far the most common issue with WebRTC is networking related, have you configured a fixed endpoint? Are you always connecting from the same network as the host is on, or do the problems happen from remote networks?

Scope666 commented 1 year ago

Browsers I've tried Chrome and Safari, also the native Home Assistant app on iOS. All seem to behave the same.

Right now I have Home Assistant, ring-mqtt and go2rtc all set to "host" for network type. I'm open to your suggestions for changing that, but in the past when I try other combos, the containers seem to have problems talking to each other.

My primary use case is I have a cameras page in Home Assistant. I have it setup where if I click the main thumb, it goes live, or if I click the little video icon at the bottom, it plays back the most recent event. This all worked perfectly for a long time, and with the combo would start playback within a second or two, vs the stock HLS way it's like a 10+ second pause.

99% of the time I'm on the same network as HA, ring-mqtt, go2rtc, etc.

Example of one that won't play (Front door, sometimes you'll hear audio, but no video)

image

Log for this attempt

2023-06-26T05:25:52.569Z ring-rtsp [Front Door] State indicates live stream is activating
2023-06-26T05:25:52.669Z ring-mqtt [Front Door] Live stream session successfully initialized, starting worker
2023-06-26T05:25:52.670Z ring-wrtc [Front Door] Live stream WebRTC worker received start command
2023-06-26T05:25:52.677Z ring-wrtc [Front Door] Live stream transcoding process is starting
2023-06-26T05:25:52.856Z ring-wrtc [Front Door] Websocket signaling for Ring cloud connected successfully
2023-06-26T05:25:56.170Z ring-wrtc [Front Door] Live stream transcoding process has started
2023-06-26T05:25:56.379Z ring-attr [Front Door] ring/dr4yes-fjo6-0/camera/e04f43a453ea/stream/attributes {"status":"active"}
2023-06-26T05:25:56.379Z ring-wrtc [Front Door] Live stream WebRTC session is connected
2023-06-26T05:25:56.398Z ring-rtsp [Front Door] State indicates live stream is active
2023-06-26T05:25:57.507Z ring-rtsp [go2rtc] DBG [rtsp] new consumer stream=e04f43a453ea_live
2023-06-26T05:25:57.811Z ring-rtsp [go2rtc] DBG [exec] run launch=5.304962677s
2023-06-26T05:25:57.811Z ring-rtsp [go2rtc] DBG [streams] start producer url=exec:/app/ring-mqtt-latest/scripts/start-stream.sh e04f43a453ea live ring/dr4yes-fjo6-0/camera/e04f43a453ea {output}
2023-06-26T05:25:59.231Z ring-mqtt [Front Door] Requesting an updated interval snapshot
2023-06-26T05:26:00.665Z ring-mqtt [Front Door] Successfully retrieved updated snapshot
2023-06-26T05:26:00.665Z ring-mqtt [Front Door] ring/dr4yes-fjo6-0/camera/e04f43a453ea/snapshot/image <binary_image_data>
2023-06-26T05:26:00.665Z ring-attr [Front Door] ring/dr4yes-fjo6-0/camera/e04f43a453ea/snapshot/attributes {"timestamp":1687757161}
2023-06-26T05:26:02.709Z ring-rtsp [go2rtc] WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:224 > error="read tcp 192.168.1.200:8554->192.168.1.200:39568: read: connection reset by peer"

In HA I have "Generic Cameras" set up, the Front Door looks like this: rtsp://192.168.1.200:8554/e04f43a453ea_live

Scope666 commented 1 year ago

I see this if I try to manually play the WebRTC event stream in go2rtc:

2023-06-26T05:51:46.308Z ring-rtsp [go2rtc] WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:159 > error=EOF url="exec:/app/ring-mqtt-latest/scripts/start-stream.sh e04f43a453ea event ring/dr4yes-fjo6-0/camera/e04f43a453ea {output}"

I did some more experimenting. Consistently that one newer camera always works for the live stream, and the NON-TRANSCODED Event stream. If I switch to the transcoded one it fails to playback. Other 3 cameras always fail on live or any event. There has to be something different about that one newer camera... the other 3 are all close to 5 years old.

Scope666 commented 1 year ago

More experimenting, the 3 cameras that don't work, show this when I play the working MSE version via the go2rtc "info" UI for video codec being sent:

"video, recvonly, H.264 High 4.1",

The one working, newer camera shows this:

"video, recvonly, H.264 Main 3.2",

Different H.264 levels coming from Ring. Most likely a go2rtc bug.

tsightler commented 1 year ago

All of my cameras are 5 years old. It doesn't make much sense really, especially when you start talking about event streams because those are always transcoded by the local ffmpeg anyway, so they should be 100% consistent in their output type.

One thing I thing I did notice in your log:

2023-06-26T05:25:57.811Z ring-rtsp [go2rtc] DBG [exec] run launch=5.304962677s

That's too long, anything over 5 seconds will almost always hang because the RTSP timeout is 5 seconds (I can't change this). For reliable streaming the stream needs to start consistenly in <4 seconds. To give you some example, none of my cameras take longer than 2.5 seconds, most of the time <2 seconds, to start.

Now, this wouldn't impact event streams, because those don't have the same overhead, they just stream the recording file straight from AWS S3 to RTSP via ffmpeg process.

I can't argue that it might be a go2rtc bug, have you tried downgrading the go2rtc container to a prior version to see if it works again, since you say it previously worked just fine?

Scope666 commented 1 year ago

Tried a bunch of versions, here's debug log output from 1.2.0. Same exact behavior, the one camera that sends Main 3.2 works perfectly, the other 3 sending High 4.1 don't. Again, this is trying playback in Home Assistant, or manually trying to play the WebRTC stream. MSE streams seem to work most of the time, but I don't think H.A. can use them.

17:54:03.481 INF go2rtc version 1.2.0 linux/amd64
17:54:03.481 INF [api] listen addr=:1984
17:54:03.482 INF [srtp] listen addr=:8443
17:54:03.482 INF [webrtc] listen addr=:8555
17:54:44.885 DBG [streams] probe producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
17:54:46.673 DBG [streams] start producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
17:55:00.919 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
17:55:00.934 DBG [streams] probe producer url=rtsp://192.168.1.200:8554/9c7613b476f6_event
17:55:01.317 DBG [streams] start producer url=rtsp://192.168.1.200:8554/9c7613b476f6_event
17:55:09.510 DBG [streams] probe producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:55:11.558 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/9c7613b476f6_event
17:55:12.167 DBG [streams] start producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:56:08.407 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:56:31.915 DBG [streams] probe producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:56:34.264 DBG [streams] start producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:56:42.101 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:56:58.036 DBG [webrtc] new consumer url=camera.front_door_live
17:56:58.041 DBG [streams] probe producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
17:56:59.912 DBG [streams] start producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live

The one "good" camera starts up QUICK, like around 1 second, maybe 2 at the MOST. The 5+ seconds you're seeing in the log is probably it trying to auto-negotiate with one of the "bad" cameras, and failing.

Scope666 commented 1 year ago

Camera that works in HA on the left, one of the ones that doesn't on the right: (playing back MSE to get info)

image

tsightler commented 1 year ago

So I just checked my cameras and they are all using H.264 High 4.1 with no problems:

{
  "producers": [
    {
      "type": "RTSP active producer",
      "url": "rtsp://03cabcc9-ring-mqtt:8554/f83331d131bd_event",
      "remote_addr": "172.30.33.4:8554",
      "user_agent": "go2rtc/1.5.0",
      "medias": [
        "video, recvonly, H.264 High 4.1",
        "audio, recvonly, MPEG4-GENERIC/16000",
        "audio, recvonly, OPUS/48000/2"
      ],
      "receivers": [
        "96 H264, bytes=2150451, senders=1",
        "98 OPUS/48000/2, bytes=47273, senders=1",
        "97 MPEG4-GENERIC/16000, bytes=26789, senders=0"
      ],
      "recv": 2247493
    }
  ],
  "consumers": [
    {
      "type": "WebRTC/WebSocket async passive consumer",
      "remote_addr": "udp4 prflx 192.168.1.135:61975 related :0",
      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58",
      "medias": [
        "video, sendonly, VP8, RTX, VP9, H264, AV1, RED, ULPFEC, FLEXFEC-03",
        "audio, sendonly, OPUS/48000/2, RED/48000/2, G722/8000, PCMU/8000, PCMA/8000, CN/8000, TELEPHONE-EVENT/48000, TELEPHONE-EVENT/8000, L16"
      ],
      "senders": [
        "102 H264, bytes=2150451, receivers=1",
        "111 OPUS/48000/2, bytes=47273, receivers=1"
      ],
      "send": 2225314
    }
  ]
}

I'm not sure why your consumer line is so different than mine, are your examples above from MSE?

Scope666 commented 1 year ago

Yeah, that's with them both playing MSE. If I try WebRTC or HA with the "bad" one, it doesn't populate any codec info.

I know you have a similar setup to mine, so perhaps we can look for what's different between them to find the culprit.

How do you have your "network type" for your Dockers for HA, ring-mqtt and go2rtc ? ... all Bridge? ... custom network? Mine are all "host" as that's how HA works the best for me with my MANY devices, and ring-mqtt & go2rtc communicate with HA better with host also. When I've tried having those 2 in a bridge, I have trouble with them talking to each other.

tsightler commented 1 year ago

Well, these days my setups all use HAOS, so ring-mqtt is just running as an HA addon. The addon uses the identical Docker image so that's the same, just the Docker container runs on the HA host. Same for go2rtc. So basically, it's the same for the most part. The only real difference is that the ring-mqtt addon runs in Docker network mode, while the go2rtc addon runs on host network, so not that different really. I wouldn't really expect that to be the problem, but I did have a user with a similar problem that was seeing poor network performance in Docker due to a static IP on the host, it was weird, but seemed to be a known issue.

I really don't think you should discount the stream startup delay. The idea that this is caused by negotiation doesn't really hold up as the stream is negotiated between ring-mqtt and Ring media servers without any knowledge of the capabilities of the frontend system that requested the connection. Also, this is basic SDP based negotiation, so there's no back-and-forth, ring-mqtt provides an SDP offer that includes High profile, Ring media servers provide an SDP answer that either accepts this, or something lower. It always takes only a single offer and a single answer, even if there is a protocol mismatch.

Not only that, but the logs clearly show a 3.5 second delay from transcoding process moving from "starting" to "started". That feels really, really long. Normally I would expect that to take <1 second (an RPi 4 takes maybe .7-.8 seconds, my much faster system typicalyl takes .5 seconds). 3.5 seconds is forever in relative terms.

I don't know if you are up for it, but if any of your cameras aren't too intrusive, and you are willing to share one with me for a short time, I can see if it works for me on my setup. That would eliminate a few variables including anything to do with the cameras specifically or the specific components in the chain, since all of that would be the same. I understand this is intrusive, but I wouldn't need access for long just to give it a test.

Scope666 commented 1 year ago

I don't mind at all, I appreciate the help! I'd also even be down to do a Zoom or something so we could poke around, if you're up for that also. Let me know how I can share the camera(s) to you.

Also, this is a stupid powerful machine ... 12th gen i7 running Unraid, and the overall utilization is very low, so something strange is going on. I'd like to look at the logs for the GOOD camera and see how long that's taking. In the UI the stream comes up fast, like it always did.

image
tsightler commented 1 year ago

Yeah, I saw that it was an i7-12700 in the OP so I didn't think CPU would likely be the problem, I'm thinking more likely to be something strange with Docker networking as I've seen that problem crop up time and again, but it's super difficult to troubleshoot. A quick Google search turned up mulitple issues with this on unRAID, although nothing super recent. That's why I thought it might be interesting to test with my setup just because it removes a lot of those variables since I'll test on my known working system. I also have a similarly overpowered system for my production HA setup, a VM running on an AMD Ryzen 7 5800.

If you are open to sharing a camera or two you can send me an invite from the Ring app. My email is the same as the username here at Gmail. You can also email me there directly first if you feel more comfortable rather than talking here in public. I really need to setup a Discord channel I guess.

Scope666 commented 1 year ago

Invite sent. btw, just as a test I installed RTSPtoWEB and manually added the streams generated from ring-mqtt. All cameras can playback WebRTC in there, but I can't get the automatic Home Assistant - RTSPtoWebRTC thing to work. I'm just testing it in the RTSPtoWEB UI.

image
Scope666 commented 1 year ago

Another update, I got the RTSPtoWebRTC add-on working correctly in HA, all 4 cameras are working, and starting up stupid fast. (with RTSPtoWEB instead of go2rtc, everything starts in around 1 second)

Solution was here: https://github.com/deepch/RTSPtoWeb/issues/103#issuecomment-1046296328

Scope666 commented 1 year ago

More experimenting today. If I run go2rtc on a different machine (used a Windows box as a test) the WebRTC streams work for the 3 "bad" cameras. So your theory about it being an Unraid - networking issue seems correct. Weird that RTSPtoWEB doesn't have this problem, and also weird that it all used to work perfectly. Unraid recently had a pretty big update, I'm wondering if it simply doesn't play nice with go2rtc anymore...

tsightler commented 1 year ago

Have you increased the default UDP buffers (rmem_max/rmem_default)? I know that they are really low by default on most Linux distros and I saw some reports of other docker containers having issues with UDP on unRAID. The RTSP streams are all UDP so it can overflow the buffers quickly, especially during startup. I actually convinced the HAOS devs to increase their default back about a year ago, but most Linux systems have a paltry 100K or so which just isn't much for 1080p streaming and can be quickly overrun during the initial startup burst. Since the cameras that work are using the highest profile with the highest bitrate, perhaps that is the issue?

Unfortunately, I don't know anything about unRAID specifically to even know if it's possible to increase the buffers, but I'm assuming there's some way to access the native Linux OS and change it, but there is a section in the ring-mqtt FAQ around increasing it for HAOS (from the past before they had increased the defaults), and the steps are the same for any Linux.

Scope666 commented 1 year ago

I found where it's set, are these high enough or should I increase?

root@xbmcdown:~# cat /etc/sysctl.conf
# from https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers
# increase the number of inotify watches
fs.inotify.max_user_watches=524288

# from: https://www.andrisoft.com/support/portal/kb/article/os-level-optimisations
# better support for 10Gb ethernet
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# prevent Docker from changing the iptable's FORWARD chain policy to DROP which break VM networking
net.ipv4.ip_forward = 1

# from: https://github.com/kubernetes-sigs/kind/issues/2240
# handle net.netfilter.nf_conntrack_count max exceeded
# note: the setting below may need adjustment and may be more settings that need to be applied,
# refer to: https://discuss.aerospike.com/t/how-to-handle-net-netfilter-nf-conntrack-count-max-exceeded/5051
net.netfilter.nf_conntrack_max=131072
tsightler commented 1 year ago

Well, rmem_max looks big enough, however, what is really important is to set rmem_default so that any socket automatically gets that size without explicitly having to request larger. The rmem_default for most Linux is 100K, but I recommend at least 2-4MB, so I'd suggest adding the following directly under the rmem_max setting:

net.core.rmem_default = 4194304

Definitely a long shot, but worth a try I guess.

Scope666 commented 1 year ago

I tried it, no change. Such a strange problem to diagnose. Weird that the MSE streams work perfectly fine for all cameras, but WebRTC only works on the one newer camera with a lower H.264 Level.

BTW, another thing I tried is I spun up a VM with HAOS and loaded go2rtc in there, no change in behavior, which points me even more towards an Unraid networking issue.

tsightler commented 1 year ago

That's why I keep coming back to UDP and maybe a conflict with firewall rules or something. If I'm not mistaken, and I might be as, unlike WebRTC and RTSP, I know very little about the MSE protocol, but I believe it uses byte streams over HTTP, while WebRTC requires dynamic UDP/TCP ports.

Did you happen to explicitly specify a WebRTC port and candidate list in the go2rtc config and verify that this port is fully open? Normally I wouldn't expect this to be required, but maybe if unRAID has a built-in firewall it's blocking some of this? Wouldn't really explain why one camera happens to work though, that is really the biggest mystery.

Scope666 commented 1 year ago

I didn't really do anything special with the go2rtc config, other than kill the rstp port as it was conflicting with the one in ring-mqtt. Also, webrtc works perfectly fine in RTSPtoWEB, it's only go2rtc that has the issue, and only with those 3 cameras, and only recently stopped working. In the past go2rtc was working fine for MONTHS. It's something specific to Unraid - go2rtc and because of a recent change with one of them.

tsightler commented 1 year ago

Seems like it has to be unRAID since, if it was go2rtc, I'd likely be seeing lots of other similar reports. Also, old versions of go2rtc are still out there. The streaming core was rewritten in go2rtc v1.3.0 and that caused lots of little issues. Versions v1.1.2 fake limited the H.264 profile, which would have been back in Feb or so.

Just for kicks I've modified the dev branch to request only H.264 baseline profile 3.1 if you want to give that a try. I think this is unlikely to be the problem as, based on what I can tell, go2rtc is just reporting what was negotiated as the highest supported profile, not what the stream actually contains, but it's an easy thing to test as you can just set BRANCH=dev to have the container pull down that version.

I'm really struggling to come up with any ideas, however, there is one possibility to get more insight. If you modify the go2rtc config to get a trace debugging log for RTSP and WebRTC, that might tell us something. If you do that and send it to me, I can compare it to my local instance and we'll see all the negotiation happening and maybe can spot something not working.

Scope666 commented 1 year ago

Ok, let me know if this is ok for the go2rtc config for debug logging:

rtsp:
  listen: ""

log:
  level: info  # default level
  api: debug
  rtsp: debug
  streams: debug
  webrtc: debug
tsightler commented 1 year ago

I would change rtsp and webrtc to "trace" instead of "debug".

Scope666 commented 1 year ago

This is all that go2rtc logs, even though the last 3 fail to playback, go2rtc doesn't seem to know it's failing.

16:03:38.547 INF [api] listen addr=:1984
16:03:38.547 INF [srtp] listen addr=:8443
16:03:38.547 INF [webrtc] listen addr=:8555/tcp
16:03:57.114 DBG [streams] start producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
16:04:07.363 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
16:04:12.934 DBG [streams] start producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
16:04:35.434 DBG [streams] start producer url=rtsp://192.168.1.200:8554/44ead8254f04_live
16:04:37.174 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
16:04:49.658 DBG [streams] start producer url=rtsp://192.168.1.200:8554/44ead8fd33c0_live
16:04:49.669 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/44ead8254f04_live
16:05:03.897 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/44ead8fd33c0_live

EDIT: Will change to "trace" for both.

Scope666 commented 1 year ago

Ok, here's Trace logs for 2 attempts. The 1st is the good camera, the 2nd is one of the bad ones. (Front Door)

16:06:52.346 INF go2rtc version 1.5.0 linux/amd64
16:06:52.347 INF [api] listen addr=:1984
16:06:52.347 INF [srtp] listen addr=:8443
16:06:52.347 INF [webrtc] listen addr=:8555/tcp
16:07:16.931 TRC [webrtc] offer:
v=0
o=- 4019406115941217544 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:e2Ic
a=ice-pwd:rPEYyd5VWGaq+YnETHJXKzcj
a=ice-options:trickle
a=fingerprint:sha-256 DF:60:56:30:FF:A7:B8:AA:84:84:7E:C7:2F:7B:74:AB:22:DE:C6:62:DA:DF:B0:44:22:4B:EA:90:8D:BE:CA:96
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 35 36 37 38 102 103 104 105 106 107 108 109 127 125 39 40 41 42 43 44 45 46 47 48 112 113 114 115 116 117 118 49
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:e2Ic
a=ice-pwd:rPEYyd5VWGaq+YnETHJXKzcj
a=ice-options:trickle
a=fingerprint:sha-256 DF:60:56:30:FF:A7:B8:AA:84:84:7E:C7:2F:7B:74:AB:22:DE:C6:62:DA:DF:B0:44:22:4B:EA:90:8D:BE:CA:96
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:35 VP9/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=fmtp:35 profile-id=1
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:37 VP9/90000
a=rtcp-fb:37 goog-remb
a=rtcp-fb:37 transport-cc
a=rtcp-fb:37 ccm fir
a=rtcp-fb:37 nack
a=rtcp-fb:37 nack pli
a=fmtp:37 profile-id=3
a=rtpmap:38 rtx/90000
a=fmtp:38 apt=37
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:103 rtx/90000
a=fmtp:103 apt=102
a=rtpmap:104 H264/90000
a=rtcp-fb:104 goog-remb
a=rtcp-fb:104 transport-cc
a=rtcp-fb:104 ccm fir
a=rtcp-fb:104 nack
a=rtcp-fb:104 nack pli
a=fmtp:104 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:105 rtx/90000
a=fmtp:105 apt=104
a=rtpmap:106 H264/90000
a=rtcp-fb:106 goog-remb
a=rtcp-fb:106 transport-cc
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack
a=rtcp-fb:106 nack pli
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=106
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
a=rtpmap:125 rtx/90000
a=fmtp:125 apt=127
a=rtpmap:39 H264/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:41 H264/90000
a=rtcp-fb:41 goog-remb
a=rtcp-fb:41 transport-cc
a=rtcp-fb:41 ccm fir
a=rtcp-fb:41 nack
a=rtcp-fb:41 nack pli
a=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f
a=rtpmap:42 rtx/90000
a=fmtp:42 apt=41
a=rtpmap:43 H264/90000
a=rtcp-fb:43 goog-remb
a=rtcp-fb:43 transport-cc
a=rtcp-fb:43 ccm fir
a=rtcp-fb:43 nack
a=rtcp-fb:43 nack pli
a=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f
a=rtpmap:44 rtx/90000
a=fmtp:44 apt=43
a=rtpmap:45 AV1/90000
a=rtcp-fb:45 goog-remb
a=rtcp-fb:45 transport-cc
a=rtcp-fb:45 ccm fir
a=rtcp-fb:45 nack
a=rtcp-fb:45 nack pli
a=rtpmap:46 rtx/90000
a=fmtp:46 apt=45
a=rtpmap:47 AV1/90000
a=rtcp-fb:47 goog-remb
a=rtcp-fb:47 transport-cc
a=rtcp-fb:47 ccm fir
a=rtcp-fb:47 nack
a=rtcp-fb:47 nack pli
a=fmtp:47 profile=1
a=rtpmap:48 rtx/90000
a=fmtp:48 apt=47
a=rtpmap:112 H264/90000
a=rtcp-fb:112 goog-remb
a=rtcp-fb:112 transport-cc
a=rtcp-fb:112 ccm fir
a=rtcp-fb:112 nack
a=rtcp-fb:112 nack pli
a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
a=rtpmap:113 rtx/90000
a=fmtp:113 apt=112
a=rtpmap:114 H264/90000
a=rtcp-fb:114 goog-remb
a=rtcp-fb:114 transport-cc
a=rtcp-fb:114 ccm fir
a=rtcp-fb:114 nack
a=rtcp-fb:114 nack pli
a=fmtp:114 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=64001f
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 red/90000
a=rtpmap:117 rtx/90000
a=fmtp:117 apt=116
a=rtpmap:118 ulpfec/90000
a=rtpmap:49 flexfec-03/90000
a=rtcp-fb:49 goog-remb
a=rtcp-fb:49 transport-cc
a=fmtp:49 repair-window=10000000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:e2Ic
a=ice-pwd:rPEYyd5VWGaq+YnETHJXKzcj
a=ice-options:trickle
a=fingerprint:sha-256 DF:60:56:30:FF:A7:B8:AA:84:84:7E:C7:2F:7B:74:AB:22:DE:C6:62:DA:DF:B0:44:22:4B:EA:90:8D:BE:CA:96
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144
a=candidate:3327440176 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 56531 typ host generation 0 ufrag e2Ic network-cost 999
a=candidate:3327440176 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 56533 typ host generation 0 ufrag e2Ic network-cost 999
a=candidate:3327440176 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 56535 typ host generation 0 ufrag e2Ic network-cost 999
16:07:16.932 TRC [rtsp] client request:
DESCRIBE rtsp://192.168.1.200:8554/9c7613b476f6_live RTSP/1.0
User-Agent: go2rtc/1.5.0
CSeq: 1
Accept: application/sdp
Require: www.onvif.org/ver20/backchannel
16:07:18.763 TRC [rtsp] client response:
RTSP/1.0 200 OK
Cseq: 1
Content-Length: 524
Content-Type: application/sdp
v=0
o=- 1 1 IN IP4 0.0.0.0
s=go2rtc/1.5.0
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z01AKKaAeAIn5UA=,aO48gAA=; profile-level-id=4D4028
a=control:trackID=0
m=audio 0 RTP/AVP 97
a=rtpmap:97 MPEG4-GENERIC/48000/2
a=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=119056E500
a=control:trackID=1
m=audio 0 RTP/AVP 98
a=rtpmap:98 OPUS/48000/2
a=fmtp:98 sprop-stereo=1
a=control:trackID=2
16:07:18.763 TRC [rtsp] client request:
SETUP rtsp://192.168.1.200:8554/9c7613b476f6_live/trackID=0 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=0-1
CSeq: 2
16:07:18.763 TRC [rtsp] client response:
RTSP/1.0 200 OK
Transport: RTP/AVP/TCP;unicast;interleaved=0-1
Cseq: 2
Session: 45819855;timeout=60
16:07:18.763 TRC [rtsp] client request:
SETUP rtsp://192.168.1.200:8554/9c7613b476f6_live/trackID=2 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=4-5
CSeq: 3
Session: 45819855
16:07:18.764 TRC [rtsp] client response:
RTSP/1.0 200 OK
Transport: RTP/AVP/TCP;unicast;interleaved=4-5
Cseq: 3
Session: 22288895;timeout=60
16:07:18.764 DBG [streams] start producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
16:07:18.764 TRC [rtsp] client request:
PLAY rtsp://192.168.1.200:8554/9c7613b476f6_live RTSP/1.0
CSeq: 4
Session: 45819855
16:07:18.764 TRC [rtsp] client response:
RTSP/1.0 200 OK
Cseq: 4
Session: 22288895
16:07:18.794 TRC [webrtc] answer
v=0
o=- 3107859205819475371 1687896438 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 F4:4F:21:89:EE:BE:09:4D:17:6C:EA:07:41:25:41:DA:AC:D6:98:4D:73:69:73:CB:95:FF:45:14:B4:8A:19:41
a=extmap-allow-mixed
a=group:BUNDLE 0 1 2
m=audio 9 UDP/TLS/RTP/SAVPF 111 0 8
c=IN IP4 0.0.0.0
a=setup:active
a=mid:0
a=ice-ufrag:QzOTTwhTrBktfjTU
a=ice-pwd:PAyPQwdJIKeyMmJusKQaPubEpOaUQBcZ
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=rtcp-fb:111 transport-cc 
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:1175130809 cname:go2rtc
a=ssrc:1175130809 msid:go2rtc go2rtc-audio
a=ssrc:1175130809 mslabel:go2rtc
a=ssrc:1175130809 label:go2rtc-audio
a=msid:go2rtc go2rtc-audio
a=sendonly
a=candidate:227965208 1 udp 2130706431 192.168.1.200 46435 typ host
a=candidate:227965208 2 udp 2130706431 192.168.1.200 46435 typ host
a=candidate:3528925834 1 udp 2130706431 172.18.0.1 56544 typ host
a=candidate:3528925834 2 udp 2130706431 172.18.0.1 56544 typ host
a=candidate:574494503 1 udp 2130706431 192.168.122.1 36930 typ host
a=candidate:574494503 2 udp 2130706431 192.168.122.1 36930 typ host
a=candidate:462606888 1 udp 1694498815 73.194.113.79 58278 typ srflx raddr 0.0.0.0 rport 58278
a=candidate:462606888 2 udp 1694498815 73.194.113.79 58278 typ srflx raddr 0.0.0.0 rport 58278
a=candidate:2959411448 1 tcp 2124414975 192.168.1.200 8555 typ host tcptype passive
a=candidate:2959411448 2 tcp 2124414975 192.168.1.200 8555 typ host tcptype passive
a=candidate:1873040234 1 tcp 2124414975 172.18.0.1 8555 typ host tcptype passive
a=candidate:1873040234 2 tcp 2124414975 172.18.0.1 8555 typ host tcptype passive
a=candidate:2681038535 1 tcp 2124414975 192.168.122.1 8555 typ host tcptype passive
a=candidate:2681038535 2 tcp 2124414975 192.168.122.1 8555 typ host tcptype passive
a=end-of-candidates
m=video 9 UDP/TLS/RTP/SAVPF 102 106 112
c=IN IP4 0.0.0.0
a=setup:active
a=mid:1
a=ice-ufrag:QzOTTwhTrBktfjTU
a=ice-pwd:PAyPQwdJIKeyMmJusKQaPubEpOaUQBcZ
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:102 H264/90000
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtcp-fb:102 goog-remb 
a=rtcp-fb:102 transport-cc 
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack 
a=rtcp-fb:102 nack pli
a=rtpmap:106 H264/90000
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:106 goog-remb 
a=rtcp-fb:106 transport-cc 
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack 
a=rtcp-fb:106 nack pli
a=rtpmap:112 H264/90000
a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
a=rtcp-fb:112 goog-remb 
a=rtcp-fb:112 transport-cc 
a=rtcp-fb:112 ccm fir
a=rtcp-fb:112 nack 
a=rtcp-fb:112 nack pli
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:1308081781 cname:go2rtc
a=ssrc:1308081781 msid:go2rtc go2rtc-video
a=ssrc:1308081781 mslabel:go2rtc
a=ssrc:1308081781 label:go2rtc-video
a=msid:go2rtc go2rtc-video
a=sendonly
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:active
a=mid:2
a=sendrecv
a=sctp-port:5000
a=ice-ufrag:QzOTTwhTrBktfjTU
a=ice-pwd:PAyPQwdJIKeyMmJusKQaPubEpOaUQBcZ
16:07:31.002 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/9c7613b476f6_live
16:07:31.002 TRC [rtsp] client request:
TEARDOWN rtsp://192.168.1.200:8554/9c7613b476f6_live RTSP/1.0
CSeq: 5
Session: 45819855
16:07:36.242 TRC [webrtc] offer:
v=0
o=- 8688816586531986325 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:PbK/
a=ice-pwd:o4+diB62YMnzEDZ19S+eAQG0
a=ice-options:trickle
a=fingerprint:sha-256 B8:05:C9:24:68:F1:87:0D:1F:B7:13:8B:8C:FE:8C:95:DB:D7:9D:2F:46:D4:AC:9D:EA:4A:43:22:39:EC:51:92
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 35 36 37 38 102 103 104 105 106 107 108 109 127 125 39 40 41 42 43 44 45 46 47 48 112 113 114 115 116 117 118 49
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:PbK/
a=ice-pwd:o4+diB62YMnzEDZ19S+eAQG0
a=ice-options:trickle
a=fingerprint:sha-256 B8:05:C9:24:68:F1:87:0D:1F:B7:13:8B:8C:FE:8C:95:DB:D7:9D:2F:46:D4:AC:9D:EA:4A:43:22:39:EC:51:92
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:35 VP9/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=fmtp:35 profile-id=1
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:37 VP9/90000
a=rtcp-fb:37 goog-remb
a=rtcp-fb:37 transport-cc
a=rtcp-fb:37 ccm fir
a=rtcp-fb:37 nack
a=rtcp-fb:37 nack pli
a=fmtp:37 profile-id=3
a=rtpmap:38 rtx/90000
a=fmtp:38 apt=37
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:103 rtx/90000
a=fmtp:103 apt=102
a=rtpmap:104 H264/90000
a=rtcp-fb:104 goog-remb
a=rtcp-fb:104 transport-cc
a=rtcp-fb:104 ccm fir
a=rtcp-fb:104 nack
a=rtcp-fb:104 nack pli
a=fmtp:104 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:105 rtx/90000
a=fmtp:105 apt=104
a=rtpmap:106 H264/90000
a=rtcp-fb:106 goog-remb
a=rtcp-fb:106 transport-cc
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack
a=rtcp-fb:106 nack pli
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=106
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
a=rtpmap:125 rtx/90000
a=fmtp:125 apt=127
a=rtpmap:39 H264/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:41 H264/90000
a=rtcp-fb:41 goog-remb
a=rtcp-fb:41 transport-cc
a=rtcp-fb:41 ccm fir
a=rtcp-fb:41 nack
a=rtcp-fb:41 nack pli
a=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f
a=rtpmap:42 rtx/90000
a=fmtp:42 apt=41
a=rtpmap:43 H264/90000
a=rtcp-fb:43 goog-remb
a=rtcp-fb:43 transport-cc
a=rtcp-fb:43 ccm fir
a=rtcp-fb:43 nack
a=rtcp-fb:43 nack pli
a=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f
a=rtpmap:44 rtx/90000
a=fmtp:44 apt=43
a=rtpmap:45 AV1/90000
a=rtcp-fb:45 goog-remb
a=rtcp-fb:45 transport-cc
a=rtcp-fb:45 ccm fir
a=rtcp-fb:45 nack
a=rtcp-fb:45 nack pli
a=rtpmap:46 rtx/90000
a=fmtp:46 apt=45
a=rtpmap:47 AV1/90000
a=rtcp-fb:47 goog-remb
a=rtcp-fb:47 transport-cc
a=rtcp-fb:47 ccm fir
a=rtcp-fb:47 nack
a=rtcp-fb:47 nack pli
a=fmtp:47 profile=1
a=rtpmap:48 rtx/90000
a=fmtp:48 apt=47
a=rtpmap:112 H264/90000
a=rtcp-fb:112 goog-remb
a=rtcp-fb:112 transport-cc
a=rtcp-fb:112 ccm fir
a=rtcp-fb:112 nack
a=rtcp-fb:112 nack pli
a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
a=rtpmap:113 rtx/90000
a=fmtp:113 apt=112
a=rtpmap:114 H264/90000
a=rtcp-fb:114 goog-remb
a=rtcp-fb:114 transport-cc
a=rtcp-fb:114 ccm fir
a=rtcp-fb:114 nack
a=rtcp-fb:114 nack pli
a=fmtp:114 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=64001f
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 red/90000
a=rtpmap:117 rtx/90000
a=fmtp:117 apt=116
a=rtpmap:118 ulpfec/90000
a=rtpmap:49 flexfec-03/90000
a=rtcp-fb:49 goog-remb
a=rtcp-fb:49 transport-cc
a=fmtp:49 repair-window=10000000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:PbK/
a=ice-pwd:o4+diB62YMnzEDZ19S+eAQG0
a=ice-options:trickle
a=fingerprint:sha-256 B8:05:C9:24:68:F1:87:0D:1F:B7:13:8B:8C:FE:8C:95:DB:D7:9D:2F:46:D4:AC:9D:EA:4A:43:22:39:EC:51:92
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144
a=candidate:2108489375 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 55961 typ host generation 0 ufrag PbK/ network-cost 999
a=candidate:2108489375 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 55963 typ host generation 0 ufrag PbK/ network-cost 999
a=candidate:2108489375 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 55965 typ host generation 0 ufrag PbK/ network-cost 999
16:07:36.243 TRC [rtsp] client request:
DESCRIBE rtsp://192.168.1.200:8554/e04f43a453ea_live RTSP/1.0
Require: www.onvif.org/ver20/backchannel
User-Agent: go2rtc/1.5.0
CSeq: 1
Accept: application/sdp
16:07:38.798 TRC [rtsp] client response:
RTSP/1.0 200 OK
Cseq: 1
Content-Length: 572
Content-Type: application/sdp
v=0
o=- 1 1 IN IP4 0.0.0.0
s=go2rtc/1.5.0
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z2QAKaw0yAeAIn5cBagICAoAAAfQAAE4gdDAAjhAACOEF3lxoYAEcIAARwgu8uFA,aO48MAA=; profile-level-id=640029
a=control:trackID=0
m=audio 0 RTP/AVP 97
a=rtpmap:97 MPEG4-GENERIC/48000/2
a=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=119056E500
a=control:trackID=1
m=audio 0 RTP/AVP 98
a=rtpmap:98 OPUS/48000/2
a=fmtp:98 sprop-stereo=1
a=control:trackID=2
16:07:38.798 TRC [rtsp] client request:
SETUP rtsp://192.168.1.200:8554/e04f43a453ea_live/trackID=0 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=0-1
CSeq: 2
16:07:38.798 TRC [rtsp] client response:
RTSP/1.0 200 OK
Transport: RTP/AVP/TCP;unicast;interleaved=0-1
Cseq: 2
Session: 43638195;timeout=60
16:07:38.799 TRC [rtsp] client request:
SETUP rtsp://192.168.1.200:8554/e04f43a453ea_live/trackID=2 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=4-5
CSeq: 3
Session: 43638195
16:07:38.799 TRC [rtsp] client response:
RTSP/1.0 200 OK
Transport: RTP/AVP/TCP;unicast;interleaved=4-5
Cseq: 3
Session: 81136086;timeout=60
16:07:38.799 DBG [streams] start producer url=rtsp://192.168.1.200:8554/e04f43a453ea_live
16:07:38.799 TRC [rtsp] client request:
PLAY rtsp://192.168.1.200:8554/e04f43a453ea_live RTSP/1.0
CSeq: 4
Session: 43638195
16:07:38.799 TRC [rtsp] client response:
RTSP/1.0 200 OK
Cseq: 4
Session: 81136086
16:07:38.823 TRC [webrtc] answer
v=0
o=- 2046398686863851402 1687896458 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 AE:09:D2:DD:1B:79:C9:70:67:B5:3A:43:52:A8:0A:32:4C:5D:1D:4B:3B:04:EF:AE:09:65:C8:E1:7B:42:1B:DA
a=extmap-allow-mixed
a=group:BUNDLE 0 1 2
m=audio 9 UDP/TLS/RTP/SAVPF 111 0 8
c=IN IP4 0.0.0.0
a=setup:active
a=mid:0
a=ice-ufrag:jtEVfCoTGgXcaGyo
a=ice-pwd:HvFRrHuXCRNHsoiBPOBDgJJXuPwAgQSg
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=rtcp-fb:111 transport-cc 
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:803358232 cname:go2rtc
a=ssrc:803358232 msid:go2rtc go2rtc-audio
a=ssrc:803358232 mslabel:go2rtc
a=ssrc:803358232 label:go2rtc-audio
a=msid:go2rtc go2rtc-audio
a=sendonly
a=candidate:227965208 1 udp 2130706431 192.168.1.200 36130 typ host
a=candidate:227965208 2 udp 2130706431 192.168.1.200 36130 typ host
a=candidate:3528925834 1 udp 2130706431 172.18.0.1 53330 typ host
a=candidate:3528925834 2 udp 2130706431 172.18.0.1 53330 typ host
a=candidate:574494503 1 udp 2130706431 192.168.122.1 55701 typ host
a=candidate:574494503 2 udp 2130706431 192.168.122.1 55701 typ host
a=candidate:462606888 1 udp 1694498815 73.194.113.79 60887 typ srflx raddr 0.0.0.0 rport 60887
a=candidate:462606888 2 udp 1694498815 73.194.113.79 60887 typ srflx raddr 0.0.0.0 rport 60887
a=candidate:2959411448 1 tcp 2124414975 192.168.1.200 8555 typ host tcptype passive
a=candidate:2959411448 2 tcp 2124414975 192.168.1.200 8555 typ host tcptype passive
a=candidate:1873040234 1 tcp 2124414975 172.18.0.1 8555 typ host tcptype passive
a=candidate:1873040234 2 tcp 2124414975 172.18.0.1 8555 typ host tcptype passive
a=candidate:2681038535 1 tcp 2124414975 192.168.122.1 8555 typ host tcptype passive
a=candidate:2681038535 2 tcp 2124414975 192.168.122.1 8555 typ host tcptype passive
a=end-of-candidates
m=video 9 UDP/TLS/RTP/SAVPF 102 106 112
c=IN IP4 0.0.0.0
a=setup:active
a=mid:1
a=ice-ufrag:jtEVfCoTGgXcaGyo
a=ice-pwd:HvFRrHuXCRNHsoiBPOBDgJJXuPwAgQSg
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:102 H264/90000
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtcp-fb:102 goog-remb 
a=rtcp-fb:102 transport-cc 
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack 
a=rtcp-fb:102 nack pli
a=rtpmap:106 H264/90000
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:106 goog-remb 
a=rtcp-fb:106 transport-cc 
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack 
a=rtcp-fb:106 nack pli
a=rtpmap:112 H264/90000
a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
a=rtcp-fb:112 goog-remb 
a=rtcp-fb:112 transport-cc 
a=rtcp-fb:112 ccm fir
a=rtcp-fb:112 nack 
a=rtcp-fb:112 nack pli
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:2496046327 cname:go2rtc
a=ssrc:2496046327 msid:go2rtc go2rtc-video
a=ssrc:2496046327 mslabel:go2rtc
a=ssrc:2496046327 label:go2rtc-video
a=msid:go2rtc go2rtc-video
a=sendonly
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:active
a=mid:2
a=sendrecv
a=sctp-port:5000
a=ice-ufrag:jtEVfCoTGgXcaGyo
a=ice-pwd:HvFRrHuXCRNHsoiBPOBDgJJXuPwAgQSg
16:08:33.799 TRC [rtsp] client request:
OPTIONS rtsp://192.168.1.200:8554/e04f43a453ea_live RTSP/1.0
Session: 43638195
CSeq: 5
Scope666 commented 1 year ago

Another one, just a single attempt with the Backyard Live (bad one) ... a little easier to read with spaces intact:

16:26:19.711 INF go2rtc version 1.5.0 linux/amd64
16:26:19.711 INF [api] listen addr=:1984
16:26:19.712 INF [srtp] listen addr=:8443
16:26:19.712 INF [webrtc] listen addr=:8555/tcp
16:27:05.730 TRC [webrtc] offer:
v=0
o=- 4595606351547977654 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8 13 110 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Guf8
a=ice-pwd:FHIiXbiNhbBocEB8AmuAV8c7
a=ice-options:trickle
a=fingerprint:sha-256 58:79:39:44:3D:92:1B:9E:9A:B6:A9:4E:17:C4:E7:0D:9D:8F:39:F5:E3:14:E0:5E:04:60:3E:EA:5E:3F:F6:E2
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 35 36 37 38 102 103 104 105 106 107 108 109 127 125 39 40 41 42 43 44 45 46 47 48 112 113 114 115 116 117 118 49
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Guf8
a=ice-pwd:FHIiXbiNhbBocEB8AmuAV8c7
a=ice-options:trickle
a=fingerprint:sha-256 58:79:39:44:3D:92:1B:9E:9A:B6:A9:4E:17:C4:E7:0D:9D:8F:39:F5:E3:14:E0:5E:04:60:3E:EA:5E:3F:F6:E2
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:35 VP9/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=fmtp:35 profile-id=1
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:37 VP9/90000
a=rtcp-fb:37 goog-remb
a=rtcp-fb:37 transport-cc
a=rtcp-fb:37 ccm fir
a=rtcp-fb:37 nack
a=rtcp-fb:37 nack pli
a=fmtp:37 profile-id=3
a=rtpmap:38 rtx/90000
a=fmtp:38 apt=37
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:103 rtx/90000
a=fmtp:103 apt=102
a=rtpmap:104 H264/90000
a=rtcp-fb:104 goog-remb
a=rtcp-fb:104 transport-cc
a=rtcp-fb:104 ccm fir
a=rtcp-fb:104 nack
a=rtcp-fb:104 nack pli
a=fmtp:104 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:105 rtx/90000
a=fmtp:105 apt=104
a=rtpmap:106 H264/90000
a=rtcp-fb:106 goog-remb
a=rtcp-fb:106 transport-cc
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack
a=rtcp-fb:106 nack pli
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=106
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
a=rtpmap:125 rtx/90000
a=fmtp:125 apt=127
a=rtpmap:39 H264/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:41 H264/90000
a=rtcp-fb:41 goog-remb
a=rtcp-fb:41 transport-cc
a=rtcp-fb:41 ccm fir
a=rtcp-fb:41 nack
a=rtcp-fb:41 nack pli
a=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f
a=rtpmap:42 rtx/90000
a=fmtp:42 apt=41
a=rtpmap:43 H264/90000
a=rtcp-fb:43 goog-remb
a=rtcp-fb:43 transport-cc
a=rtcp-fb:43 ccm fir
a=rtcp-fb:43 nack
a=rtcp-fb:43 nack pli
a=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f
a=rtpmap:44 rtx/90000
a=fmtp:44 apt=43
a=rtpmap:45 AV1/90000
a=rtcp-fb:45 goog-remb
a=rtcp-fb:45 transport-cc
a=rtcp-fb:45 ccm fir
a=rtcp-fb:45 nack
a=rtcp-fb:45 nack pli
a=rtpmap:46 rtx/90000
a=fmtp:46 apt=45
a=rtpmap:47 AV1/90000
a=rtcp-fb:47 goog-remb
a=rtcp-fb:47 transport-cc
a=rtcp-fb:47 ccm fir
a=rtcp-fb:47 nack
a=rtcp-fb:47 nack pli
a=fmtp:47 profile=1
a=rtpmap:48 rtx/90000
a=fmtp:48 apt=47
a=rtpmap:112 H264/90000
a=rtcp-fb:112 goog-remb
a=rtcp-fb:112 transport-cc
a=rtcp-fb:112 ccm fir
a=rtcp-fb:112 nack
a=rtcp-fb:112 nack pli
a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
a=rtpmap:113 rtx/90000
a=fmtp:113 apt=112
a=rtpmap:114 H264/90000
a=rtcp-fb:114 goog-remb
a=rtcp-fb:114 transport-cc
a=rtcp-fb:114 ccm fir
a=rtcp-fb:114 nack
a=rtcp-fb:114 nack pli
a=fmtp:114 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=64001f
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 red/90000
a=rtpmap:117 rtx/90000
a=fmtp:117 apt=116
a=rtpmap:118 ulpfec/90000
a=rtpmap:49 flexfec-03/90000
a=rtcp-fb:49 goog-remb
a=rtcp-fb:49 transport-cc
a=fmtp:49 repair-window=10000000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:Guf8
a=ice-pwd:FHIiXbiNhbBocEB8AmuAV8c7
a=ice-options:trickle
a=fingerprint:sha-256 58:79:39:44:3D:92:1B:9E:9A:B6:A9:4E:17:C4:E7:0D:9D:8F:39:F5:E3:14:E0:5E:04:60:3E:EA:5E:3F:F6:E2
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144
a=candidate:1769305400 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 54535 typ host generation 0 ufrag Guf8 network-cost 999
a=candidate:1769305400 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 54537 typ host generation 0 ufrag Guf8 network-cost 999
a=candidate:1769305400 1 udp 2113937151 68d6397c-7980-481c-920b-3b4dee2d8bd9.local 54539 typ host generation 0 ufrag Guf8 network-cost 999

16:27:05.730 TRC [rtsp] client request:
DESCRIBE rtsp://192.168.1.200:8554/44ead8254f04_live RTSP/1.0
Accept: application/sdp
Require: www.onvif.org/ver20/backchannel
User-Agent: go2rtc/1.5.0
CSeq: 1

16:27:07.700 TRC [rtsp] client response:
RTSP/1.0 200 OK
Content-Type: application/sdp
Cseq: 1
Content-Length: 580

v=0
o=- 1 1 IN IP4 0.0.0.0
s=go2rtc/1.5.0
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z2QAKaw0yAeAIn5f/AL0AvmoCAgKAAAH0AABOIHQwAI4QAAjhBd5caGABHCAAEcILvLhQA==,aO48MAA=; profile-level-id=640029
a=control:trackID=0
m=audio 0 RTP/AVP 97
a=rtpmap:97 MPEG4-GENERIC/48000/2
a=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=119056E500
a=control:trackID=1
m=audio 0 RTP/AVP 98
a=rtpmap:98 OPUS/48000/2
a=fmtp:98 sprop-stereo=1
a=control:trackID=2

16:27:07.701 TRC [rtsp] client request:
SETUP rtsp://192.168.1.200:8554/44ead8254f04_live/trackID=0 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=0-1
CSeq: 2

16:27:07.701 TRC [rtsp] client response:
RTSP/1.0 200 OK
Cseq: 2
Session: 27961238;timeout=60
Transport: RTP/AVP/TCP;unicast;interleaved=0-1

16:27:07.701 TRC [rtsp] client request:
SETUP rtsp://192.168.1.200:8554/44ead8254f04_live/trackID=2 RTSP/1.0
Transport: RTP/AVP/TCP;unicast;interleaved=4-5
CSeq: 3
Session: 27961238

16:27:07.701 TRC [rtsp] client response:
RTSP/1.0 200 OK
Transport: RTP/AVP/TCP;unicast;interleaved=4-5
Cseq: 3
Session: 74771758;timeout=60

16:27:07.701 DBG [streams] start producer url=rtsp://192.168.1.200:8554/44ead8254f04_live
16:27:07.701 TRC [rtsp] client request:
PLAY rtsp://192.168.1.200:8554/44ead8254f04_live RTSP/1.0
CSeq: 4
Session: 27961238

16:27:07.701 TRC [rtsp] client response:
RTSP/1.0 200 OK
Cseq: 4
Session: 74771758

16:27:07.738 TRC [webrtc] answer
v=0
o=- 590750052890136842 1687897627 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 74:63:B0:CB:BF:B9:BE:28:71:88:5D:6C:86:38:34:F3:1C:50:CB:C3:47:C1:BB:1A:7E:93:DC:E2:BC:41:B0:72
a=extmap-allow-mixed
a=group:BUNDLE 0 1 2
m=audio 9 UDP/TLS/RTP/SAVPF 111 0 8
c=IN IP4 0.0.0.0
a=setup:active
a=mid:0
a=ice-ufrag:rNIXFhoCCkuDbkFB
a=ice-pwd:auqoXOoCGtMyQgufezvyskYeWleKQYLf
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=rtcp-fb:111 transport-cc
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:2517043649 cname:go2rtc
a=ssrc:2517043649 msid:go2rtc go2rtc-audio
a=ssrc:2517043649 mslabel:go2rtc
a=ssrc:2517043649 label:go2rtc-audio
a=msid:go2rtc go2rtc-audio
a=sendonly
a=candidate:227965208 1 udp 2130706431 192.168.1.200 38027 typ host
a=candidate:227965208 2 udp 2130706431 192.168.1.200 38027 typ host
a=candidate:3528925834 1 udp 2130706431 172.18.0.1 43497 typ host
a=candidate:3528925834 2 udp 2130706431 172.18.0.1 43497 typ host
a=candidate:574494503 1 udp 2130706431 192.168.122.1 45650 typ host
a=candidate:574494503 2 udp 2130706431 192.168.122.1 45650 typ host
a=candidate:462606888 1 udp 1694498815 73.194.113.79 43660 typ srflx raddr 0.0.0.0 rport 43660
a=candidate:462606888 2 udp 1694498815 73.194.113.79 43660 typ srflx raddr 0.0.0.0 rport 43660
a=candidate:2959411448 1 tcp 2124414975 192.168.1.200 8555 typ host tcptype passive
a=candidate:2959411448 2 tcp 2124414975 192.168.1.200 8555 typ host tcptype passive
a=candidate:1873040234 1 tcp 2124414975 172.18.0.1 8555 typ host tcptype passive
a=candidate:1873040234 2 tcp 2124414975 172.18.0.1 8555 typ host tcptype passive
a=candidate:2681038535 1 tcp 2124414975 192.168.122.1 8555 typ host tcptype passive
a=candidate:2681038535 2 tcp 2124414975 192.168.122.1 8555 typ host tcptype passive
a=end-of-candidates
m=video 9 UDP/TLS/RTP/SAVPF 102 106 112
c=IN IP4 0.0.0.0
a=setup:active
a=mid:1
a=ice-ufrag:rNIXFhoCCkuDbkFB
a=ice-pwd:auqoXOoCGtMyQgufezvyskYeWleKQYLf
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:102 H264/90000
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=rtpmap:106 H264/90000
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:106 goog-remb
a=rtcp-fb:106 transport-cc
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack
a=rtcp-fb:106 nack pli
a=rtpmap:112 H264/90000
a=fmtp:112 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f
a=rtcp-fb:112 goog-remb
a=rtcp-fb:112 transport-cc
a=rtcp-fb:112 ccm fir
a=rtcp-fb:112 nack
a=rtcp-fb:112 nack pli
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=ssrc:2313221689 cname:go2rtc
a=ssrc:2313221689 msid:go2rtc go2rtc-video
a=ssrc:2313221689 mslabel:go2rtc
a=ssrc:2313221689 label:go2rtc-video
a=msid:go2rtc go2rtc-video
a=sendonly
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:active
a=mid:2
a=sendrecv
a=sctp-port:5000
a=ice-ufrag:rNIXFhoCCkuDbkFB
a=ice-pwd:auqoXOoCGtMyQgufezvyskYeWleKQYLf

16:27:21.945 DBG [streams] stop producer url=rtsp://192.168.1.200:8554/44ead8254f04_live
16:27:21.945 TRC [rtsp] client request:
TEARDOWN rtsp://192.168.1.200:8554/44ead8254f04_live RTSP/1.0
CSeq: 5
Session: 27961238
tsightler commented 1 year ago

Unfortunately, a line-by-line comparison between your non-working session, and my working session, is basically identical. Only differences being the things you would expect to be different (signatures, dynamic ports, IP addresses of peers).

It is clear that go2rtc is indeed responding with the actual stream profile because it starts the stream first and then responds (you can see the delay), so it's clear that your one camera is indeed negotiating a H.264 Main profile instead of High, but I don't really understand why as the same camera negotiated high profile for me when you shared it.

Unfortunately, I've run out of ideas, and since the problem seems isolated to go2rtc, and not ring-mqtt itself, as RTSPtoWeb works, as does go2rtc on another server, etc., I'm going to have to ask that you open an issue on the go2rtc project and see if the maintainer there can provide any possible clues based on your traces. If you link the case here I can also follow and participate.

Scope666 commented 1 year ago

I appreciate you going above and beyond on this, and also I wanted to let you know that I think ring-mqtt is one of the most amazing projects that I use with Home Assistant. It added SO much more functionality, and I even got rid of the official integration long ago, because ring-mqtt does everything it did and more.

tsightler commented 1 year ago

Issue was eventually tracked to a network switch. Comment from @Scope666 :

OK, it gets weirder... the TP Link switch has my old Asus Router plugged into it, I use it for an additional 5Ghz AP for that side of the house. I temp changed it's SSID so I could pin myself to it, and WebRTC works to my laptop! Now that's still going though the TP Link switch, but it's not going back to the main router over that long ethernet run at that point.

I just removed go2rtc from the Win11 VM, put it back in Unraid Docker, streams are still playing if I connect to the Asus's Wi-Fi instead of the Ubiquiti.

So at this point it's either a bug in the router or APs, or I have a setting wrong somewhere. WebRTC is definitely good at the server, and out the TP Link to the Asus AP, but it's dying on the way back to the router, or in the router itself.

Since you run Ubiquiti too, if you have any ideas for settings to look at, please let me know! I feel like we're getting close now...

EDIT: As a test I took the long ethernet run out of the TP Link and put it straight into the Unraid box, all cameras work with WebRTC. I then had the idea to go uplink -- Asus router switch port -- Unraid box. THAT seems to be working, so I think the culprit is the TP Link switch!!!

EDIT2: So that was it, all this time. That stupid TP Link switch, the timing makes sense too from when I started having problems. Now why the 3 cameras, but not the 4th, and only with go2rtc and not RTSPtoWEB, we'll never know.

Scope666 commented 1 year ago

Your theory about the buffer size and the burst makes total sense to me. If you think about it, using the Asus switch ports avoids a lot of the "smart" crap, it just works. I would still consider this a bug or flaw in this TP Link switch, it should be able to handle a scenario like that considering modern streaming technologies.