rtsp source issues in 1.9.3 and 1.9.4

kirsch33 commented 1 week ago

another user posted this on the Frigate repo and i've been experiencing the same issue myself. confirmed everything is fine running 1.9.2 or before. both 1.9.3 and 1.9.4 have the issue alluded to below: https://github.com/blakeblackshear/frigate/discussions/12117

I can provide my own logs this evening as well to help diagnose the issue.

I scanned open issues and didnt see one that appeared to be related to this--but if I missed it and one already exists my bad.

ofcedwards commented 1 week ago

I'm able to consistently reproduce this. Let me know if there's a certain way I should set logging to provide useful information. From what I've looked at on my own, I can't see anything that would be the culprit.

dagleaves commented 6 days ago

Since you are using an audio stream for webrtc, do you notice the number of passive consumers for the audio increasing and never clearing until the crash?

I am noticing a similar issue where the memory will climb as the number of consumers continues to climb until (for me) go2rtc crashes and restarts. I don't run out of memory though.

dagleaves commented 6 days ago

Related to this, see if you see this in your go2rtc logs

Normal audio producer stop

11:22:22.011 AM | debug | [streams] stop producer url=ffmpeg:CAMERA-1#audio=opus
11:22:22.011 AM | trace | [exec] kill with signal=9
11:22:22.017 AM | debug | [rtsp] handle error=EOF  # <-- this is sometimes not there

Producer stop that keeps an audio consumer up for me

11:22:22.120 AM | debug | [streams] stop producer url=ffmpeg:CAMERA-2#audio=opus
11:22:22.120 AM | trace | [exec] kill with signal=9
11:22:22.125 AM | debug | [rtsp] handle error=read tcp 127.0.0.1:8554->127.0.0.1:45878: read: connection reset by peer

Note: 45878 is the port listed for the passive audio rtsp+tcp consumer that I see for FFmpeg receiving the aac audio from the camera.

Error that causes stream to die and not come back up for me

11:22:22.125 AM | debug | [rtsp] handle error="read tcp 127.0.0.1:8554->127.0.0.1:56678: i/o timeout"

This happens here: https://github.com/AlexxIT/go2rtc/blob/a4885c2c3abce58074d04878bba0d72105642a9b/pkg/rtsp/conn.go#L143-L147

I have found that using the direct camera connection URL instead of the go2rtc stream for the opus audio ffmpeg stream fixes my issues, but slows down the connection so there is a lot of delay when first connecting. Something happening with the go2rtc audio set up that causes timeout issues. Haven't figured out where yet.

AlexxIT / go2rtc