search

AlexxIT / go2rtc

Ultimate camera streaming application with support RTSP, RTMP, HTTP-FLV, WebRTC, MSE, HLS, MP4, MJPEG, HomeKit, FFmpeg, etc.
https://github.com/AlexxIT/Blog
MIT License
6.94k stars 496 forks source link

Nest source encounters 409 and retry attempts hit 429 rate-limit #1397

Open trvrnrth opened 1 month ago

trvrnrth commented 1 month ago

I have a nest hello gen 1 wired doorbell source which, after some time experiences a 409 response:

error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door

It appears that there is no back-off on attempts to re-connect so these are then rate-limited:

error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door

The full error message grabbed from the Device Access Sample App at this time is:

{
  "error": {
    "code": 429,
    "message": "Rate limited for the GenerateWebRtcStream API for the user.",
    "status": "RESOURCE_EXHAUSTED"
  }
}

At this point a restart is not sufficient to re-connect. go2rtc must be stopped for some time to allow the rate limit to "clear" before being started again (in my case this is a case of manually stopping, waiting a few minutes then starting the HA Frigate add-on).

I am unclear what causes the initial 409 conflict response, but implementing a back-off on at least the 429 response would be good. I guess it might make sense to just back-off on any error which would lessen the chance of hitting the rate limit in the first place.

Here is a full extract from the logs at the time of last failure:

2024-10-12 15:16:29.799077795  16:16:29.798 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:front_door#audio=aac
2024-10-12 15:16:50.871397445  16:16:50.871 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:16:55.987067326  16:16:55.986 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:01.104080094  16:17:01.103 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:06.220382167  16:17:06.220 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:11.337320055  16:17:11.337 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:16.453962537  16:17:16.453 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:21.569329842  16:17:21.569 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:26.684904880  16:17:26.684 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:31.799685805  16:17:31.799 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:36.916763767  16:17:36.916 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:42.031527043  16:17:42.031 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:47.146558111  16:17:47.146 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:52.264210419  16:17:52.264 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:17:57.380472619  16:17:57.380 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:02.495376712  16:18:02.495 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:07.613469154  16:18:07.613 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:12.728694209  16:18:12.728 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:17.846215123  16:18:17.845 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:22.962979410  16:18:22.962 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:28.079459957  16:18:28.079 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:33.194993369  16:18:33.194 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:38.313387009  16:18:38.313 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:43.428449201  16:18:43.428 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:48.543098321  16:18:48.542 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:53.660881178  16:18:53.660 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:18:58.779714906  16:18:58.779 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:03.892740883  16:19:03.892 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:09.009398444  16:19:09.009 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:14.128929625  16:19:14.128 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:19.246458352  16:19:19.246 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:24.360992865  16:19:24.360 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:29.477766616  16:19:29.477 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:34.593178617  16:19:34.592 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:39.709458289  16:19:39.709 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:44.825299325  16:19:44.825 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:49.941154786  16:19:49.940 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:19:55.060783500  16:19:55.060 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:00.177158992  16:20:00.177 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:00.177373129  16:20:00.177 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37294: write: broken pipe"
2024-10-12 15:20:05.294597204  16:20:05.293 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:05.294615471  16:20:05.294 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37302: write: broken pipe"
2024-10-12 15:20:10.410614638  16:20:10.410 WRN [rtsp] error="streams: nest: wrong status: 409 Conflict, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:10.410963374  16:20:10.410 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37314: write: broken pipe"
2024-10-12 15:20:15.525828154  16:20:15.525 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:20.643628744  16:20:20.643 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:25.758949395  16:20:25.758 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:30.874456050  16:20:30.874 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:30.874686424  16:20:30.874 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45290: write: broken pipe"
2024-10-12 15:20:35.989727907  16:20:35.989 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:35.990225268  16:20:35.989 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45292: write: broken pipe"
2024-10-12 15:20:41.106395530  16:20:41.106 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:41.106723038  16:20:41.106 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:45308: write: broken pipe"
2024-10-12 15:20:46.222196430  16:20:46.221 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:46.222541964  16:20:46.222 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37156: write: broken pipe"
2024-10-12 15:20:51.336820870  16:20:51.336 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:51.336971870  16:20:51.336 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37166: write: broken pipe"
2024-10-12 15:20:56.452101686  16:20:56.451 WRN [rtsp] error="streams: nest: wrong status: 429 Too Many Requests, exec: rtsp://127.0.0.1:8554/front_door?audio: Invalid data found when processing input\n" stream=front_door
2024-10-12 15:20:56.452461684  16:20:56.452 WRN github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:225 > error="write tcp 127.0.0.1:8554->127.0.0.1:37168: write: broken pipe"
trvrnrth commented 3 weeks ago

I had a bit of time to do some local debugging of this and realised I'd not provided my stream config in the issue above:

streams:
  front_door:
    - nest:?client_id=REDACTED&client_secret=REDACTED&device_id=REDACTED&project_id=REDACTED&refresh_token=REDACTED
    - "ffmpeg:front_door#audio=aac

The testing I've done suggests that the fixed 5 second retry I'm seeing stems from the ffmpeg audio transcode I have in place so I am currently testing with a 60 second timeout as follows to see if this helps:

      - "ffmpeg:front_door#audio=aac#input=-fflags nobuffer -flags low_delay -timeout 60000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i {input}"
softcoder commented 1 week ago

Did this ever get resolved? I'm trying this inn the go2rtc addon in the frigate docker container and can't seem to get my nest cam working?

trvrnrth commented 1 week ago

Adjusting the timeouts (in both the ffmpeg transcode and also externally in frigate) mitigates the 429 rate limiting, however I am still seeing the stream getting stuck receiving 409 responses (now at 60 second intervals).

At this time I can replicate the responses in the Device Access Sample App as follows:

{
  "error": {
    "code": 409,
    "message": "Failed to retrieve answer SDP due to timeout, please try again.",
    "status": "ABORTED"
  }
}

Restarting appears to be sufficient to resolve whatever the underlying issue is, although on my last restart I did see a single 409 response before the stream started working again.

hnws commented 1 week ago

@AlexxIT may I know that if I want to add this exponential backoff ability, or maybe even restart the nest subscription, which file should I start looking into?

AlexxIT commented 1 week ago

Related to nest files: