search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.24k stars 1.76k forks source link

[Config Support]: Using wyze webrtc stream with go2rtc #7529

Closed kevkid closed 1 year ago

kevkid commented 1 year ago

Describe the problem you are having

What I am trying to achieve:

Get low latency, high quality live view with audio -> Saves to video with the same quality + audio.

Context:

I have several wyze cameras (2 v3 and 2 Pan v3). I want to use webrtc as a source to get the lowest latency. I am running docker-wyze-bridge to grab the streams from my camera. I can output via rtsp stream or webrtc stream. Originally, I used the rtsp firmware, but found out it was deprecated. So now I am using the wyze-bridge

Current Setup:

I currently am getting the rtsp stream from the bridge. It works fine, albeit some latency and some stuttering. Not the worst, but can be improved. I have to encode the audio to AAC in the bridge, this introduces a little bit of latency. In order to get a low latency live view I need either MSE or webrtc, the jsmpeg is slow and capped at 10fps with no audio. To get these to work I need to enable go2rtc and restream the input stream coming from the bridge (which is restreaming the input coming from the camera). I then use the restream output from go2rtc as the input to the camera. I do this for 4 cameras.

Thoughts:

I believe the fact that I have to restream/enable go2rtc is causing latency. I think that while I do get a lower latency, higher quality stream with audio compared jsmpeg there is room for improvement. I believe using the webrtc stream will improve the stability, latency, and quality of the feed. I would like to use webrtc as a source. Do we really have to restream to get access to MSE/webrtc view?

### Visual Diagram:

Camera 2 ------------------\
Camera 1 ------------------( Wyze Bridge ) =====webrtc/rtsp==== ( Frigate )

(I have built the docker image from the source code as I could not find a :latest tag and wanted to try out the latest go2rtc enabling webrtc input)

I tried setting the go2rtc stream to the following for back_side and you will see the output in the logs:

# -----------go2rtc------------
go2rtc:
  streams:
    back_side: webrtc:http://<wyze-bridge-ip>:8889/backyard-side-camera

Version

0.13.0-7C629C1

Frigate config file

ui:
  use_experimental: false
  live_mode: mse
mqtt:
  host: 0.0.0.0
  port: 1883
detectors:
  coral:
    type: edgetpu
    device: usb
  #cpu:
  #  type: cpu
  #  num_threads: 2
cameras:
  front_porch: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/front_porch # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_porch_sub # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      # Optional: desired fps for your camera for the input with the detect role (default: shown below)
      # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
      fps: 5
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution
  back_yard: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/back_yard # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/back_yard_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      # Optional: desired fps for your camera for the input with the detect role (default: shown below)
      # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
      fps: 5
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution

  front_yard: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/front_yard # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_yard # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      # Optional: desired fps for your camera for the input with the detect role (default: shown below)
      # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
      fps: 5
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution

  back_side: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://127.0.0.1:8554/back_side # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/back_side # <----- Update for your camera
          input_args: preset-rtsp-restream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      # Optional: desired fps for your camera for the input with the detect role (default: shown below)
      # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
      fps: 5
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution

# -----------go2rtc------------
go2rtc:
  streams:
    #back_side: webrtc:http://<wyze-bridge-ip>:8889/backyard-side-camera
    back_yard:
      - rtsp://127.0.0.1:8557/back-yard-cam
    back_yard_sub:
      - rtsp://127.0.0.1:8557/back-yard-cam-sub
    front_porch:
      - rtsp://127.0.0.1:8557/front-porch-cam
    front_porch_sub:
      - rtsp://127.0.0.1:8557/front-porch-cam-sub
    front_yard:
      - rtsp://127.0.0.1:8557/front-yard-cam
    back_side:
      - rtsp://127.0.0.1:8557/backyard-side-camera
#  webrtc:
#    candidates:
#      - wyze-bridge-ip:8555
#      - stun:8555
objects:
  track:
    - person
    - car
    - dog
    - cat

record:
  # Optional: Enable recording (default: shown below)
  # WARNING: If recording is disabled in the config, turning it on via
  #          the UI or MQTT later will have no effect.
  # WARNING: Frigate does not currently support limiting recordings based
  #          on available disk space automatically. If using recordings,
  #          you must specify retention settings for a number of days that
  #          will fit within the available disk space of your drive or Frigate
  #          will crash.
  enabled: True
  # Optional: Number of minutes to wait between cleanup runs (default: shown below)
  # This can be used to reduce the frequency of deleting recording segments from disk if you want to minimize i/o
  expire_interval: 60
  # Optional: Retention settings for recording
  retain:
    # Optional: Number of days to retain recordings regardless of events (default: shown below)
    # NOTE: This should be set to 0 and retention should be defined in events section below
    #       if you only want to retain recordings of events.
    days: 14
    # Optional: Mode for retention. Available options are: all, motion, and active_objects
    #   all - save all recording segments regardless of activity
    #   motion - save all recordings segments with any detected motion
    #   active_objects - save all recording segments with active/moving objects
    # NOTE: this mode only applies when the days setting above is greater than 0
    mode: all
  # Optional: Event recording settings
  events:
    # Optional: Maximum length of time to retain video during long events. (default: shown below)
    # NOTE: If an object is being tracked for longer than this amount of time, the retained recordings
    #       will be the last x seconds of the event unless retain->days under record is > 0.
    # max_seconds: 300
    # Optional: Number of seconds before the event to include (default: shown below)
    pre_capture: 15
    # Optional: Number of seconds after the event to include (default: shown below)
    post_capture: 15
    # Optional: Objects to save recordings for. (default: all tracked objects)
    objects:
      - person
      - car
      - dog
      - cat
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
    required_zones: []
    # Optional: Retention settings for recordings of events
    retain:
      # Required: Default retention days (default: shown below)
      default: 14
      # Optional: Mode for retention. (default: shown below)
      #   all - save all recording segments for events regardless of activity
      #   motion - save all recordings segments for events with any detected motion
      #   active_objects - save all recording segments for event with active/moving objects
      #
      # NOTE: If the retain mode for the camera is more restrictive than the mode configured
      #       here, the segments will already be gone by the time this mode is applied.
      #       For example, if the camera retain mode is "motion", the segments without motion are
      #       never stored, so setting the mode to "all" here won't bring them back.
      mode: motion
      # Optional: Per object retention days
      #objects:
      #  person: 15
ffmpeg:
  output_args:
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac

Relevant log output

2023-08-20 00:45:08.063340500  00:45:08.063 WRN [rtsp] error="SetRemoteDescription called with no ice-ufrag" stream=back_side
frigate  | 2023-08-20 00:45:08.065381429  [2023-08-20 00:45:08] frigate.video                  ERROR   : back_side: Unable to read frames from ffmpeg process.
frigate  | 2023-08-20 00:45:08.065663335  [2023-08-20 00:45:08] frigate.video                  ERROR   : back_side: ffmpeg process is not running. exiting capture thread...

Frigate stats

No response

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

Using the bridge I can access webrtc streams directly in the browser with much less latency. I can even get audio from it if I encode it with opus. I get a url like this: http://<wyze-bridge-ip>:8889/backyard-side-camera

From my understanding, using the latest go2rtc, we can use webrtc as a stream input as shown by this pr: https://github.com/blakeblackshear/frigate/pull/7250 and this readme: https://github.com/AlexxIT/go2rtc#source-webrtc

NickM-27 commented 1 year ago

A few things here:

  1. You can do AAC audio transcoding for frigate recordings only and not need to do it for live view. In 0.13 PCM audio can provide audio for MSE and WebRTC
  2. You can also enable opus transcoding for audio in frigate's webRTC like the docs explain
  3. Your example for using wyze webrtc is not following the go2rtc docs for how to do that, it not working has nothing to do with frigate

(I have built the docker image from the source code as I could not find a :latest tag and wanted to try out the latest go2rtc enabling webrtc input)

There is no need to do this, Frigate has dev images already built and available on the repo

kevkid commented 1 year ago

Hi @NickM-27 Thank you for the response. So This is correct in 0.13 PCM audio works for both MSE and WebRTC as you stated in point 1. But I am not so sure about using PCM audio directly from the camera and then add that as an input to the go2rtc, that it will pass the audio to the camera -> recording. Here is an example of what my connection looked like:

rtsp://127.0.0.1:8557/back-yard-cam

I also tried: ffmpeg:rtsp://127.0.0.1:8557/back-yard-cam#video=copy#audio=copy

and sure the live view would be fine, and even viewing the RTSP stream in vlc direct from the bridge gives audio (VLC confirms the audio track). Where as if we pull the RTSP stream from the retransmit it does not include an audio channel (according to VLC). I was perplexed by this, but I suspect it has to do with how it packages the restream (?) Not completely clear why it drops the audio channel (maybe something to do with the fact that mp4 cant do PCM? not sure). But I know for a fact that even though my output args looked like: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac There would be no audio track on the video.

Now my stream looks like:

go2rtc:
  streams:
    # back_yard:    webrtc:http://<bridge-ip>:5000/signaling/back-yard-cam?kvs#format=wyze  # <--This is the correct format for webrtc input as desired by the original post
    back_yard: ffmpeg:rtsp://127.0.0.1:8557/back-yard-cam#video=copy#audio=aac

and my output args look like:

  output_args:
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy

for 2, yes this is correct. would look something like: ffmpeg:rtsp://127.0.0.1:8557/back-yard-cam#video=copy#audio=opus for 3 you are also correct. The answer I got to was: 

go2rtc:
  streams:
    # back_yard:    webrtc:http://<bridge-ip>:5000/signaling/back-yard-cam?kvs#format=wyze

But Interestingly, I can avoid all of this transcoding if I could output to MKV rather than MP4 as mp4 limits the types of audio I can use (to aac). If I could just mux the stream into mkv directly and view it (even if without audio on the web interface). Is there a way to do this?

NickM-27 commented 1 year ago

This is because frigate configured go2rtc to only return AAC audio by default. Simply add ?video&audio to the end of the rtsp restream url and the audio will be included.

Is there a way to do this?

No, MP4 only is supported for a number of reasons.

kevkid commented 1 year ago

Thank you for your help. Closing