Cannot use hardware acceleration to split stream

[felipecrs]

I just bought a new camera: https://aliexpress.com/item/1005005731238053.html

It has 2 lens, and it outputs both in a single RTSP stream, like this:

So, I want to crop the stream into two. I managed to do it with:

streams:
  test_dual:
    - rtsp://192.168.1.160:554/user=admin_password=admin_channel=0_stream=1&onvif=0.sdp?real_stream
  test_top:
    # works, just to check if hardware accel is working
    # - ffmpeg:nova_dupla#audio=copy#video=h264#hardware=vaapi
    # works, but very slow as no hardware accel
    # - ffmpeg:nova_dupla#audio=copy#video=h264#raw=-filter_complex [0]crop=iw:ih/2:0:0
    # does not work with hardware accel
    # - ffmpeg:nova_dupla#audio=copy#video=h264#hardware=vaapi#raw=-filter_complex [0]crop=iw:ih/2:0:0

Unfortunately hardware acceleration cannot work together with -filter_complex, apparently. Error in logs:

14:47:50.910 ERR github.com/AlexxIT/go2rtc/internal/mjpeg/init.go:178 > error="exec: exit status 1"

Then:

023-08-12 17:59:49.629580580  14:59:49.629 DBG [exec] run url="exec:ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/test_dual?video&audio -filter_complex [0]crop=iw:ih/2:0:0 -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf \"format=vaapi|nv12,hwupload\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-08-12 17:59:51.121899571  14:59:51.121 DBG [exec] run launch=2.984648115s
2023-08-12 17:59:51.597230374  Filtergraph 'format=vaapi|nv12,hwupload' was specified through the -vf/-af/-filter option for output stream 0:0, which is fed from a complex filtergraph.
2023-08-12 17:59:51.597239168  -vf/-af/-filter and -filter_complex cannot be used together for the same stream.
2023-08-12 17:59:51.599419782  14:59:51.599 DBG [exec] run url="exec:ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/test_dual?video&audio -filter_complex [0]crop=iw:ih/2:0:0 -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf \"format=vaapi|nv12,hwupload\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"

Camera stream details:

{
  "producers": [
    {
      "type": "RTSP active producer",
      "url": "rtsp://192.168.1.160:554/user=admin_password=admin_channel=0_stream=1&onvif=0.sdp/",
      "remote_addr": "192.168.1.160:554",
      "user_agent": "go2rtc/1.6.2",
      "medias": ["video, recvonly, H265", "audio, recvonly, PCMA/8000"],
      "receivers": [
        "98 H265, bytes=518351, senders=1",
        "8 PCMA/8000, bytes=221120, senders=1"
      ],
      "recv": 753919
    }
  ],
  "consumers": [
    {
      "type": "RTSP passive consumer",
      "url": "rtsp://127.0.0.1:8554/test_dual?video&audio",
      "remote_addr": "127.0.0.1:60892",
      "user_agent": "go2rtc/ffmpeg",
      "medias": ["video, sendonly, ANY", "audio, sendonly, ANY"],
      "senders": [
        "96 H265, bytes=518351, receivers=1",
        "97 PCMA/8000, bytes=221120, receivers=1"
      ],
      "send": 758095
    }
  ]
}

So I wonder if there is some issue in go2rtc.

[felipecrs]

By the way, it would be awesome if go2rtc had support for splitting the stream into two in a more integrated way:

1. Simplified configuration, maybe something like #crop=top and #crop=bottom
2. Somehow allow to capture the two output streams from the same FFMPEG process, to avoid transcoding twice, like this:

ffmpeg -i input -filter_complex "[0]crop=iw:ih/2:0:0[top];[0]crop=iw:ih/2:0:oh[bottom]" -map "[top]" top.mp4 -map "[bottom]" bottom.mp4

But directly mapping each output to different go2rtc streams.

[AlexxIT]

Hard task. Will check someday

[Vistritium]

@felipecrs I'm facing the same issue. Please update if you resolved this issue somehow or had other workaround.

[felipecrs]

I did not resolve neither found any workaround.

[felipecrs]

1. Simplified configuration, maybe something like #crop=top and #crop=bottom

Maybe more like #split=vertically or #split=horizontally.

[felipecrs]

@AlexxIT did you mean that allowing to use #raw=-filter_complex [0]crop=iw:ih/2:0:0 with #hardware is a hard task, or did you mean that capturing the two outputs from a single ffmpeg command is a hard task?

If the first one worked, it would already make my life better.

[AlexxIT]

Both. Split input via hardware. And handle two outputs.

[felipecrs]

Ok, I have some news.

1. #raw=-vf crop=in_w:in_h/2:0:in_h/2 I came up with this alternative command which is not filter_complex
2. #hardware still doesn't work, but now instead of failing the stream, the filter simply doesn't get applied anymore it seems.

I can still use #hardware to do H265 to H264 transcoding and then applying the filter on the transcoded result, to minimize my CPU impact (given that I needed to transcode first).

[tibmeister]

How do you do the transcoding first then apply the filter using go2rtc? Do you have to setup mutliple streams or something?

[felipecrs]

How do you do the transcoding first then apply the filter using go2rtc? Do you have to setup mutliple streams or something?

That's correct. First stream to do the transcoding, then another stream consuming from the first stream.

[giacomitti]

@felipecrs do you are using the split of two cameras in Frigate? In first 15-20min the cpu is around 50%, after it it´s go to 100% and not decrease.

[felipecrs]

I no longer am. I replaced the camera with a 16:9 one.

But when I was, using the approach mentioned above (first transcode with hardware acceleration and then split), CPU usage was not a problem.

Intel N100