Cannot use hardware acceleration to split stream

[felipecrs]

I just bought a new camera: https://aliexpress.com/item/1005005731238053.html

It has 2 lens, and it outputs both in a single RTSP stream, like this:

So, I want to crop the stream into two. I managed to do it with:

streams:
  test_dual:
    - rtsp://192.168.1.160:554/user=admin_password=admin_channel=0_stream=1&onvif=0.sdp?real_stream
  test_top:
    # works, just to check if hardware accel is working
    # - ffmpeg:nova_dupla#audio=copy#video=h264#hardware=vaapi
    # works, but very slow as no hardware accel
    # - ffmpeg:nova_dupla#audio=copy#video=h264#raw=-filter_complex [0]crop=iw:ih/2:0:0
    # does not work with hardware accel
    # - ffmpeg:nova_dupla#audio=copy#video=h264#hardware=vaapi#raw=-filter_complex [0]crop=iw:ih/2:0:0

Unfortunately hardware acceleration cannot work together with -filter_complex, apparently. Error in logs:

14:47:50.910 ERR github.com/AlexxIT/go2rtc/internal/mjpeg/init.go:178 > error="exec: exit status 1"

Then:

023-08-12 17:59:49.629580580  14:59:49.629 DBG [exec] run url="exec:ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/test_dual?video&audio -filter_complex [0]crop=iw:ih/2:0:0 -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf \"format=vaapi|nv12,hwupload\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"
2023-08-12 17:59:51.121899571  14:59:51.121 DBG [exec] run launch=2.984648115s
2023-08-12 17:59:51.597230374  Filtergraph 'format=vaapi|nv12,hwupload' was specified through the -vf/-af/-filter option for output stream 0:0, which is fed from a complex filtergraph.
2023-08-12 17:59:51.597239168  -vf/-af/-filter and -filter_complex cannot be used together for the same stream.
2023-08-12 17:59:51.599419782  14:59:51.599 DBG [exec] run url="exec:ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/test_dual?video&audio -filter_complex [0]crop=iw:ih/2:0:0 -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -c:a copy -vf \"format=vaapi|nv12,hwupload\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp {output}"

Camera stream details:

{
  "producers": [
    {
      "type": "RTSP active producer",
      "url": "rtsp://192.168.1.160:554/user=admin_password=admin_channel=0_stream=1&onvif=0.sdp/",
      "remote_addr": "192.168.1.160:554",
      "user_agent": "go2rtc/1.6.2",
      "medias": ["video, recvonly, H265", "audio, recvonly, PCMA/8000"],
      "receivers": [
        "98 H265, bytes=518351, senders=1",
        "8 PCMA/8000, bytes=221120, senders=1"
      ],
      "recv": 753919
    }
  ],
  "consumers": [
    {
      "type": "RTSP passive consumer",
      "url": "rtsp://127.0.0.1:8554/test_dual?video&audio",
      "remote_addr": "127.0.0.1:60892",
      "user_agent": "go2rtc/ffmpeg",
      "medias": ["video, sendonly, ANY", "audio, sendonly, ANY"],
      "senders": [
        "96 H265, bytes=518351, receivers=1",
        "97 PCMA/8000, bytes=221120, receivers=1"
      ],
      "send": 758095
    }
  ]
}

So I wonder if there is some issue in go2rtc.

[felipecrs]

By the way, it would be awesome if go2rtc had support for splitting the stream into two in a more integrated way:

1. Simplified configuration, maybe something like #crop=top and #crop=bottom
2. Somehow allow to capture the two output streams from the same FFMPEG process, to avoid transcoding twice, like this:

ffmpeg -i input -filter_complex "[0]crop=iw:ih/2:0:0[top];[0]crop=iw:ih/2:0:oh[bottom]" -map "[top]" top.mp4 -map "[bottom]" bottom.mp4

But directly mapping each output to different go2rtc streams.

[AlexxIT]

Hard task. Will check someday

[Vistritium]

@felipecrs I'm facing the same issue. Please update if you resolved this issue somehow or had other workaround.

[felipecrs]

I did not resolve neither found any workaround.

[felipecrs]

1. Simplified configuration, maybe something like #crop=top and #crop=bottom

Maybe more like #split=vertically or #split=horizontally.

[felipecrs]

@AlexxIT did you mean that allowing to use #raw=-filter_complex [0]crop=iw:ih/2:0:0 with #hardware is a hard task, or did you mean that capturing the two outputs from a single ffmpeg command is a hard task?

If the first one worked, it would already make my life better.

[AlexxIT]

Both. Split input via hardware. And handle two outputs.

[felipecrs]

Ok, I have some news.

1. #raw=-vf crop=in_w:in_h/2:0:in_h/2 I came up with this alternative command which is not filter_complex
2. #hardware still doesn't work, but now instead of failing the stream, the filter simply doesn't get applied anymore it seems.

I can still use #hardware to do H265 to H264 transcoding and then applying the filter on the transcoded result, to minimize my CPU impact (given that I needed to transcode first).