H264 hw encoding from H265 stream: hw accelerator failed to decode picture

[gvillo]

Hi! first I would like to say thank you for this software, it's neat! I spent hours building a great setup for my surveillance system in my home.

Unfortunately I am blocked now and I was wondering if someone can guide me to figure out a solution for this issue that I am having here, I have a Reolink TrackMix WiFi (yes, I know regarding poor RTSP of that brand), running latest firmware (works ok with tinycam pro/VLC using RTSP), WiFi connection is stable and good.

Go2RTC 1.9.4 running in Frigate in a MiniPC Core i5-1350p 32GB, this is the only camera that I am getting the following error:

1:24:30.336 AM  warn    undefined error=EOF url=ffmpeg:camara.calle_wide#video=h264#hardware#audio=opus#height=720#raw=-b:v 1M -maxrate 2M -bufsize 6M -g 50 caller=github.com/AlexxIT/go2rtc/internal/streams/producer.go:170

1:24:30.326 AM  debug   [rtsp] disconnect stream=camara.calle_wide

1:24:30.326 AM  debug   [rtsp] handle error=EOF

1:24:30.323 AM  debug   [exec] Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0

1:24:30.323 AM  debug   [exec] Impossible to convert between the formats supported by the filter 'Parsed_scale_vaapi_2' and the filter 'auto_scale_0'

1:24:27.379 AM  debug   [exec] run rtsp launch=2.016021351s

1:24:27.287 AM  debug   [exec] [hevc @ 0x58965697c280] Failed to end picture decode issue: 23 (internal decoding error).
[hevc @ 0x58965697c280] hardware accelerator failed to decode picture
Error while decoding stream #0:0: Input/output error

I was looking at the issues and I couldn't find anything like this. I have other reolink cameras (2 more 4K and they are working ok using HW transcoding).

The workaround was removing #hardware, so running CPU works ok for now, I can live with it, but wondering if there is a ffmpeg setting (or might be a bugfix for go2rtc) that mitigates this and be have to use HW transcoding.

My config:

streams:
  camara.calle_wide:     #2160p >> 6mbps >> h265/AAC (16:9) >> 25 fps >> needs MSE instead of RTC
    - ffmpeg:rtsp://SSSSS:XXXXXXXXX$@192.168.##.##/h265Preview_01_main#input=rtsp-reolink-low-latency

  camara.calle_wide_720p: ffmpeg:camara.calle_wide#video=h264#hardware#audio=opus#height=720#raw=-b:v 1M -maxrate 2M -bufsize 6M -g 50
  camara.calle_wide_1080p: ffmpeg:camara.calle_wide#video=h264#hardware#audio=opus#height=1080#raw=-b:v 2M -maxrate 3M -bufsize 6M -g 50

ffmpeg:
  rtsp-reolink-low-latency: -fflags +genpts+discardcorrupt -flags low_delay -timeout
      5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i {input}
  rtsp-reolink: -fflags +genpts+discardcorrupt -timeout 5000000 -user_agent go2rtc/ffmpeg
      -rtsp_flags prefer_tcp -i {input}

It doesn't matter which input I use (with or without ffmpeg, with or without low-latency), it's the same output.

Thanks!!

[emily-pesce]

I am having a similar challenge. Long time since I've wanted to smash my computer into the desk.

Trying to convert reolink h265 stream to h264 (primarily for HKSV). I cannot make it work in hardware.

Config

 streams:
  frontyard:
  - rtsp://user:pass@10.22.22.123:554/Preview_01_main
  frontyard_h264:
  - ffmpeg:frontyard#video=h264/vaapi

Error

[rtsp] error=streams: exec: [hevc @ 0x55eb3e0e3b40] Could not find ref with POC 21 Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0' Error reinitializing filters! Failed to inject frame into filter network: Function not implemented Error while processing the decoded data for stream #0:0 stream=frontyard_h264

Desperate googling led me to try outputting to nv12 in some arg (I am going crosseyed) and I got a similar "failed to decode picture" error.

My hardware is a coffee lake iGPU on Debian bookwork with go2rtc running in a privileged docker (with access to /dev/dri/renderD128)

🆘

[gvillo]

v1.9.2 seems to be the most stable version of go2rtc, besides it doesn't fix this issue, I made a workaround using the camera's substream and I was facing another issue with 1.9.4 with ffmpeg:[...]video=copy with no logs at all (video stops from time to time during the day, camera's substream works ok), v1.9.2 works smoothly without any issues on my side (same cameras, same firmwares, same PoE and WiFi conditions).

Edit: I'll try 1.9.3 and see if I am facing the same issue with the substream #video=copy (or #video=h264#hardware)

Edit 2: 1.9.3 failed in the same way as 1.9.4 after 20 mins, so v1.9.2 ftw!

[AlexxIT]

It's bad idea to use raw ffmpeg params with any transcoding presets. They can be incompatible. You should use exec source and write whole ffmpeg command yourself.

[gvillo]

Cool, thanks for that advice, I know you can append anything and you can break stuff, but I am still getting the same error logs without the raw unfortunately. The video=copy issue that I am mentioning I wasn't appending raw at all.

[AlexxIT]

Well. video=copy is another issue. Maybe related to updates v1.9.3 or v1.9.4.

[emily-pesce]

My computer is unshmashed but hours of my life I will never get back have led to what I think is a solution (to my issue, unclear if it's exactly the same as the OPs).

problem

I could use VAAPI to:

• decode hevc
• encode h264

But if I tried to transcode hevc to h264 all in hardware (e.g., no hwupload or hwdownload) I would get this error: Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0'

If I took the hevc file and (re)encoded it to hevc using VAAPI and then transcode it would work. I am still unclear why it won't work outright, but I did find a setting that seems to solve the issue.

solution

-noautoscale (this is used after the input in the ffmpeg command)

more details

I cranked the loglevel (-v or -loglevel) to 48 and saw this preceding the vague error above.

[graph 0 input from stream 0:0 @ 0x5600bc331080] w:1920 h:1080 pixfmt:vaapi tb:1/90000 fr:49500000/4678801 sar:0/1
[scaler_out_0_0 @ 0x5600bc2c6e00] w:1920 h:1080 flags:'' interl:0
[auto_scale_0 @ 0x5600bc687000] w:iw h:ih flags:'' interl:0
[scaler_out_0_0 @ 0x5600bc2c6e00] auto-inserting filter 'auto_scale_0' between the filter 'Parsed_null_0' and the filter 'scaler_out_0_0'
Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0

This helped: [scaler_out_0_0 @ 0x5600bc2c6e00] auto-inserting filter 'auto_scale_0' between the filter 'Parsed_null_0' and the filter 'scaler_out_0_0'

I still have no idea why ffmpeg felt the need to insert this devil filter (auto_scale_0) but this log led me to the deepest depths of the internet where I found the (undocumented?) rarely mentioned config "-noautoscale".

As soon as I threw that in my ffmpeg command it worked. RELIEF.

examples

In the examples below input.mp4 was a hevc filed with video and audio a la:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf59.27.100
  Duration: 00:00:52.22, start: 0.000000, bitrate: 2300 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main), 1 reference frame (hev1 / 0x31766568), yuv420p(tv, left), 1920x1080 (1920x1088), 2292 kb/s, 10.58 fps, 90k tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, mono, fltp, 15 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]

Does NOT work:

ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i input.mp4 -c:v h264_vaapi output.mp4

Does work:

ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi  -i input.mp4 -c:v h264_vaapi -noautoscale output.mp4

[emily-pesce]

It's bad idea to use raw ffmpeg params with any transcoding presets. They can be incompatible. You should use exec source and write whole ffmpeg command yourself.

Yes, so, what I wrote above was debugging. The fully formed FFmpeg command was indeed added to go2rtc via exec. (This is all configured in Frigate which manages the go2rtc config, but if I configure directly in go2rtc it's the same outcome.)

For those wondering, here's how the exec looks:

streams:
  frontyard:
  - rtsp://USER:PASS@10.11.11.123:554/Preview_01_main
  frontyard_h264:
  - exec:ffmpeg -hide_banner -hwaccel vaapi -hwaccel_output_format vaapi -avoid_negative_ts
    make_zero -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1 -flags
    low_delay -timeout 5000000 -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/frontyard
    -c:v h264_vaapi -vf scale_vaapi=-2:1080 -noautoscale -profile:v high -level:v
    4.1 -b:v 3M -rc_mode QVBR -idr_interval 40 -rtsp_transport tcp -f rtsp {output}

AlexxIT thanks for building go2rtc. It's amazing.

[rpm5099]

@emily-pesce I am very glad that I found this information you posted. I am currently in ffmpeg hell with the Trackmix Wifi - I wish I had never bought these and now I have 2 of them. I am working on Frigate, which I'm guessing you are as well, and I'm going in circles with transcoding HEVC to H264 as well. I have read in a few forums that there is a firmware option for this piece of crap that makes it stream in H264 on the main stream. I am currently in contact with Reolink to see if I can get a copy of it. If I can, I will surely make it accessible to you. Otherwise, I think I've wasted enough time, these cameras are going in the trash and I'm going to get something that streams in H264. I'm NOT dealing with HEVC and all their licensing crap, it's never going to be adopted because of that and a better algorithm will surpass it at some point. It's obnoxious that it ever existed in the first place.

[emily-pesce]

@rpm5099, hi!

I recall the hell all too well. It sucked.

I have had the TrackMix Wifi running for a month (or so) in Frigate with VAAPI-based hardware transcoding from 265 to 264. LMK if I can help you. I use the custom ffmpeg exec in the go2rtc section of the frigate config as per above and then the restream for frigate itself and to Scrypted (so I can pass on to Homekit Secure Video).

I would love if there is a (semi)official h264 firmware for the camera; please do pass along! (If there were native 264 from the camera it would cut down on the electricity I am burning doing the transcoding!)

[rpm5099]

@emily-pesce Thank you very much for replying - still no luck with Reolink. They are trying to warm me up to H265 like I don't know what it is. I've asked for it to be escalated to the appropriate engineer. As far as transcoding, so far I've not even been able to get the arguments to even pass with vaapi or otherwise. I'm going to keep working on it, although I really would prefer not to. I am probably going to take you up on your offer. It's sounds like you've already been down this road.

[rpm5099]

Ok, this actually works. Just a lot more arguments that need to be added:

ffmpeg -v verbose -hide_banner -i rtsp://xxx@10.0.0.32:554/Preview_01_main -map 0 -c:v libx264 -crf 18 -c:a copy -c:s copy h264.mkv

This does not: ffmpeg -v 48 -hide_banner -hwaccel cuvid -c:v hevc_cuvid -hwaccel_output_format cuda -i rtsp://xxx@10.0.0.32:554/Preview_01_main -noautoscale -map 0 -c:v libx264 -crf 18 -c:a copy -c:s copy h264_2.mkv

Result: Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0'

-noautoscale seems to make no difference here

[rpm5099]

Update from Reolink - they replied and told me that they "pushed" the firmware to the one camera that I had given them the information for. I just wrote back explaining that I did not want them to do that, I have other cameras and I must have a copy of the firmware binary that I can install myself.

I found this page which I hadn't seen before. Currently it does not have the h264 version, but if I get a copy of it I will send it to the project owner. I'll post a google drive link here.

v3.0.0.3748_2408281837 - h265 only, latest version from reolink v3.0.0.3748_2408132492 - h264 only, waiting for a copy

I would also note that the smoothness, clarity and color of the H265 did look considerably better than the H264. H264 appears slightly choppy and the colors washed out, but it smells like maybe that's because considerable more effort was put into optimizing the quality of H265 for the release version of the firmware. It actually does not look good at all, I can see why they tried to talk me out of that firmware. I am still going to pursue the transcoding option if I can do it in a way that preserves the original quality of the H265 stream. But these will be my last Reolink cameras.

ffprobe output
Input #0, rtsp, from 'rtsp://xxx@10.0.0.32:554/h265Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
  Duration: N/A, start: 0.000250, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 3840x2160, 20 fps, 40 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

[rpm5099]

@emily-pesce Here you go Reolink TrackMix Wifi Firmware v3.0.0.3748_2408132492 - H.264 main stream

You'll probably be disappointed with the quality, but give it a shot.

[emily-pesce]

Thank you! Do you know if it's possible to flash back to current after using this h264 version (in case it stinks)?

[rpm5099]

@emily-pesce Yes, you can restore the HEVC by installing the latest firmware for the camera.

[rpm5099]

I'll summarize for anyone else working on HEVC -> H264 transcoding.

While I'm sure that there must be some settings compromise that would provide decent quality H264 from HEVC without being enormous, I have not been able to figure out how or if it's possible to do that. All of the attempts I made at converting HEVC from the Reolink Trackmix Wifi either look terrible or increase the size by orders of magnitude. I was able to find working settings using both cuda/gpu and libx264/cpu:

-crf seems to be allowed here but makes no difference. -b:v:0 indicates lossless - it makes the video's look perfect but data size is huge. Without it the files are tiny and look terrible. I could not find any other way to tweak the settings. ffmpeg -v 48 -threads 8 -hwaccel cuda -rtsp_flags prefer_tcp -i rtsp://XXX@10.0.0.32:554/Preview_01_main -map 0 -c:v h264_nvenc -b:v 0 --c:a copy output.mkv

This uses a massive amount of CPU for a single 4k stream, I do not think it is a practical approach if you dont have serious hardware or have more than a few cameras. -crf does appear to provide a good set of options - 0 == lossless/huge, 51 == terrible quality/tiny files. ffmpeg -v 48 -threads 8 -hwaccel cuda -rtsp_flags prefer_tcp -i rtsp://XXX@10.0.0.32:554/Preview_01_main -map 0 -c:v libx264 -crf 18 -c:a copy hevc4.mkv

I did not attempt to transcode HEVC to AV1, but in retrospect I think that makes more sense than doing the work to convert to H264 and losing the benefits of superior compression.

I can't speak for other cameras but for the Reolink Trackmix (Reolink did provide me with firmware that forces the main stream to be H264) the quality is so bad that I decided to go back to HEVC even if that means that the video will only be viewable in Chrome. I think that this may be due to the fact that Reolink did not put any emphasis on tweaking/perfecting the H264 stream because it was never intended to use that encoding. Hopefully in the future cameras will have the option to stream in AV1 so that we can get the benefits of the latest video codecs without all the limitations imposed by HEVC's absurd licensing process.

[emily-pesce]

I went back to the hevc firmware too. there is some sort of brightness bug in the h264 firmware. :(

What is your issue with transcoding? File size? And quality?

First, to do all the transcoding in HW you have to write into and the write out of the same hardware space. E.g., for me that means: on the input side (before -i): -hwaccel vaapi -hwaccel_output_format vaapi on the output side (after -i): -c:v h264_vaapi -noautoscale

You may need the hwaccel_output_format argument. Also, is h264nvenc the pair to cuda (-hwaccel cuda)? I just don't know. If these aren't set right it'll copy out of and then back into memory during transcode which slows everthing down.

On quality, I've found this works well enough!

-b:v 2000k -bufsize 3000k -maxrate 2000k -rc_mode QVBR -bf 0 -g 80 -r 20

this sets bitrate to 2000k, a (probably too big) buffer of 3000k and a max bitrate of 2000k - so basically a 2000k target. I think -rc_mode is a VAAPI specific thing I found, but this is the setting that says optimize for a VBR that is high quality. This also sets b-frames to 0 (none), a 20 fps and, (I think) an iframe every 4 seconds (80 frames).

[rpm5099]

I have no issues with encoding itself, but weighing the benefit of compatibility with firefox against the large increase in size and computational cost it does not appear to be worth it. The reason I was pursuing encoding was actually that the RTSP feed for HEVC from the Trackmix stutters quite a bit, I find it painful to watch things like a person walking and I was told ffmpeg transcoding should smooth that out. I also see the stuttering when viewing it live via VLC but of course in the Reolink app it looks fine. I'm starting to think this is a flaw in the Trackmix firmware. I can't say if the H264 has the same problem, the image was too poor to even tell. I'm pretty disappointed overall with the Trackmix, if I could do it over again I would not have bought them.

BTW, if adding -hwaccel_output_format cuda with -hwaccel cuda gets the auto scale filter error, and adding -noautoscale makes no difference. It requires some other filter arguments that I can't get working.

[emily-pesce]

BTW, if adding -hwaccel_output_format cuda with -hwaccel cuda gets the auto scale filter error, and adding -noautoscale makes no difference. It requires some other filter arguments that I can't get working.

🤬

I got you. it's frustrating. I have the same stubborn streak: "going to make this work!!!"

Sometimes we just need to find a new product. ;)

Good luck!