QSV intel hardware acceleration - video decoding

[Arol450]

Environment: ZM: V1.36.34 (master) with modifications to source code FFMPEG: N-109920-gac6eec1fc2 (GIT) with --enable-libvpl compiled Kernel: 6.2.0-1-drm-tip-git-g798a0dbb51e8 Graphics card: Intel ARC A770

I am trying to get zoneminder to use Intel's QSV hardware accelertated decoding to work. While VAAPI seems to work (very not efficient) QSV is just not working.

I pulled the git master and modified zm_ffmpeg_camera.cpp as follows: case AV_HWDEVICE_TYPE_CUDA: return AV_PIX_FMT_CUDA; +case AV_HWDEVICE_TYPE_QSV: +return AV_PIX_FMT_QSV;

ifdef AV_HWDEVICE_TYPE_MMAL

And I changed DecoderHWAccelName to qsv.

ffmpeg -hwaccels:

Hardware acceleration methods: vdpau vaapi qsv drm opencl vulkan

I confirmed ffmpeg to work with qsv HW acceleration: ffmpeg -hwaccel qsv -c:v hevc_qsv -i someHEVCinput.mp4 -c:v av1_qsv -profile:v main -preset veryslow -qscale 75 output.mp4

Can somebody please point me in the right direction? Thank you!

[Arol450]

Here is some debug output - sorry for the delay, I was fighting deprecated functions in php.

zmc_m11[36996].DB3-zm_ffmpeg_camera.cpp/388 [Found video stream at index 0, audio stream at index -1] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/394 [Failed to find decoder (h264_mmal)] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/400 [Failed to find decoder (h264_nvmpi)] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/461 [vdpau] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/461 [vaapi] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/461 [qsv] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/461 [drm] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/461 [opencl] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/461 [vulkan] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/468 [Found hwdevice qsv] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/488 [Decoder h264 hwConfig doesn't match our type: qsv != vaapi, pix_fmt vaapi.] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/488 [Decoder h264 hwConfig doesn't match our type: qsv != vdpau, pix_fmt vdpau.] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/476 [Decoder h264 does not support config 2.] zmc_m11[36996].DB1-zm_ffmpeg_camera.cpp/521 [Failed to find suitable hw_pix_fmt.]

[Arol450]

I did a fresh git clone of the master. This time no changes of my own. Interested how your commits https://github.com/ZoneMinder/zoneminder/commit/6505bba826fc0c293cd348cb60cc2b923b02a750 and https://github.com/ZoneMinder/zoneminder/commit/c1c25aeb761a933ee3b368e37b5c24c8653f2e72 behaved with the hardware accelerated decoding.

I get pretty much the same debug output once I start zoneminder: zmc_m11[2734].DB3-zm_ffmpeg_camera.cpp/388 [Found video stream at index 0, audio stream at index -1] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/394 [Failed to find decoder (h264_mmal)] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/400 [Failed to find decoder (h264_nvmpi)] zmc_m11[2734].DB1-zm_ffmpeg.cpp/246 [Dumping stream index i(0) index(0)] zmc_m11[2734].DB1-zm_ffmpeg.cpp/252 [ Stream #0:0] zmc_m11[2734].DB1-zm_ffmpeg.cpp/260 [frame_size:0 stream timebase: 1/90000] zmc_m11[2734].DB1-zm_ffmpeg.cpp/265 [codec: h264 video] zmc_m11[2734].DB1-zm_ffmpeg.cpp/283 [, SAR 0:1 ] zmc_m11[2734].DB1-zm_ffmpeg.cpp/175 [12 fps] zmc_m11[2734].DB1-zm_ffmpeg.cpp/177 [90k stream tb numerator] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/461 [vdpau] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/461 [vaapi] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/461 [qsv] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/461 [drm] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/461 [opencl] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/461 [vulkan] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/468 [Found hwdevice qsv] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/488 [Decoder h264 hwConfig doesn't match our type: qsv != vaapi, pix_fmt vaapi.] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/488 [Decoder h264 hwConfig doesn't match our type: qsv != vdpau, pix_fmt vdpau.] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/476 [Decoder h264 does not support config 2.] zmc_m11[2734].DB1-zm_ffmpeg_camera.cpp/521 [Failed to find suitable hw_pix_fmt.] zmc_m11[2734].DB3-zm_ffmpeg.cpp/68 [nal_unit_type: 7(SPS), nal_ref_idc: 3] zmc_m11[2734].DB3-zm_ffmpeg.cpp/68 [nal_unit_type: 8(PPS), nal_ref_idc: 3] zmc_m11[2734].DB3-zm_ffmpeg.cpp/68 [Decoding VUI] mc_m11[2734].DB1-zm_ffmpeg_camera.cpp/555 [Thread count? 1] zmc_m11[2734].DB1-zm_ffmpeg.cpp/215 [Dumping codec_context codec_type 0 video codec_id 27 h264 width 1920 height 1080 timebase 0/1 format yuvj420p profile 77 level 40 gop_size 12 has_b_frames 0 max_b_frames 0 me_cmp 0 me_range 0 qmin 2 qmax 31 bit_rate 0 extradata:60:xxx]

I look at the code before line 488: if ((config->methods & AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX) && (config->device_type == type) that leads to this error message. Where is the list/define so that I could make that match?

[Arol450]

If I use vaapi as accelerator - which is really not very efficient, I get the following: zmc_m11[3422].DB3-zm_ffmpeg_camera.cpp/388 [Found video stream at index 0, audio stream at index -1] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/394 [Failed to find decoder (h264_mmal)] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/400 [Failed to find decoder (h264_nvmpi)] zmc_m11[3422].DB1-zm_ffmpeg.cpp/246 [Dumping stream index i(0) index(0)] zmc_m11[3422].DB1-zm_ffmpeg.cpp/252 [ Stream #0:0] zmc_m11[3422].DB1-zm_ffmpeg.cpp/260 [frame_size:0 stream timebase: 1/90000] zmc_m11[3422].DB1-zm_ffmpeg.cpp/265 [codec: h264 video] zmc_m11[3422].DB1-zm_ffmpeg.cpp/283 [, SAR 0:1 ] zmc_m11[3422].DB1-zm_ffmpeg.cpp/175 [12 fps] zmc_m11[3422].DB1-zm_ffmpeg.cpp/177 [90k stream tb numerator] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/461 [vdpau] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/461 [vaapi] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/461 [qsv] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/461 [drm] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/461 [opencl] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/461 [vulkan] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/468 [Found hwdevice vaapi] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/484 [Decoder h264 does support our type vaapi.] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/488 [Decoder h264 hwConfig doesn't match our type: vaapi != vdpau, pix_fmt vdpau.] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/476 [Decoder h264 does not support config 2.] zmc_m11[3422].DB1-zm_ffmpeg_camera.cpp/500 [Selected hw_pix_fmt 44 vaapi] zmc_m11[3422].DB2-zm_ffmpeg.cpp/68 [Trying to use DRM render node for device 0.] zmc_m11[3422].DB2-zm_ffmpeg.cpp/68 [libva: VA-API version 1.17.0] zmc_m11[3422].DB2-zm_ffmpeg.cpp/68 [libva: Trying to open /usr/lib/dri/iHD_drv_video.so] zmc_m11[3422].DB2-zm_ffmpeg.cpp/68 [libva: Found init function __vaDriverInit_1_17] zmc_m11[3422].DB2-zm_ffmpeg.cpp/68 [libva: va_openDriver() returns 0] zmc_m11[3422].DB2-zm_ffmpeg.cpp/68 [Initialised VAAPI connection: version 1.17]

[Arol450]

I did one more thing: I added the following to zm_ffmpeg_camera.cpp:

if ((mVideoCodec = avcodec_find_decoder_by_name("h264_qsv")) == nullptr) { Debug(1, "Failed to find decoder (h264_qsv)"); } else { Debug(1, "Success finding decoder (h264_qsv)"); }

And sure enough I get a the following in the debug logs: zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/394 [Failed to find decoder (h264_mmal)] zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/402 [Success finding decoder (h264_qsv)] zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/406 [Failed to find decoder (h264_nvmpi)]

So naively I was thinking everything should be okay now, as mVideoCodec should be "filled" with the right info.

Further down in the debug log I get: zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/474 [Found hwdevice qsv] zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/494 [Decoder h264 hwConfig doesn't match our type: qsv != vaapi, pix_fmt vaapi.] zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/494 [Decoder h264 hwConfig doesn't match our type: qsv != vdpau, pix_fmt vdpau.] zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/482 [Decoder h264 does not support config 2, type: qsv.] zmc_m11[189830].DB1-zm_ffmpeg_camera.cpp/527 [Failed to find suitable hw_pix_fmt.]

Still no cigar. Any ideas? Any help would be appreciated.

[connortechnology]

It looks like the qsv hwaccel wants either vaapi or vdpau, not qsv format.

[connortechnology]

Also, the codec's name is h264... but should be h264_qsv. We are looking up the codec by codec_id... but maybe we should be looking it up by name specifically, like in videostore.

[connortechnology]

Ugh.. all that code is bad...I knew it when I merged it.

[connortechnology]

02/26/23 12:37:18.144132 zmc_m2[101272].ERR-zm_ffmpeg_camera.cpp/525 [Failed to create hwaccel device. Generic error in an external library]

02/26/23 12:37:18.145070 zmc_m2[101272].WAR-zm_ffmpeg_camera.cpp/604 [Monitor dimensions are 640x480 but camera is sending 1920x1080]

02/26/23 12:37:19.312112 zmc_m2[101274].WAR-zm_ffmpeg.cpp/68 [Error initializing an internal MFX session: unsupported (-3)]

02/26/23 12:37:19.312248 zmc_m2[101274].WAR-zm_ffmpeg.cpp/68 [Error initializing an MFX session]

[connortechnology]

I think this is because my hardware doesn't support h265. So.. maybe by tonight I'll polish up this code and push it.

[Arol450]

My assessment as well. First the codec is called h264_qsv, later h264. First: [Success finding decoder (h264_qsv)] Later: [Decoder h264 does not support config 2, type: qsv.] This is all very confusing to me.

[connortechnology]

If setting up the qsv codec fails, it will fall back to whatever find_decoder(codec_id) returns. I have updated the code to require specifying the codec for decoder. So no go into the monitor and specify h264_qsv.

[Arol450]

I will have a look at the new code and give it a try.

[Arol450]

Wow, I am impressed. I am not too sure what to of the debug log atm, but I am very hopeful. A couple of things: 1) hwaccel_device.c_str() seems to be returning nothing. 2) What does "Decoder h264_qsv does support our type qsv" mean? 3) Use Intel(R) oneVPL to create MFX session: Isn't MFX deprecated? 4) But then it seems MFX hands over to oneVPL: Use Intel(R) oneVPL to create MFX session. What do you think? I will test more.

zmc_m11[45338].DB3-zm_ffmpeg_camera.cpp/388 [Found video stream at index 0, audio stream at index -1] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/396 [Success finding decoder h264_qsv] zmc_m11[45338].DB1-zm_ffmpeg.cpp/246 [Dumping stream index i(0) index(0)] zmc_m11[45338].DB1-zm_ffmpeg.cpp/252 [ Stream #0:0] zmc_m11[45338].DB1-zm_ffmpeg.cpp/260 [frame_size:0 stream timebase: 1/90000] zmc_m11[45338].DB1-zm_ffmpeg.cpp/265 [codec: h264 video] zmc_m11[45338].DB1-zm_ffmpeg.cpp/283 [, SAR 0:1 ] zmc_m11[45338].DB1-zm_ffmpeg.cpp/175 [12 fps] zmc_m11[45338].DB1-zm_ffmpeg.cpp/177 [90k stream tb numerator] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/425 [vdpau] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/425 [vaapi] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/425 [qsv] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/425 [drm] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/425 [opencl] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/425 [vulkan] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/432 [Found hwdevice qsv] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/448 [Decoder h264_qsv does support our type qsv.] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/440 [Decoder h264_qsv does not support config 1.] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/464 [Selected hw_pix_fmt 114 qsv] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [Trying to use DRM render node for device 0, with matching kernel driver (i915).] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [libva: VA-API version 1.17.0] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [libva: User requested driver 'iHD'] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [libva: Trying to open /usr/lib/dri/iHD_drv_video.so] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [libva: Found init function __vaDriverInit_1_17] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [libva: va_openDriver() returns 0] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [Initialised VAAPI connection: version 1.17] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x41524742 -> bgra.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x42475241 -> argb.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x41424752 -> rgba.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x52474241 -> abgr.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x58524742 -> bgr0.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x42475258 -> 0rgb.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x58424752 -> rgb0.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x52474258 -> 0bgr.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30335241 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30334241 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30335258 -> x2rgb10le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30334258 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x36314752 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x56555941 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x56555958 -> vuyx.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30303859 -> gray.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x3231564e -> nv12.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x3132564e -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x32595559 -> yuyv422.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x59565955 -> uyvy422.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x32315659 -> yuv420p.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30323449 -> yuv420p.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x50313134 -> yuv411p.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x48323234 -> yuv422p.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x56323234 -> yuv440p.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x50343434 -> yuv444p.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x33434d49 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30313050 -> p010le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x32313050 -> p012le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x36313050 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30313259 -> y210le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x32313259 -> y212le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x36313259 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x30313459 -> xv30le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x32313459 -> xv36le.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x36313459 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x50424752 -> unknown.] zmc_m11[45338].DB3-zm_ffmpeg.cpp/68 [Format 0x50524742 -> unknown.] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.1.0 ().] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [Driver not found in known nonstandard list, using standard behaviour.] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [Use Intel(R) oneVPL to create MFX session, API version is 2.8, the required implementation version is 1.3] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [Initialize MFX session: implementation version is 2.8] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/480 [Created hwdevice for ] zmc_m11[45338].DB2-zm_ffmpeg.cpp/68 [The input looks like it is Annex B already] zmc_m11[45338].INF-zm_ffmpeg.cpp/68 [Invalid pkt_timebase, passing timestamps as-is.] zmc_m11[45338].DB1-zm_ffmpeg_camera.cpp/519 [Thread count? 1] zmc_m11[45338].DB1-zm_ffmpeg.cpp/215 [Dumping codec_context codec_type 0 video codec_id 27 h264 width 1920 height 1080 timebase 0/1 format yuvj420p profile 77 level 40 gop_size 12 has_b_frames 0 max_b_frames 0 me_cmp 0 me_range 0 qmin 2 qmax 31 bit_rate 0 extradata:60:00000001674d40288d8d703c0113f2e02dc040405000003e800005dc0e860016f700005f5e0bbcb8d0c002dee0000bebc177970a0000000168ee3880]

[Arol450]

So far I like what I see. Very much. I have 10 cams up and capturing and analyzing. No storage so far. The ARK graphics card is bored at around 15%. CPU load is way way way down now. My super old Dahua cameras I have turned off for now. There is something weird going on - my computer will freeze when I turn them on. I will investigate. I will keep testing this over night and see what happens. Will report back.

There are a couple of changes coming up. An official kernel 6.2 will be released any day now. I will try and replace my drm-tip-git experimental kernel for that. I also managed to get the old Nvidia card back into the computer, no luck with compiling the driver yet though (custom kernel). I will test all the other decoder options I have hardware for later.

Thank you for your help!

[Arol450]

Fun (not) thing: I can run exactly 10 cameras (streams). And all zmc processes are at around 30-60% CPU utilisation. My system is around 50-60% idle at that time. As soon as I add the 11th stream - no matter from what source - ALL processes nearly double in utilisation and my system becomes sloooow. Then - when I disable one stream - any stream, to bring it down to 10 again, the utilisation of all zmc processes goes back to the initial value. All streams are handled with h264_qsv at the moment.

[Arol450]

On the excess CPU usage problem: 1) Still happens with the official 2.6 kernel and recompiled ffmpeg 2) I cannot find anything in the logs that would account for the jump in utilisation once an 11th camera gets enabled. 3) tmpfs/shm usage is 827M (6%). Any ideas where else I could look for an explanation?

[connortechnology]

I have seen similar without hwaccel. Seems like there is a tipping point where once there isn't enough cpu to go around, everything starts working harder. Is it too much context switching? Maybe. Is it just timing around a lock? Maybe. I don't know. Need to do some profiling.

[Arol450]

I did some more testing. I am on an official 6.2 kernel now from Arch. Installed a second graphics card, NVIDA 1060. Recompiled ffmpeg-full-git. Verified functionality of qsv and cuda hwaccels on the command line. Nvidia and Intel kernel modules are all 'happy'.

1) I am happy to report that qsv and cuda HW decoders both work, even in a mixed environment. 2) Switching to all cuda backed decoding I have the same symptoms with everything sloooowing down to barely usable at around 10 streams. Meaning CPU load nearly doubles per zmc process once I add the 10th stream. 3) Same slowing down symptomatic with all qsv. 4) Same symptomatic with a mixed environment, although I have not and will not try all permutations. 5) I removed the Intel card, popped in the old HDD and went back to the official release of zoneminder and I can run 15 streams cuda backed without the slowing down symptoms.

[Arol450]

I also ran 12 x ffmpeg -hwaccel qsv -c:v h264_qsv -rtsp_transport tcp -i rtsp://xx:xx@ip/pathx -c:v av1_qsv -profile:v main -preset veryslow -f null - with different IP addresses every time to most of my cameras. Each ffmpeg process consumes about 5-10% of CPU. And the grpahics card doing both decoding encoding is at 30% total.

[Arol450]

I looked at the context switches, for 9, 11, 12 cameras up and active. Seems the context switches go down, while the nonvoluntary ones go way up. I have 16 cores, this should not be a fight about CPU cores....

pidstat -w 5

------- UID PID cswch/s nvcswch/s Command

13:57:24 33 3476 10.00 0.00 nph-zms 13:57:24 33 3505 10.00 0.00 nph-zms 13:57:24 33 3556 9.80 0.00 nph-zms 13:57:24 33 3668 1.00 0.00 zmdc.pl 13:57:24 33 3697 538.40 2.40 zmc 13:57:24 33 3701 206.40 1.20 zmc 13:57:24 33 3704 511.60 1.80 zmc 13:57:24 33 3708 514.00 2.60 zmc 13:57:24 33 3712 471.60 1.40 zmc 13:57:24 33 3716 498.60 2.60 zmc 13:57:24 33 3720 528.80 5.40 zmc 13:57:24 33 3724 487.80 3.00 zmc 13:57:24 33 3728 503.80 2.20 zmc 13:57:24 33 3758 0.60 0.00 zmwatch.pl 13:57:24 33 3765 3.00 0.00 zmstats.pl 13:57:24 33 3847 10.00 0.20 nph-zms 13:57:24 33 3883 10.00 0.00 nph-zms 13:57:24 33 3922 10.00 0.00 nph-zms

13:58:49 33 3476 10.00 0.00 nph-zms 13:58:49 33 3505 10.00 0.00 nph-zms 13:58:49 33 3556 10.00 0.00 nph-zms 13:58:49 33 3668 1.00 0.00 zmdc.pl 13:58:49 33 3697 527.40 3.00 zmc 13:58:49 33 3701 193.00 1.00 zmc 13:58:49 33 3704 521.00 1.40 zmc 13:58:49 33 3708 544.20 1.40 zmc 13:58:49 33 3712 493.00 2.00 zmc 13:58:49 33 3716 497.40 2.20 zmc 13:58:49 33 3720 462.20 1.00 zmc 13:58:49 33 3724 525.60 1.40 zmc 13:58:49 33 3728 520.80 2.60 zmc 13:58:49 33 4085 499.20 1.40 zmc 13:58:49 33 4130 6.60 0.00 zmc 13:58:49 33 3847 10.00 0.20 nph-zms 13:58:49 33 3883 10.00 0.00 nph-zms 13:58:49 33 3922 10.00 0.00 nph-zms 13:58:49 33 4156 10.00 0.00 nph-zms

14:00:34 33 3476 10.00 0.00 nph-zms 14:00:34 33 3505 10.00 0.00 nph-zms 14:00:34 33 3556 10.00 0.00 nph-zms 14:00:34 33 3668 1.00 0.00 zmdc.pl 14:00:34 33 3697 97.80 21.00 zmc 14:00:34 33 3701 26.00 0.80 zmc 14:00:34 33 3704 130.40 5.80 zmc 14:00:34 33 3708 62.20 18.60 zmc 14:00:34 33 3712 81.00 14.80 zmc 14:00:34 33 3716 153.60 3.00 zmc 14:00:34 33 3720 107.40 10.00 zmc 14:00:34 33 3724 101.60 5.80 zmc 14:00:34 33 3728 103.80 5.80 zmc 14:00:34 33 4085 113.60 5.60 zmc 14:00:34 33 4130 83.20 4.20 zmc 14:00:34 33 4187 82.40 5.80 zmc 14:00:34 33 3758 0.60 0.20 zmwatch.pl 14:00:34 33 3847 10.00 0.00 nph-zms 14:00:34 33 3883 10.00 0.00 nph-zms 14:00:34 33 3922 10.00 0.00 nph-zms

[Arol450]

I did a rollback to 1.36.33 on the same computer, everything else the same. Using cuda as hwaccel, 12 zmc processes behave very very nicely.

pidstat -w 5

15:40:32 33 4771 526.20 1.60 zmc 15:40:32 33 4775 140.80 1.20 zmc 15:40:32 33 4779 536.20 2.60 zmc 15:40:32 33 4783 533.60 2.60 zmc 15:40:32 33 4787 309.60 1.20 zmc 15:40:32 33 4791 503.40 2.20 zmc 15:40:32 33 4794 584.60 1.20 zmc 15:40:32 33 4799 522.40 0.00 zmc 15:40:32 33 4803 540.20 0.60 zmc 15:40:32 33 4810 429.60 0.80 zmc 15:40:32 33 5126 507.80 0.80 zmc 15:40:32 33 5232 521.00 0.60 zmc

So it must be something that happened between then and when the new zm_ffmpeg_camera.cpp code came into place.

[Arol450]

I figured it out. Most likely it is the multi-threading in master. I reverted back to 1.36.33 and did the following (basically back-ported your recent changes and hard-coded the decoder name):

case AV_HWDEVICE_TYPE_QSV: return AV_PIX_FMT_QSV;

// Test const AVCodec *mVideoCodec = nullptr; if ((mVideoCodec = avcodec_find_decoder_by_name("h264_qsv")) == nullptr) { Debug(1, "Test: The QSV decoder is not present in libavcodec."); } else { Debug(1, "Test: Success finding the QSV decoder."); }

// Test if (!mVideoCodec) { mVideoCodec = avcodec_find_decoder(mVideoStream->codecpar->codec_id); if (!mVideoCodec) { // Try and get the codec from the codec context Error("Test: Can't find codec for video stream from %s", mMaskedPath.c_str()); return -1; } }

15 h264 streams get decoded with no weirdness in context switching. Graphics card is pretty bored, CPU utilisation is minimal. I recommend turning off multi-threading if HW acceleration is being used. Or at least CUDA and/or QSV, can't speak for the other HW accelerator schemes.

Thanks for your help. Closing.

[connortechnology]

Do you mean the multi-threading in avcodec? Well that should be easy to toggle on/off to find out.

[Arol450]

I agree. That is another if then else

[connortechnology]

Um... the multi threading is commented out. There really isn't much difference in the decoding code between master and release-1.36. I don't suppose you could bisect it?