Output Stalls / Hangs (never completes / finishes)

bsikar commented 3 months ago

Whenever I run this within tdarr or by itself the output never seems to finish:

I see this, but it never exits/returns.

The process still is attached to the gpu once finished and the pid doesnt stop.

This is the command that I tried to run:

/usr/bin/qsvencc --device auto --codec av1 --avhw --icq 25 --output-depth 10 --fixed-func --va --quality best --la-depth 80 --la-window-size 0 --la-quality auto --tune perceptual --scenario-info archive --extbrc --i-adapt --b-adapt --bframes 10 --b-pyramid --adapt-ltr --adapt-cqm --ref 8 --gop-len 350 --gop-pred-dist 16 --trellis auto --level auto --profile auto --colorrange auto --colormatrix auto --colorprim auto --transfer auto --chromaloc auto --max-cll copy --master-display copy --video-metadata copy --audio-copy --chapter-copy --sub-copy --metadata copy --gpu-copy --output-thread -1 -i /mnt/media-and-fetching/media/test/The Boys (2019) - S01E02 - Cherry [Bluray-1080p Remux][DTS-HD MA 5.1][AVC].mkv -o /mnt/media-and-fetching/tdarr-cache/tdarr-workDir-node-UCP3-Le94-worker-eager-esok-ts-1722286951393/1722286952487/The Boys (2019) - S01E02 - Cherry [Bluray-1080p Remux][DTS-HD MA 5.1][AVC].mkv

The set up of this is proxmox -> lxc unprivileged container -> gpu passthrough (a310-eco)

Trying display: wayland
error: XDG_RUNTIME_DIR not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.21.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 24.2.4 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP

root@tdarr:~# qsvencc --check-hw
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
Success: QuickSyncVideo (hw encoding) available
Supported Encode Codecs for device auto:
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
libva info: va_openDriver() returns 0
H.264/AVC FF
H.265/HEVC FF
VP9 FF
AV1 FF

bsikar commented 3 months ago

i updated versions and have the same issue:

root@tdarr:~# qsvencc --version
QSVEncC (x64) 7.68 (r3389) by rigaya, Jul  6 2024 09:12:14 (gcc 13.2.0/Linux)
 Intel Media SDK API v2.12
 reader: raw, avs, vpy, avsw, avhw
 vpp:    resize, deinterlace, denoise, detail-enhance, image-stab, delego, sub
root@tdarr:~# qsvencc --check-avversion
ffmpeg     version: 6.1.1-3ubuntu5
avutil     version:  58. 29. 100
avcodec    version:  60. 31. 102
avformat   version:  60. 16. 100
avfilter   version:   9. 12. 100
swresample version:   4. 12. 100

root@tdarr:~# vainfo 
Trying display: wayland
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.21.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 24.2.4 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP
root@tdarr:~#

after 5 minute of running:

bsikar commented 3 months ago

i downgraded and also had the same issue:

root@tdarr:/mnt/media-and-fetching/media/isolated-test# qsvencc --check-avversion
ffmpeg     version: 4.4.2-0ubuntu0.22.04.1
avutil     version:  56. 70. 100
avcodec    version:  58.134. 100
avformat   version:  58. 76. 100
avfilter   version:   7.110. 100
swresample version:   3.  9. 100

root@tdarr:/mnt/media-and-fetching/media/isolated-test# qsvencc --version
QSVEncC (x64) 7.67 (r3371) by rigaya, Jun 29 2024 04:41:04 (gcc 11.4.0/Linux)
 Intel Media SDK API v2.09
 reader: raw, avs, vpy, avsw, avhw
 vpp:    resize, deinterlace, denoise, detail-enhance, image-stab, delego, sub
root@tdarr:/mnt/media-and-fetching/media/isolated-test# vainfo
Trying display: wayland
error: XDG_RUNTIME_DIR not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.22.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_22
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.21.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 24.2.4 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSliceLP
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointEncSliceLP
      VAProfileVP9Profile1            : VAEntrypointVLD
      VAProfileVP9Profile1            : VAEntrypointEncSliceLP
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointEncSliceLP
      VAProfileVP9Profile3            : VAEntrypointVLD
      VAProfileVP9Profile3            : VAEntrypointEncSliceLP
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointVLD
      VAProfileHEVCMain422_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain422_12         : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointEncSliceLP
      VAProfileHEVCMain444_12         : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointVLD
      VAProfileHEVCSccMain            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain10          : VAEntrypointVLD
      VAProfileHEVCSccMain10          : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444         : VAEntrypointVLD
      VAProfileHEVCSccMain444         : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointEncSliceLP
      VAProfileHEVCSccMain444_10      : VAEntrypointVLD
      VAProfileHEVCSccMain444_10      : VAEntrypointEncSliceLP
root@tdarr:/mnt/media-and-fetching/media/isolated-test#

bsikar commented 3 months ago

it works perfectly when ran in a windows10 (win10-tiny) on proxmox as a VM. I might try this again with a linux VM. This might be an issue with how LXC (unpriviliged) works on proxmox. I will test it as follows:

Currently Tested	Needs to be Tested
Ubuntu 24.04 (LXC Unprivileged) @ v7.68 ❌	Ubuntu 24.04 (LXC Privileged) @ v7.68 ❌
Ubuntu 22.04 (LXC Unprivileged) @ v7.67 ❌
Ubuntu 22.04 (LXC Unprivileged) @ v7.68 ❌	Ubuntu 22.04 (LXC Privileged) @ v7.68 ❌
	Ubuntu 24.04 (VM) @ v7.68 🟡
	Ubuntu 22.04 (VM) @ v7.68 🟡
Windows 10 (VM) @ v7.68 ✅

rigaya commented 3 months ago

Unfortunately, I don't have problem like that in native Ubuntu 22.04 environment.

qsvencc_ubuntu2204_system

qsvencc_ubuntu2204_av1_icq

Adding --log-level app=debug will show more messages, which might enable us to narrow down the cause. In normal situation, it should go like below after showing encoding status.

encoded 62436 frames, 267.60 fps, 10063.97 kbps, 2499.35 MB
encode time 0:03:53, CPULoad: 0.0%
frame type IDR   179
frame type I     179,  total size    41.19 MB
frame type P   62257,  total size  2458.16 MB
RunEncode2: finished.
Clear vpp filters...
Closing m_pmfxDEC/ENC/VPP...
Clear pipeline tasks and allocated frames...
clReleaseProgram...
clReleaseProgram: fin.
Closing enc status...
Closing device...
Close device 2...
Closing session...
Closing device...
Closing allocator...
allocVA: QSVAllocator: Closed.
Device 2 closed.
Closing CL Context...
Closed CL m_copy program.
Closed CL Queue.
Closed CL Context.
Closed CL Platform.
Closing audio readers (if used)...
Closing writer...
avout: Closing...
avout: closed output thread...
avout: closed queues...
[mp4 @ 0x593a44464f00] Starting second pass: moving the moov atom to the beginning of the file
avout: Closed avformat context.
avout: Closed File Pointer.
avout: Closed format.
avout: Closed video context.
avout: Closed video.
avout: Closed.
avout: Closing...
avout: closed output thread...
avout: closed queues...
avout: Closed format.
avout: Closed video.
avout: Closed.
Closing reader...
avqsv: Closing...
Closing Input thread.
Closed Input thread.
avqsv: Closed Stream Packet Buffer.
Closing avformat context...
Closed avformat context.
Free formatOptions...
Freed formatOptions.
avqsv: Closed format.
Close parser...
Closed parser.
Close codecCtx for parser...
Closed codecCtx for parser.
Free first video packet...
Freed first video packet.
Free extra data...
Freed extra data.
avqsv: Closed video.
Closing EncodeStatus...
avqsv: Cleared frame pos list.
avqsv: Closed.
avqsv: Closing...
avqsv: Closed Stream Packet Buffer.
avqsv: Closed format.
avqsv: Closed video.
avqsv: Cleared frame pos list.
avqsv: Closed.
avqsv: Closing...
avqsv: Close...
Closing perf monitor...
Closed pipeline.

bsikar commented 3 months ago

I see, okay thank you so much. I will do more local testing with that flag and update in 1 large post any findings I have later. I will need a couple of days to do this since there is an issue with windows and the arc cards on proxmox (tldr, if i shutdown the windows vm it stops my pve host and I need someone to manually visit my server to reboot it).

Also, thank you for this software, it is really amazing and works super well :) I was doing some testing on the windows vm and its awesome.

bsikar commented 3 months ago

https://gist.github.com/bsikar/8f3f400c6dfd4c118a6ac02730ce8a01

this contains: tdarr-ubuntu-22-privileged.log tdarr-ubuntu-22-unprivileged.log tdarr-ubuntu-24-unprivileged.log

i was unable to get a 24-priviliged working, but i will now test the vm setups. (these logs show whats already been shown above, but with the extra flag you mentioned about)

rigaya commented 3 months ago

Thank you for the debug logs.

Although I'm not sure, it might be some OpenCL related issue, adding --no-avoid-idle-clock might make it work fine.

bsikar commented 3 months ago

Thank you, I will try that. I will also try and do vm setup today. Hopefully once I find a working solution, anyone else that has these issues with Proxmox can follow what I did :)

Error: Unknown option: --no-avoid-idle-clock

Did you mean option(s) below?
  --avoid-idle-clock

I will use the flag --avoid-idle-clock

I think --avoid-idle-clock off is what you want, but i can also try with --avoid-idle-clock on just to see what happens

bsikar commented 3 months ago

Thank you so much for helping me with this, i think that did the trick!!

https://gist.github.com/bsikar/53e79772bb340671bf2dc50e029ce547

i used --avoid-idle-clock off and its now working. you are so smart, thank you so much!!!

rigaya commented 3 months ago

Thank you for confirming, nice to hear that --avoid-idle-clock off worked.

Although I'm unsure why this is needed, it seems like the OpenCL thread has hung up.

Thank again for reporting and sharing the log results.

rigaya / QSVEnc

Output Stalls / Hangs (never completes / finishes) #210