stevezau / plex_generate_vid_previews

Speed up the process of generating preview thumbnails for your Plex media library.
66 stars 12 forks source link

NVIDIA HW Acceleration seems overly slow #51

Closed Crunch41 closed 3 days ago

Crunch41 commented 1 week ago

Just following up on my issue with gpu hw acceleration not working as requested, I left the script running for as of the console logs below just over 6 hours with my gpu & cpu threads both set to 6 threads & upon checking I have a variety of mkv files that can apparently use my gpu & a bunch that won't as shown by my console logs. I also tested with only gpu threads in my env file & the same issue occurs but upon relaunching the script everything appears to be using the gpu as it should so think that resolved itself.

Looking at some of those generation times even some of the files with "HW = True" seem much slower than if I were to just directly generate the thumbnails with plex itself. My PC is running with a 4090 so for it to be functioning like this seems extremely odd but granted this might just be a h265 + limitations of gigabit SMB issue & things are running as expected.

No matter the amount of threads this still seems to just be overly slow in general so not sure what's going on with this one but I've added my console logs below.

The script will also remain frozen upon starting it with something along the lines of this until it seems to complete a file then it will start to show files being generated & the status of hw acceleration.

2024/10/09 03:10:08 | ℹ️ - Getting the media files from library 'Movies' 2024/10/09 03:10:09 | ℹ️ - Got 243 media files for library Movies ⠧ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12% 0:00:07 30/243


`2024/10/08 21:21:35 | ℹ️  - Getting the media files from library 'Movies'
2024/10/08 21:21:35 | ℹ️  - Got 243 media files for library Movies
⠋ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 21:36:51 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/10 Lives (2024)
{tmdb-567811}/10 Lives (2024) {tmdb-567811} [WEBDL-1080p][EAC3 2.0]-WB60.mkv HW=True TIME=911.8seconds SPEED=5.54x
⠧ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 21:38:02 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Arcadian (2024)
{tmdb-1051896}/Arcadian (2024) {tmdb-1051896} [WEBDL-1080p][EAC3 5.1]-FLUX.mkv HW=True TIME=980.9seconds SPEED=5.62x
⠴ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 21:43:37 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Alien Romulus (2024)
{tmdb-945961}/Alien Romulus (2024) {tmdb-945961} [CAM][AC3 2.0]-AMC.mkv HW=True TIME=1314.3seconds SPEED=5.11x
⠇ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 22:29:02 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Abigail (2024)
{tmdb-1111873}/Abigail (2024) {tmdb-1111873} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv HW=False
TIME=4035.2seconds SPEED=1.63x
⠇ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 22:37:27 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Atlas (2024)
{tmdb-614933}/Atlas (2024) {tmdb-614933} [WEBDL-1080p][EAC3 Atmos 5.1]-FLUX.mkv HW=False TIME=1539.9seconds SPEED=4.69x
⠇ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 22:44:30 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Alone (2020)
{tmdb-509635}/Alone (2020) {tmdb-509635} [Remux-1080p][DTS-HD MA 5.1][AVC]-FraMeSToR.mkv HW=True TIME=4968.6seconds
SPEED=1.18x
⠹ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 22:46:50 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/57 Seconds (2023)
{tmdb-937249}/57 Seconds (2023) {tmdb-937249} [Remux-1080p][DTS-HD MA 5.1]-FraMeSToR.mkv HW=False TIME=5106.3seconds
SPEED=1.17x
⠋ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 22:47:50 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/As They Made Us (2022)
{tmdb-644100}/As They Made Us (2022) {tmdb-644100} [WEBDL-2160p][EAC3 5.1]-dB.mkv HW=False TIME=3189.2seconds
SPEED=1.87x
⠸ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 22:55:28 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/The Adventures of Tintin
(2011) {tmdb-17578}/The Adventures of Tintin (2011) {tmdb-17578} [Remux-1080p][DTS-HD MA 7.1][AVC]-RATiObump.mkv
HW=False TIME=5608.2seconds SPEED=1.14x
⠇ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:02:55 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Aquaman and the Lost Kingdom
(2023) {tmdb-572802}/Aquaman and the Lost Kingdom (2023) {tmdb-572802} [Bluray-2160p][DV HDR10][TrueHD Atmos
7.1]-W4NK3R.mkv HW=True TIME=6073.3seconds SPEED=1.23x
⠧ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:26:18 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Beaconsfield (2012)
{tmdb-103883}/Beaconsfield (2012) {tmdb-103883} [WEBDL-1080p][EAC3 2.0]-iJP.mkv HW=False TIME=695.4seconds SPEED=11.2x
⠹ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:28:06 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Ad Astra (2019)
{tmdb-419704}/Ad Astra (2019) {tmdb-419704} [Hybrid][Remux-2160p][HDR10Plus][TrueHD Atmos 7.1][HEVC]-WiLDCAT.mkv HW=True
TIME=7583.5seconds SPEED=0.974x
⠋ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:35:39 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Battle Over Britain (2023)
{tmdb-1136318}/Battle Over Britain (2023) {tmdb-1136318} [Remux-1080p][DTS-HD MA 5.1][AVC]-TRiToN.mkv HW=False
TIME=1494.8seconds SPEED=3.23x
⠇ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:35:39 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/2 Fast 2 Furious (2003)
{tmdb-584}/2 Fast 2 Furious (2003) {tmdb-584} [Remux-2160p][HDR10][DTS-X 7.1][HEVC]-PmP.mkv HW=False TIME=8018.0seconds
SPEED=0.805x
⠦ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:40:04 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Arrival (2016)
{tmdb-329865}/Arrival (2016) {tmdb-329865} [Remux-2160p][HDR10][DTS-HD MA 7.1][HEVC]-FraMeSToR.mkv HW=True
TIME=6393.4seconds SPEED=1.09x
⠼ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2432024/10/08 23:49:01 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/1917 (2019)
{tmdb-530915}/1917 (2019) {tmdb-530915} [Remux-2160p][HDR10Plus][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv HW=True
TIME=8843.5seconds SPEED=0.808x
⠧ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   1% -:--:--   3/2432024/10/08 23:56:08 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Angel Has Fallen (2019)
{tmdb-423204}/Angel Has Fallen (2019) {tmdb-423204} [Remux-2160p][DV HDR10][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv
HW=True TIME=9268.1seconds SPEED=0.784x
⠸ Working... ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   4% -:--:--   9/2432024/10/09 00:06:51 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Beverly Hills Cop Axel F
(2024) {tmdb-280180}/Beverly Hills Cop Axel F (2024) {tmdb-280180} [WEBDL-2160p][DV HDR10][EAC3 Atmos 5.1]-FLUX.mkv
HW=True TIME=941.5seconds SPEED=7.52x
⠴ Working... ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   4% -:--:--   9/2432024/10/09 00:09:15 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Beetlejuice Beetlejuice
(2024) {tmdb-917496}/Beetlejuice Beetlejuice (2024) {tmdb-917496} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv
HW=True TIME=1636.2seconds SPEED=3.84x
⠇ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:14:45 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/The Bikeriders (2024)
{tmdb-1008409}/The Bikeriders (2024) {tmdb-1008409} [WEBDL-1080p][EAC3 Atmos 5.1]-LetMeRideYou.mkv HW=True
TIME=695.2seconds SPEED=10.1x
⠇ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:17:03 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Bad Boys for Life (2020)
{tmdb-38700}/Bad Boys for Life (2020) {tmdb-38700} [Remux-2160p][HDR10][DTS-X 7.1][HEVC]-FraMeSToR.mkv HW=True
TIME=4181.1seconds SPEED=1.78x
⠏ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:19:57 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/The Belko Experiment (2016)
{tmdb-341006}/The Belko Experiment (2016) {tmdb-341006} [Remux-1080p][DTS-HD MA 5.1][AVC]-FraMeSToR.mkv HW=False
TIME=1870.7seconds SPEED=2.85x
⠦ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:20:21 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Begin Again (2013)
{tmdb-198277}/Begin Again (2013) {tmdb-198277} [Remux-1080p Proper][DTS-HD MA 5.1][AVC]-FraMeSToR.mkv HW=False
TIME=2109.6seconds SPEED=2.97x
⠸ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:20:55 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Bad Boys (1995)
{tmdb-9737}/Bad Boys (1995) {tmdb-9737} [Remux-2160p][HDR10][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv HW=True
TIME=5317.5seconds SPEED=1.34x
⠼ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:26:27 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Bird Box (2018)
{tmdb-405774}/Bird Box (2018) {tmdb-405774} [WEBDL-2160p][DV HDR10][EAC3 Atmos 5.1][HEVC]-SiC.mkv HW=True
TIME=880.6seconds SPEED=8.45x
⠼ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:31:49 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/The Beekeeper (2024)
{tmdb-866398}/The Beekeeper (2024) {tmdb-866398} [Remux-2160p][DV HDR10][TrueHD Atmos 7.1][HEVC]-TRiToN.mkv HW=False
TIME=4363.4seconds SPEED=1.46x
⠧ Working... ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   6% -:--:--  15/2432024/10/09 00:33:44 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Bad Boys II (2003)
{tmdb-8961}/Bad Boys II (2003) {tmdb-8961} [Remux-2160p][HDR10][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv HW=False
TIME=5179.6seconds SPEED=1.7x
⠹ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:34:13 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Boneyard (2024)
{tmdb-1114738}/Boneyard (2024) {tmdb-1114738} [WEBDL-1080p][EAC3 5.1]-BYNDR.mkv HW=True TIME=303.7seconds SPEED=19.2x
⠹ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:34:33 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Blink Twice (2024)
{tmdb-840705}/Blink Twice (2024) {tmdb-840705} [WEBDL-1080p][EAC3 Atmos 5.1]-FLUX.mkv HW=True TIME=440.4seconds
SPEED=14.1x
⠸ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:36:53 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Black Lotus (2023)
{tmdb-996154}/Black Lotus (2023) {tmdb-996154} [Remux-1080p][TrueHD Atmos 7.1]-FraMeSToR.mkv HW=True TIME=967.0seconds
SPEED=5.55x
⠋ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:37:20 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Bad Boys Ride or Die (2024)
{tmdb-573435}/Bad Boys Ride or Die (2024) {tmdb-573435} [Bluray-1080p][AC3 5.1]-BHDStudio.mp4 HW=False
TIME=5258.2seconds SPEED=1.32x
⠴ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:41:18 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Beetlejuice (1988)
{tmdb-4011}/Beetlejuice (1988) {tmdb-4011} [Hybrid][Remux-2160p][DV HDR10][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv HW=True
TIME=3733.3seconds SPEED=1.48x
⠸ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:43:54 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Breathe (2024)
{tmdb-720321}/Breathe (2024) {tmdb-720321} [Bluray-1080p][EAC3 5.1]-playHD.mkv HW=False TIME=619.9seconds SPEED=9.07x
⠙ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:46:02 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Call Me Country Beyoncé and
Nashvilles Renaissance (2024) {tmdb-1276928}/Call Me Country Beyoncé and Nashvilles Renaissance (2024) {tmdb-1276928}
[WEBDL-1080p][EAC3 2.0]-FLUX.mkv HW=True TIME=150.6seconds SPEED=17.1x
⠋ Working... ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--  17/2432024/10/09 00:48:52 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Borderlands (2024)
{tmdb-365177}/Borderlands (2024) {tmdb-365177} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv HW=True
TIME=1003.7seconds SPEED=6.03x
⠙ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 01:22:38 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Code 8 Part II (2024)
{tmdb-932420}/Code 8 Part II (2024) {tmdb-932420} [WEBDL-2160p][DV HDR10][EAC3 Atmos 5.1]-FLUX.mkv HW=False
TIME=1659.2seconds SPEED=3.63x
⠙ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 01:27:04 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Bridget Joness Baby (2016)
{tmdb-95610}/Bridget Joness Baby (2016) {tmdb-95610} [Remux-1080p][DTS-HD MA 5.1][AVC]-FraMeSToR.mkv HW=True
TIME=3164.6seconds SPEED=2.33x
⠦ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:01:36 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Damaged (2024)
{tmdb-1105407}/Damaged (2024) {tmdb-1105407} [Remux-1080p][DTS-HD MA 5.1][AVC]-TRiToN.mkv HW=False TIME=4008.8seconds
SPEED=1.46x
⠋ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:02:43 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Cherry (2021)
{tmdb-544401}/Cherry (2021) {tmdb-544401} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv HW=True
TIME=4304.7seconds SPEED=1.97x
⠸ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:03:36 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Civil War (2024)
{tmdb-929590}/Civil War (2024) {tmdb-929590} [Bluray-2160p][DV HDR10Plus][TrueHD Atmos 7.1]-MainFrame.mkv HW=True
TIME=4314.7seconds SPEED=1.51x
⠸ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:05:58 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Challengers (2024)
{tmdb-937287}/Challengers (2024) {tmdb-937287} [Remux-1080p][TrueHD Atmos 7.1][AVC]-CiNEPHiLES.mkv HW=True
TIME=4865.5seconds SPEED=1.62x
⠧ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:15:06 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Code 8 (2019)
{tmdb-461130}/Code 8 (2019) {tmdb-461130} [Remux-1080p Proper][DTS-HD MA 5.1][AVC]-PmP.mkv HW=False TIME=4887.5seconds
SPEED=1.21x
⠇ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:49:03 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Danger Close The Battle of
Long Tan (2019) {tmdb-508664}/Danger Close The Battle of Long Tan (2019) {tmdb-508664} [Remux-1080p Proper][DTS-HD MA
5.1][AVC]-PmP.mkv HW=False TIME=6544.2seconds SPEED=1.08x
⠸ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/2432024/10/09 02:56:41 | ℹ️  - Generated Video Preview for //republiccommand/media/plex/movies/Day Shift (2022) {tmdb-755566}/Day Shift (2022) {tmdb-755566} [WEBDL-2160p][DV HDR10][EAC3 Atmos 5.1]-SMURF.mkv HW=True TIME=3025.1seconds SPEED=2.21x
⠴ Working... ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  12% -:--:--  30/243`            
ali-ramadhan commented 1 week ago

Are you sure that your storage media is not the bottleneck?

I've only been playing around with this script for a bit, but I've noticed that 1 GPU thread (or NVENC instance) can read from my HDD at ~100 MiB/s. With HDDs topping out at 200-250 MiB/s, 2 GPU threads are enough to saturate one HDD.

So if all your media is one HDD then you won't see any speedup using more than 2 GPU threads. In fact, you may see a slowdown as different threads/processes compete for resources and thrash the HDD.

stevezau commented 1 week ago

@Crunch41 looking at the log it seems to be using a mixture of CPU and GPU threads... but you are right, the speeds are quite slow.

@ali-ramadhan raises a good point.

Can you share your hardware specs+setup so we can help t-shoot the issue.

Also, please check your io to see if your hdd's are at capacity.

Lastly, as a t-shooting step, i'd try running it with 1 GPU thread to see how it performs.

let me know..

stevezau commented 1 week ago

@Crunch41 also, have you checked the GPU stats when the script is running?

Crunch41 commented 1 week ago

gpu is a rtx 4090 and cpu is a 7800x3d so shouldn't be any bottleneck there. Have disabled cpu threads but cpu usage seems to sit around 20-40% at the start then drops down to around 3-30% while it's generating even then. Currently running 1 gpu thread

My network utilization is extremely odd, Sometimes upon launching the script I can see over 1000-1300 mbps as I'm connected to my server via a 2.5gbe link, Then as it progresses on the same file after about 10-30 seconds the network usage drops and hovers between 80-300mbps with random spikes back over 1000mbps for a few seconds so I've got no clue what's going on there and it doesn't happen to every file.

GPU Utilization remains according to windows at around 1-10% and remains low no matter how many threads, this can fluctuate depending on the file but the usage never looks to be flat out on the gpu.

Right now I have the script running and currently my network usage is about 300-500mbps consistently cpu usage is at 13% and gpu usage is at around 3% with peaks to about 20% according to windows and while typing this it hasn't completed the file even with the constant network traffic in question.

Right now it's at 1.4k images and climbing in my windows tmp folder and it generating about 1-4 images every second by the looks of it

Edit 2 mins later this has now dropped down to around 50-60mbps after using 600 mbps + for about 5 entire minutes and is generating the images much slower but if it's like when I tested before this will resolve itself if I relaunch the script. Disk usage on the server is next to nothing apart from the script so definitely doesn't feel like a hdd bandwidth issue.

Edit 3 still on the same file & network usage has jumped back up to 400mbps+ again and has just hit 3.1k images. No idea what's going on with it lol

stevezau commented 1 week ago

Hmm ok. It is suggesting you have a bottleneck somewhere. Can you copy some files locally and run the script? That way, we can rule out the network

Crunch41 commented 1 week ago

Will try & modify the script to run locally with the same file but looks like this file wasn't using hw acceleration

[Remux-2160p][HDR10][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv HW=False TIME=1250.5seconds SPEED=6.25x

Have had this pop up a few times now so I'm imagining some video codecs aren't compatible? Just looking into the server now to try & double check for bottlenecks

Crunch41 commented 1 week ago

Ok can confirm if i specify the file after copying it to my local drive it generates much faster and gpu usage is much higher and images generate between 20-50 a second, weird thing is that the local disk usage was only about 200mbps vs the 300-1000 it pulls when using the network share. I'm not well versed in python but could it be possible it's bottlenecking when it comes to generating the images in some way then needing to send them back over the network share at the same time? Truly don't understand how the network usage is so much higher yet the speeds are 10x slower.

Almost feels like if you could get the script to copy the video file itself over into the tmp folder before starting any generation work so it generates them all locally then after that completes It'd send everything back over to the plex appdata but something like that is out of my skill set.

ali-ramadhan commented 1 week ago

That all does seem a bit weird so I'm also curious why it it's so slow. I've been running this script locally but curious about usage over a network. Do you know how RAM usage is looking throughout the whole process? Especially cached RAM (at least it's called cached memory on Linux).

My network utilization is extremely odd, Sometimes upon launching the script I can see over 1000-1300 mbps as I'm connected to my server via a 2.5gbe link, Then as it progresses on the same file after about 10-30 seconds the network usage drops and hovers between 80-300mbps with random spikes back over 1000mbps for a few seconds so I've got no clue what's going on there and it doesn't happen to every file.

I'm wondering if data is being loaded over the network and cached in RAM then processed. Then the network connection can be saturated early on as there's plenty of RAM available. But once RAM fills up (perhaps in 10-30 seconds?) now more data can only be loaded if there's space in RAM. And maybe sometimes a lot of RAM can be freed up at once leading to some network spikes.

If your local disk an SSD? I wonder if data is cached differently then. But also reading from an SSD would be much faster than reading over a 2.5 GbE network connection.

Crunch41 commented 1 week ago

Running an nvme ssd disk locally & my plex appdata for thumbnails is also on an nvme ssd on the server side so could definitely see something there but I'm at a dead end personally, Ram usage seems to only stick at around 500 MB on my pc & server ram usage appears to be untouched & as far as I can see both on the server & my pc there doesn't seem to be any obvious bottleneck between the two. Upon copying directly from the disk to my pc I can saturate my link through my network share no problem at around 230 MB but it just doesn't seem to want to go nearly at fast from server to script & back.

stevezau commented 4 days ago

@Crunch41, I'm just following up on this one. I am trying to understand if there is a bug/issue in the script.

Reading the above i get the sense there is some other bottleneck causing the slower than expected processing times as evident by your local test which suggests the script is ok but your setup/config might need some turning to get max speed?

Crunch41 commented 4 days ago

@stevezau

I did pull in some ai assisted help & modified the script to copy files over with robocopy & that immediately solved the speed issue for me. Was hitting a solid 85-98% on my network usage & files moved over a lot faster.

I also changed the script to copy each file it's working on over into my local tmp folder then as mentioned above with robocopy just use that local copy to start image generation as it was my way of getting around the slow image generation.

This was mainly due to the initial gpu usage issue but even with the local copy of the file it still reports hw=false & my gpu load remains next to nothing so I suppose the script could be adjusted back to just using the direct file off of the server. My cpu still seems to be taking the main load so I have no idea what's up with ffmpeg as I've checked and ffmpeg reports the necessary cuda & hw accel are available. I'm almost done with thumbnail generation though so I won't be able to assist with script usage for much longer.

Here's what I currently have in the base script in the process_item def to get this working so hopefully it can be put to some use if need be. I also did try & add logging for the copy percentage + the copy speed before it would start image generation but I had limited success & it would spam the console log so I just removed that.



def process_item(item_key):
    sess = requests.Session()
    sess.verify = False
    plex = PlexServer(PLEX_URL, PLEX_TOKEN, timeout=PLEX_TIMEOUT, session=sess)
    data = plex.query('{}/tree'.format(item_key))
    for media_part in data.findall('.//MediaPart'):
        if 'hash' in media_part.attrib:
            # Filter Processing by HDD Path
            if len(sys.argv) > 1:
                if sys.argv[1] not in media_part.attrib['file']:
                    return
            bundle_hash = media_part.attrib['hash']
            media_file = media_part.attrib['file']
            if not os.path.isfile(media_file):
                logger.error('Skipping as file not found {}'.format(media_file))
                continue
            try:
                bundle_file = '{}/{}{}'.format(bundle_hash[0], bundle_hash[1::1], '.bundle')
            except Exception as e:
                logger.error('Error generating bundle_file for {} due to {}:{}'.format(media_file, type(e).__name__, str(e)))
                continue
            bundle_path = os.path.join(PLEX_LOCAL_MEDIA_PATH, 'localhost', bundle_file)
            indexes_path = os.path.join(bundle_path, 'Contents', 'Indexes')
            index_bif = os.path.join(indexes_path, 'index-sd.bif')
            tmp_path = os.path.join(TMP_FOLDER, bundle_hash)
            if not os.path.isfile(index_bif):
                if not os.path.isdir(indexes_path):
                    try:
                        os.makedirs(indexes_path)
                    except OSError as e:
                        logger.error('Error generating images for {}. `{}:{}` error when creating index path {}'.format(media_file, type(e).__name__, str(e), indexes_path))
                        continue
                try:
                    if not os.path.isdir(tmp_path):
                        os.makedirs(tmp_path)
                except OSError as e:
                    logger.error('Error generating images for {}. `{}:{}` error when creating tmp path {}'.format(media_file, type(e).__name__, str(e), tmp_path))
                    continue

                # Copy the file locally using robocopy
                try:
                    logger.info(f'Copying {media_file} to {tmp_path}')
                    source_dir = os.path.dirname(media_file)
                    file_name = os.path.basename(media_file)
                    robocopy_command = f'robocopy "{source_dir}" "{tmp_path}" "{file_name}" /R:1 /W:1 /NJH /NJS /NDL /NC /NS'
                    result = subprocess.run(robocopy_command, shell=True, capture_output=True, text=True)

                    # robocopy uses unconventional exit codes, 0-7 can indicate success
                    if result.returncode > 7:
                        raise subprocess.CalledProcessError(result.returncode, robocopy_command)

                    logger.info(f'Successfully copied {media_file} to {tmp_path}')
                except subprocess.CalledProcessError as e:
                    logger.error(f'Error copying file {media_file} to {tmp_path}: {type(e).__name__}: {str(e)}')
                    if os.path.exists(tmp_path):
                        shutil.rmtree(tmp_path)
                    continue

                local_media_file = os.path.join(tmp_path, os.path.basename(media_file))

                try:
                    generate_images(local_media_file, tmp_path)
                except Exception as e:
                    logger.error('Error generating images for {}. `{}: {}` error when generating images'.format(local_media_file, type(e).__name__, str(e)))
                    if os.path.exists(tmp_path):
                        shutil.rmtree(tmp_path)
                    continue
                try:
                    generate_bif(index_bif, tmp_path)
                except Exception as e:
                    # Remove bif, as it prob failed to generate
                    if os.path.exists(index_bif):
                        os.remove(index_bif)
                    logger.error('Error generating images for {}. `{}:{}` error when generating bif'.format(local_media_file, type(e).__name__, str(e)))
                    continue
                finally:
                    if os.path.exists(local_media_file):
                        os.remove(local_media_file)
                    if os.path.exists(tmp_path):
                        shutil.rmtree(tmp_path)
stevezau commented 4 days ago

Thanks, @Crunch41.. thinking.. I don't think I'll modify the script to copy to local as I suspect it will cause more problems than it solves. But, if required, I will redirect folks to this ticket with your code above..

RE hw=false. The script uses a mixture of CPU and GPU, unless you set CPU to 0 .. (but I've never tested that).

Did you set it to 0?

Crunch41 commented 4 days ago

Yeah It'd probably be best to switch the code back to just using the direct file but this seems like it'll do the trick for now.

Not sure why the python network usage seems to be so inconsistent & slow vs robocopy but just glad it's running as it should.

& Yeah I did set my cpu to 0 threads as I wanted to try & prioritize gpu usage but even then it'll still use my cpu as required I suppose as even at 0 or when I tested 4-8 threads cpu usage was still the main contender. I currently just have gpu threads set to 1 & cpu threads on 0 as it's quicker to just copy over 1 file at a time on my end but it'll scale as needed to copy multiple files when adjusted higher.

stevezau commented 4 days ago

Hmm ok, it shouldn't use CPU so that might be a bug. I'd be happy to look into it if you are open to t-shooting it?

Crunch41 commented 4 days ago

I'm happy to try & help out, Have 1 decently big movie file that's left in my movie queue that's about 22gb so I can just cancel image generation on that before it finishes & use that as a test file.

stevezau commented 4 days ago

ok.. I think we need to add some debug statements to the script to show why it chooses CPU vs GPU and to output ffmpeg.

Are you able to edit the script and run it again (ill give you the code) or are you using the docker container??

Crunch41 commented 4 days ago

Still running via windows so I'm happy to just modify the code

stevezau commented 4 days ago

ok ill edit it now.. remind me, what gpu do you have?

Crunch41 commented 4 days ago

running a nvidia rtx 4090

cpu is a 7800x3d

stevezau commented 4 days ago

better yet, i will add proper debug logging support to the script. Give me a little while.

Crunch41 commented 4 days ago

Yeah no worries, I've got thread notifs on so just drop a message in here once you're all sorted & I'll give it a crack.

stevezau commented 4 days ago

I've just pushed some changes and released v 1.2.1 of the docker container. can you try again and show me the logs?

stevezau commented 4 days ago

Oh, you need to set ENV var LOG_LEVEL=debug

Crunch41 commented 4 days ago

getting these errors


 line 89, in <module>
    logger.add(
    ~~~~~~~~~~^
        lambda _: console.print(_, end=''),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        enqueue=True
        ^^^^^^^^^^^^
    )

     line 930, in add
    levelno = self.level(level).no

     line 1577, in level
    raise ValueError("Level '%s' does not exist" % name) from None
ValueError: Level 'debug' does not exist
stevezau commented 4 days ago

Ah. I need to fix that. But for now. can you try set the var to DEBUG (all caps) and try again?

Crunch41 commented 4 days ago

Here's what it outputs,



⠦ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2532024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/2 Fast 2 Furious (2003)
{tmdb-584}/2 Fast 2 Furious (2003) {tmdb-584} [Remux-2160p][HDR10][DTS-X 7.1][HEVC]-PmP.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\8/0c56d43afef2eb8dff7806bc21461b0987982e7.bundle
2024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/10 Lives (2024)
{tmdb-567811}/10 Lives (2024) {tmdb-567811} [WEBDL-1080p][EAC3 2.0]-WB60.mkv at Z:\appdata\plex\Library\Application
Support\Plex Media Server\Media\localhost\1/9f35f0f7f8d4c664a84b086a51a0e09be073283.bundle
2024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/57 Seconds (2023)
{tmdb-937249}/57 Seconds (2023) {tmdb-937249} [Remux-1080p][DTS-HD MA 5.1]-FraMeSToR.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\5/cb075ea17899f228d92e5dcd0e4db69f6293c6b.bundle
⠇ Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--   0/2532024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/1917 (2019)
{tmdb-530915}/1917 (2019) {tmdb-530915} [Remux-2160p][HDR10Plus][TrueHD Atmos 7.1][HEVC]-FraMeSToR.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\7/a98d0152ab5479716d44e6f5266c3fb3355bfd9.bundle
2024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/Abigail (2024)
{tmdb-1111873}/Abigail (2024) {tmdb-1111873} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\3/376045bb753c55f4b5e0e12593955fa180adadd.bundle
2024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/Ad Astra (2019)
{tmdb-419704}/Ad Astra (2019) {tmdb-419704} [Hybrid][Remux-2160p][HDR10Plus][TrueHD Atmos 7.1][HEVC]-WiLDCAT.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\3/d944f7fcd36cc7f656c61daf37589736cfbc4d3.bundle
2024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/The Adventures of Tintin
(2011) {tmdb-17578}/The Adventures of Tintin (2011) {tmdb-17578} [Remux-1080p][DTS-HD MA 7.1][AVC]-RATiObump.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\d/a437d9a54f018be4a7b3159431ac2b705028d26.bundle
⠏ Working... ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   2% -:--:--   4/2532024/10/15 12:46:34 | 🐞  - Generating bundle_file for //republiccommand/media/plex/movies/Alien Romulus (2024)
{tmdb-945961}/Alien Romulus (2024) {tmdb-945961} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv at
Z:\appdata\plex\Library\Application Support\Plex Media
Server\Media\localhost\e/d3bdeb8e4329ae13d30a6dd2477250e7fdf578e.bundle
⠇ Working... ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   3% 0:00:10   7/2532024/10/15 12:46:35 | 🐞  - Running ffmpeg
2024/10/15 12:46:35 | 🐞  - C:\Users\user\Downloads\ffmpeg-7.1-essentials_build\bin\ffmpeg.EXE -loglevel info
-skip_frame:v nokey -threads:0 1 -i //republiccommand/media/plex/movies/Alien Romulus (2024) {tmdb-945961}/Alien Romulus
(2024) {tmdb-945961} [WEBDL-2160p][DV HDR10Plus][EAC3 Atmos 5.1]-FLUX.mkv -an -sn -dn -q:v 4 -vf
fps=fps=0.5:round=up,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt70
9:m=bt709:r=tv,format=yuv420p,scale=w=320:h=240:force_original_aspect_ratio=decrease
C:\Users\user\Downloads\plexpreviews/thumbnails\ed3bdeb8e4329ae13d30a6dd2477250e7fdf578e/img-%06d.jpg
⠋ Working... ━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   3% 0:00:10   7/253
stevezau commented 4 days ago

hmm, have you got the earlier logs lines also? Where it tries to detect your GPU?

Crunch41 commented 4 days ago

2024/10/15 12:54:08 | ℹ️ - GPU Detection (with AMD Support) was recently added to this script. 2024/10/15 12:54:08 | ℹ️ - Please log issues here https://github.com/stevezau/plex_generate_vid_previews/issues 2024/10/15 12:54:08 | ℹ️ - Found NVIDIA GPU 2024/10/15 12:54:08 | ℹ️ - Getting the media files from library 'Movies' 2024/10/15 12:54:09 | ℹ️ - Got 253 media files for library Movies

stevezau commented 4 days ago

@Crunch41 are you on discord? will be faster than GH

Crunch41 commented 4 days ago

Yeah I am, discord user is the same as github

stevezau commented 4 days ago

Ok, identified the issue, pushed a fix ver 1.2.2.

@Crunch41 to verify.

Crunch41 commented 3 days ago

@stevezau can confirm gpu usage is now detectable & is in use, Usage is still low at around 5-10% on the gpu decoder & cpu usage I saw range between 25-40% even with cpu threads still on 0 but I imagine there's encoding that the cpu is just forced to do but it's a great speed up vs the previous version & I was still seeing fluctuating speeds of 500-1000 mbps. The script now also reports HW=True as it should so looks like you nailed that bug.

As for the robocopy implemented version this was somewhat more consistent and a bit faster as gpu decoder usage was around the 8-25% mark for one of my files & cpu usage sat around 40-65% & seems to be somewhat faster than using the native version of the script which I imagine is due to the file being copied then generating locally vs the slower way that ffmpeg uses with the script when not using a local file & pulling straight from the source. Robocopy would definitely be superior for windows in all of my testing if that could be used for direct ffmpeg generation as the script normally does but I'm unsure if that's even possible so can't help there & this will do the job for me.

Either the native version or the robocopy version will handle smaller files without any issues but you tend to notice when it's huge movie file sizes as without that full network speed image generation times can noticably be impacted while generating 2k+ images. I also only had a few big files to test with leftover so it could very much be different usages per file format & encoding. Either way now that the main script is correctly detecting gpu usage the impact isn't nearly as bad as it was before.

My robocopy version of the script is below which should in theory use robocopy only if you run on a windows pc which seemed to detect properly as noted in the logging otherwise it'll fall back to not using robocopy on a non windows pc or if an error occurs but I haven't tested it on linux myself so I can only vouch for windows working as of the code below. Hopefully if needed this will help someone if they ever have a similar speed issue while using a network share & I'm sure this could be modified further if need be.



#!/usr/bin/env python3
import sys
import re
import subprocess
import shutil
import glob
import os
import struct
import urllib3
import array
import time
import platform
from concurrent.futures import ProcessPoolExecutor

from dotenv import load_dotenv

load_dotenv()

PLEX_URL = os.environ.get('PLEX_URL', '')  # Plex server URL. can also use for local server: http://localhost:32400
PLEX_TOKEN = os.environ.get('PLEX_TOKEN', '')  # Plex Authentication Token
PLEX_BIF_FRAME_INTERVAL = int(os.environ.get('PLEX_BIF_FRAME_INTERVAL', 5))  # Interval between preview images
THUMBNAIL_QUALITY = int(os.environ.get('THUMBNAIL_QUALITY', 4))  # Preview image quality (2-6)
PLEX_LOCAL_MEDIA_PATH = os.environ.get('PLEX_LOCAL_MEDIA_PATH', '/path_to/plex/Library/Application Support/Plex Media Server/Media')  # Local Plex media path
TMP_FOLDER = os.environ.get('TMP_FOLDER', '/dev/shm/plex_generate_previews')  # Temporary folder for preview generation
PLEX_TIMEOUT = int(os.environ.get('PLEX_TIMEOUT', 60))  # Timeout for Plex API requests (seconds)

# Path mappings for remote preview generation. # So you can have another computer generate previews for your Plex server
# If you are running on your plex server, you can set both variables to ''
PLEX_LOCAL_VIDEOS_PATH_MAPPING = os.environ.get('PLEX_LOCAL_VIDEOS_PATH_MAPPING', '')  # Local video path for the script
PLEX_VIDEOS_PATH_MAPPING = os.environ.get('PLEX_VIDEOS_PATH_MAPPING', '')  # Plex server video path

GPU_THREADS = int(os.environ.get('GPU_THREADS', 4))  # Number of GPU threads for preview generation
CPU_THREADS = int(os.environ.get('CPU_THREADS', 4))  # Number of CPU threads for preview generation

# Set the timeout envvar for https://github.com/pkkid/python-plexapi
os.environ["PLEXAPI_PLEXAPI_TIMEOUT"] = str(PLEX_TIMEOUT)

if not shutil.which("mediainfo"):
    print('MediaInfo not found.  MediaInfo must be installed and available in PATH.')
    sys.exit(1)
try:
    from pymediainfo import MediaInfo
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install pymediainfo".')
    sys.exit(1)
try:
    import gpustat
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install gpustat".')
    sys.exit(1)

try:
    import requests
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install requests".')
    sys.exit(1)

try:
    from plexapi.server import PlexServer
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install plexapi".')
    sys.exit(1)

try:
    from loguru import logger
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install loguru".')
    sys.exit(1)

try:
    from rich.console import Console
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install rich".')
    sys.exit(1)

try:
    from rich.progress import Progress, SpinnerColumn, MofNCompleteColumn
except ImportError:
    print('Dependencies Missing!  Please run "pip3 install rich".')
    sys.exit(1)

FFMPEG_PATH = shutil.which("ffmpeg")
if not FFMPEG_PATH:
    print('FFmpeg not found.  FFmpeg must be installed and available in PATH.')
    sys.exit(1)

# Logging setup
console = Console()
logger.remove()
logger.add(
    lambda _: console.print(_, end=''),
    level=os.environ.get('LOG_LEVEL', 'INFO').upper(),
    format='<green>{time:YYYY/MM/DD HH:mm:ss}</green> | {level.icon}'
    + '  - <level>{message}</level>',
    enqueue=True
)

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

def copy_file(src, dst):
    if platform.system() == 'Windows':
        try:
            result = subprocess.run(['robocopy', os.path.dirname(src), os.path.dirname(dst), os.path.basename(src)], 
                                    check=False, capture_output=True, text=True)

            # Robocopy uses bitwise flags for its return codes
            # Values less than 8 indicate successful operation (with or without some mismatches/warnings)
            if result.returncode >= 8:
                logger.error(f"Robocopy error: {result.stderr}")
                raise subprocess.CalledProcessError(result.returncode, result.args, result.stdout, result.stderr)
            else:
                logger.info(f"Robocopy completed the copy successfully & image generation is now starting. Robocopy return code: {result.returncode}")
        except subprocess.CalledProcessError as e:
            logger.error(f"Robocopy failed: {e}")
            raise
    else:
        shutil.copy2(src, dst)

def detect_gpu():
    # Check for NVIDIA GPUs
    try:
        import pynvml
        pynvml.nvmlInit()
        num_nvidia_gpus = pynvml.nvmlDeviceGetCount()
        pynvml.nvmlShutdown()
        if num_nvidia_gpus > 0:
            return 'NVIDIA'
    except ImportError:
        logger.warning("NVIDIA GPU detection library (pynvml) not found. NVIDIA GPUs will not be detected.")
    except pynvml.NVMLError as e:
        logger.warning(f"Error initializing NVIDIA GPU detection {e}. NVIDIA GPUs will not be detected.")

    # Check for AMD GPUs
    try:
        from amdsmi import amdsmi_interface
        amdsmi_interface.amdsmi_init()
        devices = amdsmi_interface.amdsmi_get_processor_handles()
        found = None
        if len(devices) > 0:
            for device in devices:
                processor_type = amdsmi_interface.amdsmi_get_processor_type(device)
                if processor_type == amdsmi_interface.AMDSMI_PROCESSOR_TYPE_GPU:
                    found = True
        amdsmi_interface.amdsmi_shut_down()
        if found:
                vaapi_device_dir = "/dev/dri"
                if os.path.exists(vaapi_device_dir):
                    for entry in os.listdir(vaapi_device_dir):
                        if entry.startswith("renderD"):
                            return os.path.join(vaapi_device_dir, entry)
    except ImportError:
        logger.warning("AMD GPU detection library (amdsmi) not found. AMD GPUs will not be detected.")
    except Exception as e:
        logger.warning(f"Error initializing AMD GPU detection: {e}. AMD GPUs will not be detected.")

def get_amd_ffmpeg_processes():
    from amdsmi import amdsmi_init, amdsmi_shut_down, amdsmi_get_processor_handles, amdsmi_get_gpu_process_list
    try:
        amdsmi_init()
        gpu_handles = amdsmi_get_processor_handles()
        ffmpeg_processes = []

        for gpu in gpu_handles:
            processes = amdsmi_get_gpu_process_list(gpu)
            for process in processes:
                if process['name'].lower().startswith('ffmpeg'):
                    ffmpeg_processes.append(process)

        return ffmpeg_processes
    finally:
        amdsmi_shut_down()

def generate_images(video_file_param, output_folder, gpu):
    video_file = video_file_param.replace(PLEX_VIDEOS_PATH_MAPPING, PLEX_LOCAL_VIDEOS_PATH_MAPPING)

    # Create a temporary directory for the copied file
    temp_dir = os.path.join(TMP_FOLDER, 'temp_video')
    os.makedirs(temp_dir, exist_ok=True)
    temp_video_file = os.path.join(temp_dir, os.path.basename(video_file))

    try:
        # Copy the file using robocopy on Windows, or shutil.copy2 on other systems
        copy_file(video_file, temp_video_file)
    except Exception as e:
        logger.error(f"Error copying file: {e}")
        return

    media_info = MediaInfo.parse(temp_video_file)
    vf_parameters = "fps=fps={}:round=up,scale=w=320:h=240:force_original_aspect_ratio=decrease".format(
        round(1 / PLEX_BIF_FRAME_INTERVAL, 6))

    # Check if we have a HDR Format. Note: Sometimes it can be returned as "None" (string) hence the check for None type or "None" (String)
    if media_info.video_tracks:
        if media_info.video_tracks[0].hdr_format != "None" and media_info.video_tracks[0].hdr_format is not None:
            vf_parameters = "fps=fps={}:round=up,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p,scale=w=320:h=240:force_original_aspect_ratio=decrease".format(round(1 / PLEX_BIF_FRAME_INTERVAL, 6))

    args = [
        FFMPEG_PATH, "-loglevel", "info", "-skip_frame:v", "nokey", "-threads:0", "1", "-i",
        temp_video_file, "-an", "-sn", "-dn", "-q:v", str(THUMBNAIL_QUALITY),
        "-vf",
        vf_parameters, '{}/img-%06d.jpg'.format(output_folder)
    ]

    start = time.time()
    hw = False

    if gpu == 'NVIDIA':
        gpu_stats_query = gpustat.core.new_query()
        logger.debug('Trying to determine how many GPU threads running')
        if len(gpu_stats_query):
            gpu_ffmpeg = []
            for gpu_stats in gpu_stats_query:
                for process in gpu_stats.processes:
                    if 'ffmpeg' in process["command"].lower():
                        gpu_ffmpeg.append(process["command"])

            logger.debug('Counted {} ffmpeg GPU threads running'.format(len(gpu_ffmpeg)))
            if len(gpu_ffmpeg) > GPU_THREADS:
                logger.debug('Hit limit on GPU threads, defaulting back to CPU')
            if len(gpu_ffmpeg) < GPU_THREADS or CPU_THREADS == 0:
                hw = True
                args.insert(5, "-hwaccel")
                args.insert(6, "cuda")
    elif gpu:
        # Must be AMD
        gpu_ffmpeg = get_amd_ffmpeg_processes()
        logger.debug('Counted {} ffmpeg GPU threads running'.format(len(gpu_ffmpeg)))
        if len(gpu_ffmpeg) > GPU_THREADS:
            logger.debug('Hit limit on GPU threads, defaulting back to CPU')

        if len(gpu_ffmpeg) < GPU_THREADS or CPU_THREADS == 0:
            hw = True
            args.insert(5, "-hwaccel")
            args.insert(6, "vaapi")
            args.insert(7, "-vaapi_device")
            args.insert(8, gpu)
            # Adjust vf_parameters for AMD VAAPI
            vf_parameters = vf_parameters.replace("scale=w=320:h=240:force_original_aspect_ratio=decrease", "format=nv12|vaapi,hwupload,scale_vaapi=w=320:h=240:force_original_aspect_ratio=decrease")
            args[args.index("-vf") + 1] = vf_parameters

    logger.debug('Running ffmpeg')
    logger.debug(' '.join(args))
    proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    # Allow time for it to start
    time.sleep(1)

    out, err = proc.communicate()
    if proc.returncode != 0:
        err_lines = err.decode('utf-8', 'ignore').split('\n')[-5:]
        logger.error(err_lines)
        logger.error('Problem trying to ffmpeg images for {}'.format(video_file))

    logger.debug('FFMPEG Command output')
    logger.debug(out)

    # Speed
    end = time.time()
    seconds = round(end - start, 1)
    speed = re.findall('speed= ?([0-9]+\\.?[0-9]*|\\.[0-9]+)x', err.decode('utf-8', 'ignore'))
    if speed:
        speed = speed[-1]

    # Optimize and Rename Images
    for image in glob.glob('{}/img*.jpg'.format(output_folder)):
        frame_no = int(os.path.basename(image).strip('-img').strip('.jpg')) - 1
        frame_second = frame_no * PLEX_BIF_FRAME_INTERVAL
        os.rename(image, os.path.join(output_folder, '{:010d}.jpg'.format(frame_second)))

    # Clean up the temporary video file
    os.remove(temp_video_file)
    os.rmdir(temp_dir)

    logger.info('Generated Video Preview for {} HW={} TIME={}seconds SPEED={}x '.format(video_file, hw, seconds, speed))

def generate_bif(bif_filename, images_path):
    """
    Build a .bif file
    @param bif_filename name of .bif file to create
    @param images_path Directory of image files 00000001.jpg
    """
    magic = [0x89, 0x42, 0x49, 0x46, 0x0d, 0x0a, 0x1a, 0x0a]
    version = 0

    images = [img for img in os.listdir(images_path) if os.path.splitext(img)[1] == '.jpg']
    images.sort()

    f = open(bif_filename, "wb")
    array.array('B', magic).tofile(f)
    f.write(struct.pack("<I", version))
    f.write(struct.pack("<I", len(images)))
    f.write(struct.pack("<I", 1000 * PLEX_BIF_FRAME_INTERVAL))
    array.array('B', [0x00 for x in range(20, 64)]).tofile(f)

    bif_table_size = 8 + (8 * len(images))
    image_index = 64 + bif_table_size
    timestamp = 0

    # Get the length of each image
    for image in images:
        statinfo = os.stat(os.path.join(images_path, image))
        f.write(struct.pack("<I", timestamp))
        f.write(struct.pack("<I", image_index))
        timestamp += 1
        image_index += statinfo.st_size

    f.write(struct.pack("<I", 0xffffffff))
    f.write(struct.pack("<I", image_index))

    # Now copy the images
    for image in images:
        data = open(os.path.join(images_path, image), "rb").read()
        f.write(data)

    f.close()

def process_item(item_key, gpu):
    sess = requests.Session()
    sess.verify = False
    plex = PlexServer(PLEX_URL, PLEX_TOKEN, timeout=PLEX_TIMEOUT, session=sess)

    data = plex.query('{}/tree'.format(item_key))

    for media_part in data.findall('.//MediaPart'):
        if 'hash' in media_part.attrib:
            # Filter Processing by HDD Path
            if len(sys.argv) > 1:
                if sys.argv[1] not in media_part.attrib['file']:
                    return
            bundle_hash = media_part.attrib['hash']
            media_file = media_part.attrib['file']

            if not os.path.isfile(media_file):
                logger.error('Skipping as file not found {}'.format(media_file))
                continue

            try:
                bundle_file = '{}/{}{}'.format(bundle_hash[0], bundle_hash[1::1], '.bundle')
            except Exception as e:
                logger.error('Error generating bundle_file for {} due to {}:{}'.format(media_file, type(e).__name__, str(e)))
                continue

            bundle_path = os.path.join(PLEX_LOCAL_MEDIA_PATH, 'localhost', bundle_file)
            indexes_path = os.path.join(bundle_path, 'Contents', 'Indexes')
            index_bif = os.path.join(indexes_path, 'index-sd.bif')
            tmp_path = os.path.join(TMP_FOLDER, bundle_hash)

            if not os.path.isfile(index_bif):
                logger.debug('Generating bundle_file for {} at {}'.format(media_file, index_bif))

                if not os.path.isdir(indexes_path):
                    try:
                        os.makedirs(indexes_path)
                    except OSError as e:
                        logger.error('Error generating images for {}. `{}:{}` error when creating index path {}'.format(media_file, type(e).__name__, str(e), indexes_path))
                        continue

                try:
                    if not os.path.isdir(tmp_path):
                        os.makedirs(tmp_path)
                except OSError as e:
                    logger.error('Error generating images for {}. `{}:{}` error when creating tmp path {}'.format(media_file, type(e).__name__, str(e), tmp_path))
                    continue

                try:
                    generate_images(media_part.attrib['file'], tmp_path, gpu)
                except Exception as e:
                    logger.error('Error generating images for {}. `{}: {}` error when generating images'.format(media_file, type(e).__name__, str(e)))
                    if os.path.exists(tmp_path):
                        shutil.rmtree(tmp_path)
                    continue

                try:
                    generate_bif(index_bif, tmp_path)
                except Exception as e:
                    # Remove bif, as it prob failed to generate
                    if os.path.exists(index_bif):
                        os.remove(index_bif)
                    logger.error('Error generating images for {}. `{}:{}` error when generating bif'.format(media_file, type(e).__name__, str(e)))
                    continue
                finally:
                    if os.path.exists(tmp_path):
                        shutil.rmtree(tmp_path)

def run(gpu):
    # Ignore SSL Errors
    sess = requests.Session()
    sess.verify = False

    plex = PlexServer(PLEX_URL, PLEX_TOKEN, session=sess)

    for section in plex.library.sections():
        logger.info('Getting the media files from library \'{}\''.format(section.title))

        if section.METADATA_TYPE == 'episode':
            media = [m.key for m in section.search(libtype='episode')]
        elif section.METADATA_TYPE == 'movie':
            media = [m.key for m in section.search()]
        else:
            logger.info('Skipping library {} as \'{}\' is unsupported'.format(section.title, section.METADATA_TYPE))
            continue

        logger.info('Got {} media files for library {}'.format(len(media), section.title))

        with Progress(SpinnerColumn(), *Progress.get_default_columns(), MofNCompleteColumn(), console=console) as progress:
            with ProcessPoolExecutor(max_workers=CPU_THREADS + GPU_THREADS) as process_pool:
                futures = [process_pool.submit(process_item, key, gpu) for key in media]
                for future in progress.track(futures):
                    future.result()

if __name__ == '__main__':
    logger.info('GPU Detection (with AMD Support) was recently added to this script.')
    logger.info('Please log issues here https://github.com/stevezau/plex_generate_vid_previews/issues')

    if platform.system() == 'Windows':
        logger.info('Running on Windows. Robocopy will be used for file transfers.')
    else:
        logger.info('Running on non-Windows system. Default file copy method will be used.')

    if not os.path.exists(PLEX_LOCAL_MEDIA_PATH):
        logger.error(
            '%s does not exist, please edit PLEX_LOCAL_MEDIA_PATH environment variable' % PLEX_LOCAL_MEDIA_PATH)
        exit(1)

    if not os.path.exists(os.path.join(PLEX_LOCAL_MEDIA_PATH, 'localhost')):
        logger.error(
            'You set PLEX_LOCAL_MEDIA_PATH to "%s". There should be a folder called "localhost" in that directory but it does not exist which suggests you haven\'t mapped it correctly. Please fix the PLEX_LOCAL_MEDIA_PATH environment variable' % PLEX_LOCAL_MEDIA_PATH)
        exit(1)

    if PLEX_URL == '':
        logger.error('Please set the PLEX_URL environment variable')
        exit(1)

    if PLEX_TOKEN == '':
        logger.error('Please set the PLEX_TOKEN environment variable')
        exit(1)

    # detect GPU's
    gpu = detect_gpu()
    if gpu == 'NVIDIA':
        logger.info('Found NVIDIA GPU')
    elif gpu:
        logger.info(f'Found AMD GPU {gpu}')
    if not gpu:
        logger.warning('No GPUs detected. Defaulting to CPU ONLY.')
        logger.warning('If you think this is an error please log an issue here https://github.com/stevezau/plex_generate_vid_previews/issues')

    try:
        # Clean TMP Folder
        if os.path.isdir(TMP_FOLDER):
            shutil.rmtree(TMP_FOLDER)
        os.makedirs(TMP_FOLDER)
        run(gpu)
    finally:
        if os.path.isdir(TMP_FOLDER):
            shutil.rmtree(TMP_FOLDER)
stevezau commented 3 days ago

Thanks for testing @Crunch41. So it appears the script is working correctly now.. and you shared an update that allows you to copy the file locally, which provides better performance.

So, just confirming this issue can be now marked as resolved?

Crunch41 commented 3 days ago

Yeah I'd confidently say this is resolved, Appreciate the help debugging this one.