Introduce EAE transcoding support for elastic transcoding jobs

I'm not the most proficient in Go, but here is a patch that begins to add support for setting the EAE_ROOT environment variable which Plex's FFMPEG utilizes when transcoding EAE, to override the default /tmp path. I patterned it off of the functionality you added for FFMPEG_EXTERNAL_LIBS and the codec serving paths.

I also don't have a great test setup for developing this locally yet, but I'll get that up at some point. I'm also not sure if this is something you've already looked into and tried @ressu but thought it was worth a shot and some discussion. It looks promising 😃

Also, once this is tested, working well, and ready to be merged in, we could probably update the code so it doesn't bypass kube-plex for EAE. Turns out a large portion of my media hits this, and defeats the purpose of running kube-plex.

This looks like a good start. My problem with overriding EAE_ROOT was that the mount needs to be shared, Plex seems to create some files and directories in the EAE_ROOT and the Transcode process expect to see those there.

So the only solution I could come up with is to create another shared mount that would be mounted if EAE is being used. Since my usage is mostly not making use of EAE, I didn't spend more time investigating and deleted my experimental code.

Another option I think we have is to reach out to Plex to see if they could fix the EAE behavior and make it use transcoding directory instead of temp, that would solve our problems with the least effort.

I decided to spend a bit of time digging into this one.. Seems like EAE_ROOT is based on TMPDIR, so we would indeed need to move the whole TMPDIR to a shared volume. (More discussion on Plex Forums)

And it gets even worse, in another discussion it was stated that the TMPDIR must be on a volume that allows binaries to be executed and must be covered by inotify on Linux (Thread). People seem to have tried to convince changes to these issues, but Plex doesn't seem to want to discuss internals and don't appear to be willing to look for alternatives either.

Wow, those are fascinating threads (unproductive by the Plex team!).

Before audio transcoding for EAE, the only contents in /tmp/pms-*/ for EAE are some directories that are referenced/utilized in the FFMPEG source, but created by the Plex process. During audio transcoding for EAE, Plex Transcoder/FFMPEG populates the directories with the transcoded files.

Folders created by Plex and utilized by FFMPEG through EAE_ROOT:

root@plex-kube-plex:/tmp/pms-8eeb73b4-1680-45a0-b0bf-6be524254b13/EasyAudioEncoder# du -sh *                                                                                                                                                                                      
4.0K    Convert to Dolby Digital (High Quality - 640 kbps)                                                                                                                                                                                                                        
4.0K    Convert to Dolby Digital (Low Quality - 384 kbps)                                                                                                                                                                                                                         
4.0K    Convert to Dolby Digital Plus (High Quality - 384 kbps)                                                                                                                                                                                                                   
4.0K    Convert to Dolby Digital Plus (Max Quality - 1024 kbps)                                                                                                                                                                                                                   
4.0K    Convert to WAV (to 2ch or less)                                                                                                                                                                                                                                           
1.8M    Convert to WAV (to 8ch or less)

Plex log for the default, as you can see, just passing EAE_ROOT.

[Transcode/guqbxp4k265pavl4ho51nrvj/JobRunner] Job running: EAE_ROOT='/tmp/pms-8eeb73b4-1680-45a0-b0bf-6be524254b13/EasyAudioEncoder' FFMPEG_EXTERNAL_LIBS='/config/Library/Application\ Support/Plex\ Media\ Server/Codecs/fa235d6-3858-linux-x86_64/' X_PLEX_TOKEN='xxxxxxxxxxxxxxxxxxxx' '/usr/lib/plexmediaserver/Plex Transcoder' '-codec:0' 'h264' '-codec:1' 'truehd_eae' '-eae_prefix:1' 'guqbxp4k265pavl4ho51nrvj_' '-ss' '1840' '-noaccurate_seek' '-analyzeduration' '20000000' '-probesize' '20000000' '-i' '/data/Movies/Movie.mkv' '-map' '0:0' '-metadata:s:0' 'language=eng' '-codec:0' 'copy' '-filter_complex' '[0:1] aresample=async=1:ocl='\''stereo'\'':rematrix_maxval=0.000000dB:osr=48000[0]' '-map' '[0]' '-metadata:s:1' 'language=eng' '-codec:1' 'aac' '-b:1' '256k' '-f' 'dash' '-seg_duration' '5' '-dash_segment_type' 'mp4' '-init_seg_name' 'init-stream$RepresentationID$.m4s' '-media_seg_name' 'chunk-stream$RepresentationID$-$Number%05d$.m4s' '-window_size' '5' '-delete_removed' 'false' '-skip_to_segment' '369' '-time_delta' '0.0625' '-manifest_name' 'http://127.0.0.1:32400/video/:/transcode/session/guqbxp4k265pavl4ho51nrvj/d568d7a7-6d6f-43a9-96d2-47417f5c2c7b/manifest?X-Plex-Http-Pipeline=infinite' '-avoid_negative_ts' 'disabled' '-map_metadata' '-1' '-map_chapters' '-1' 'dash' '-start_at_zero' '-copyts' '-y' '-nostats' '-loglevel' 'quiet' '-loglevel_plex' 'error' '-progressurl' 'http://127.0.0.1:32400/video/:/transcode/session/guqbxp4k265pavl4ho51nrvj/d568d7a7-6d6f-43a9-96d2-47417f5c2c7b/progress'

So, Plex only seems to create the directories, which then FFMPEG populates them with the transcode files, so it might work if the directories FFMPEG needs are simply present, and possibly not requiring a shared volume

It's unfortunate Plex is not more flexible here. I think I'm leaning toward your suggestion of another shared mount if EAE support is enabled. Would it just be a shared mount for /tmp/ though? I think the discussion around it needing to be mounted with executable permissions is wrong. Most tmpfs's do not allow it for security reasons -- it would just need to read and write based off stracing I've done recently.

There is another possible option:

Using something like LD_PRELOAD to redirect the calls being made to /tmp/pms-* to the EAE_ROOT override directory (this one's a bit sketchy, and the shared volume is a better workaround).

Thanks for the discussion @ressu! I'm learning tons

Before audio transcoding for EAE, the only contents in /tmp/pms-*/ for EAE are some directories that are referenced/utilized in the FFMPEG source, but created by the Plex process. During audio transcoding for EAE, Plex Transcoder/FFMPEG populates the directories with the transcoded files.

If we only need the directories to be present, we can always create them when the transcode launcher starts. There is already quite a few things happening on transcoder start so it's not that big of an issue to add a quick loop to create the required directories. And assuming that this is indeed the case, then we can create those directories where ever we choose and we can simply set our own EAE_ROOT for the transcoder process.

https://github.com/ressu/kube-plex/blob/eb3ba1eaab47e90c52b9908e6c6b29678b40558e/cmd/transcode-launcher/main.go#L43-L54

Would it just be a shared mount for /tmp/ though? I think the discussion around it needing to be mounted with executable permissions is wrong. Most tmpfs's do not allow it for security reasons -- it would just need to read and write based off stracing I've done recently.

I think the shared mount for /tmp would be enough, I do worry slightly about the overhead though. There seems to be a lot of activity in that directory, especially when some of the detections are running (as mentioned in the threads) so it might cause a significant performance hit. We should first investigate the option to just create directories and pretending that the issue doesn't even exist 😄

I agree that the exec requirement seems a bit odd. I think they mentioned that they would be changing things on the exec front, so it could be that this is already old information. But the way I understand it, is that some of the codecs etc required executing platform specific binaries etc. So maybe this issue already went away by moving the codecs to /config, but I have to admit that I have very little information on how the older versions of Plex actually worked internally.

Using something like LD_PRELOAD to redirect the calls being made to /tmp/pms-* to the EAE_ROOT override directory (this one's a bit sketchy, and the shared volume is a better workaround).

I did consider this too, but I'm just hoping we don't have to come to that. LD_PRELOAD is generally a very fragile way of doing things. Not to mention that I'm not too comfortable with low level C coding, so it would be a horrible piece of hack (at least if it's written by me 🤣 )

I was taking a look at this and wasn't able to quickly find any files to test EAE with, but I did spot an issue with the PR. This https://github.com/ressu/kube-plex/commit/83dc4cb52b32bcf34b24ecbd57b6440831fa6e90 will allow easier testing of the EAE feature. Effectively it will bypass local transcoding if eae root has been set. If this works, we can drop the whole bypassing..

My working branch is https://github.com/ressu/kube-plex/tree/pr/brandon099/5

Yes, that helped greatly with testing, thank you! The functionality if EAE root dir is not specified, it skipped launching transcoder pods and worked great.

Here are some logs for the transcoder with everything wired up and trying to use EAE on a elastic transcoder pod with /transcode being set as the EAE_ROOT.

Here's some logs, and based off them here's what it looks like is happening (which is unsurprising, but also unfortunate).

1) The codecManager starts EAE at /tmp 2) JobRunner starts using EAE_ROOT at /tmp 3) KubePlex transcoder pod starts and updates EAE_ROOT to what I passed (/transcode for this test) 4) KubePlex creates the needed directories for EAE under /transcode 5) Confirmed files are getting written to /transcode by KubePlex pod which seems good 6) An error being thrown about EAE timeout

Jul 25, 2021 13:09:21.347 [0x7f0811ba2b38] Info — [Transcode] CodecManager: starting EAE at "/tmp/pms-e4d4bc77-ff75-46c8-81cb-86b8a2f7adc6/EasyAudioEncoder"
Jul 25, 2021 13:09:21.348 [0x7f0811ba2b38] Debug — [Transcode/JobRunner] Job running: '/config/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-1445-linux-x86_64/EasyAudioEncoder/EasyAudioEncoder'
Jul 25, 2021 13:09:21.351 [0x7f0811ba2b38] Debug — [Transcode/JobRunner] Jobs: Starting child process with pid 474
Jul 25, 2021 13:09:21.352 [0x7f0811ba2b38] Debug — [Transcode] [Universal] Using local file path instead of URL: /data/Videos/DOLBY_TRUEHD_SOUND_CHECK_71.mkv
Jul 25, 2021 13:09:21.396 [0x7f0811ba2b38] Debug — [Transcode/JobRunner] Job running: EAE_ROOT='/tmp/pms-e4d4bc77-ff75-46c8-81cb-86b8a2f7adc6/EasyAudioEncoder' FFMPEG_EXTERNAL_LIBS='/config/Library/Application\ Support/Plex\ Media\ Server/Codecs/fa235d6-3858-linux-x86_64/' X_PLEX_TOKEN='xxxxxxxxxxxxxxxxxxxx' '/usr/lib/plexmediaserver/Plex Transcoder' '-codec:0' 'h264' '-codec:1' 'truehd_eae' '-eae_prefix:1' 'kxhszaist38dg2y4thwqyaoh_' '-noaccurate_seek' '-analyzeduration' '20000000' '-probesize' '20000000' '-i' '/data/Videos/DOLBY_TRUEHD_SOUND_CHECK_71.mkv' '-map' '0:0' '-metadata:s:0' 'language=eng' '-codec:0' 'copy' '-filter_complex' '[0:1] aresample=async=1:ocl='\''stereo'\'':rematrix_maxval=0.000000dB:osr=48000[0]' '-map' '[0]' '-metadata:s:1' 'language=eng' '-codec:1' 'aac' '-b:1' '256k' '-f' 'dash' '-seg_duration' '5' '-dash_segment_type' 'mp4' '-init_seg_name' 'init-stream$RepresentationID$.m4s' '-media_seg_name' 'chunk-stream$RepresentationID$-$Number%05d$.m4s' '-window_size' '5' '-delete_removed' 'false' '-skip_to_segment' '1' '-time_delta' '0.0625' '-manifest_name' 'http://127.0.0.1:32400/video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/manifest?X-Plex-Http-Pipeline=infinite' '-avoid_negative_ts' 'disabled' '-map_metadata' '-1' '-map_chapters' '-1' 'dash' '-start_at_zero' '-copyts' '-vsync' 'cfr' '-y' '-nostats' '-loglevel' 'quiet' '-loglevel_plex' 'error' '-progressurl' 'http://127.0.0.1:32400/video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress'
Jul 25, 2021 13:09:21.397 [0x7f0811ba2b38] Debug — [Transcode/JobRunner] Jobs: Starting child process with pid 475
Jul 25, 2021 13:09:21.457 [0x7f0811bc5b38] Info — [KubePlex] Starting transcode job

Jul 25, 2021 13:09:21.484 [0x7f08124c8b38] Info — [KubePlex] Transcoder launched as job/pms-elastic-transcoder-rnkcn (namespace: default)

Jul 25, 2021 13:09:26.306 [0x7f0811bc5b38] Info — [KubePlexProxy] Transcode launcher starting...

Jul 25, 2021 13:09:26.338 [0x7f08124c8b38] Info — [KubePlexProxy] Updating environment, setting EAE_ROOT to '/transcode'

Jul 25, 2021 13:09:26.339 [0x7f0811bc5b38] Info — [KubePlexProxy] Creating EAE required directory: /transcode/Convert to Dolby Digital (High Quality - 640 kbps)

Jul 25, 2021 13:09:26.341 [0x7f08124c8b38] Info — [KubePlexProxy] Creating EAE required directory: /transcode/Convert to Dolby Digital (Low Quality - 384 kbps)

Jul 25, 2021 13:09:26.342 [0x7f0811bc5b38] Info — [KubePlexProxy] Creating EAE required directory: /transcode/Convert to Dolby Digital Plus (High Quality - 384 kbps)

Jul 25, 2021 13:09:26.344 [0x7f08124c8b38] Info — [KubePlexProxy] Creating EAE required directory: /transcode/Convert to Dolby Digital Plus (Max Quality - 1024 kbps)

Jul 25, 2021 13:09:26.345 [0x7f0811bc5b38] Info — [KubePlexProxy] Creating EAE required directory: /transcode/Convert to WAV (to 2ch or less)

Jul 25, 2021 13:09:26.347 [0x7f08124c8b38] Info — [KubePlexProxy] Creating EAE required directory: /transcode/Convert to WAV (to 8ch or less)

Jul 25, 2021 13:09:26.349 [0x7f08124c8b38] Info — [KubePlexProxy] Setting debug level to verbose on transcode process

Jul 25, 2021 13:09:26.350 [0x7f0811bc5b38] Info — [KubePlexProxy] Transcode requested with command /usr/lib/plexmediaserver/Plex Transcoder, args = [-loglevel verbose -loglevel_plex verbose -codec:0 h264 -codec:1 truehd_eae -eae_prefix:1 kxhszaist38dg2y4thwqyaoh_ -noaccurate_seek -analyzeduration 20000000 -probesize 20000000 -i /data/Videos/DOLBY_TRUEHD_SOUND_CHECK_71-thedigitaltheater.mkv -map 0:0 -metadata:s:0 language=eng -codec:0 copy -filter_complex [0:1] aresample=async=1:ocl='stereo':rematrix_maxval=0.000000dB:osr=48000[0] -map [0] -metadata:s:1 language=eng -codec:1 aac -b:1 256k -f dash -seg_duration 5 -dash_segment_type mp4 -init_seg_name init-stream$RepresentationID$.m4s -media_seg_name chunk-stream$RepresentationID$-$Number%05d$.m4s -window_size 5 -delete_removed false -skip_to_segment 1 -time_delta 0.0625 -manifest_name http://127.0.0.1:32400/video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/manifest?X-Plex-Http-Pipeline=infinite -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 dash -start_at_zero -copyts -vsync cfr -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress]

Jul 25, 2021 13:09:26.350 [0x7f08124c8b38] Info — [KubePlexProxy] Transcode begins...

Jul 25, 2021 13:09:26.392 [0x7f0811bc5b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=startup (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.392 [0x7f0812b6eb38] Debug — Completed: [10.42.1.179:52482] 204 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=startup (16 live) 0ms 203 bytes (pipelined: 1) (range: bytes=0-) 
Jul 25, 2021 13:09:26.393 [0x7f08124c8b38] Info — [Transcoder] ffmpeg version fa235d6-3858 Copyright (c) 2000-2019 the FFmpeg developers
Jul 25, 2021 13:09:26.394 [0x7f0811bc5b38] Info — [Transcoder]   built with Plex clang version 11.0.1 (https://plex.tv e0c29d5827bc4eaaa2ceb882cbeed224b0960173)
Jul 25, 2021 13:09:26.395 [0x7f08124c8b38] Info — [Transcoder]   configuration: --disable-static --enable-shared --disable-libx264 --disable-hwaccels --disable-protocol=concat --external-decoder=h264 --enable-debug --enable-muxers --fatal-warnings --disable-gmp --disable-avdevice --disable-bzlib --disable-sdl2 --disable-decoders --disable-devices --disable-encoders --disable-ffprobe --disable-ffplay --disable-doc --disable-iconv --disable-lzma --disable-schannel --disable-linux-perf --disable-mediacodec --enable-eae --disable-protocol='udp,udplite' --arch=x86_64 --target-os=linux --strip=true --cc=x86_64-linux-musl-clang --pkg-config=/data/jenkins/conan_build/871697973/plexconantool/plex-pkg-config --pkg-config-flags=--static --enable-cuda-llvm --enable-libdrm --enable-opencl --enable-cross-compile --ar=llvm-ar --nm=llvm-nm --ranlib=llvm-ranlib --extra-ldflags='-Wl,-rpath,/data/jenkins/conan_build/871697973/conan/.conan/data/libpciaccess/0.16-5/plex/stable/package/7763a87432c78a82fd36373080b064286892cea3/lib -Wl,-rpath,/data/jenkins/conan_build/871697973/conan/.conan/d
Jul 25, 2021 13:09:26.398 [0x7f0811bc5b38] Info — [Transcoder]   libavutil      56. 26.100 / 56. 26.100
Jul 25, 2021 13:09:26.398 [0x7f08124c8b38] Info — [Transcoder]   libavcodec     58. 52.100 / 58. 52.100
Jul 25, 2021 13:09:26.399 [0x7f0811bc5b38] Info — [Transcoder]   libavformat    58. 27.104 / 58. 27.104
Jul 25, 2021 13:09:26.399 [0x7f08124c8b38] Info — [Transcoder]   libavfilter     7. 49.100 /  7. 49.100
Jul 25, 2021 13:09:26.400 [0x7f0811bc5b38] Info — [Transcoder]   libswscale      5.  4.100 /  5.  4.100
Jul 25, 2021 13:09:26.401 [0x7f08124c8b38] Info — [Transcoder]   libswresample   3.  4.100 /  3.  4.100
Jul 25, 2021 13:09:26.403 [0x7f0811bc5b38] Debug — [Transcoder] Rescanning for external libs: '/shared/codecs/'
Jul 25, 2021 13:09:26.404 [0x7f08124c8b38] Debug — [Transcoder] Loading external lib /shared/codecs/libh264_decoder.so
Jul 25, 2021 13:09:26.405 [0x7f0811bc5b38] Debug — [Transcoder] Loading external lib /shared/codecs/libaac_encoder.so
Jul 25, 2021 13:09:26.410 [0x7f08124c8b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=startup (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.410 [0x7f0812b4bb38] Debug — Completed: [10.42.1.179:52482] 204 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=startup (16 live) 0ms 203 bytes (pipelined: 14) (range: bytes=0-) 
Jul 25, 2021 13:09:26.411 [0x7f0811bc5b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=opening (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.411 [0x7f0812b4bb38] Debug — Completed: [10.42.1.179:52482] 204 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=opening (16 live) 0ms 203 bytes (pipelined: 15) (range: bytes=0-) 
Jul 25, 2021 13:09:26.423 [0x7f08124c8b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=opened (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.423 [0x7f0812b6eb38] Debug — Completed: [10.42.1.179:52482] 204 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?status=opened (16 live) 0ms 203 bytes (pipelined: 16) (range: bytes=0-) 
Jul 25, 2021 13:09:26.424 [0x7f0811bc5b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/stream?index=0&id=0&codec=h264&type=video (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.425 [0x7f0812b4bb38] Debug — Completed: [10.42.1.179:52482] 200 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/stream?index=0&id=0&codec=h264&type=video (16 live) 0ms 195 bytes (pipelined: 17) (range: bytes=0-) 
Jul 25, 2021 13:09:26.425 [0x7f08124c8b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/stream?index=1&id=0&codec=truehd&type=audio (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.426 [0x7f0812b4bb38] Debug — Completed: [10.42.1.179:52482] 200 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/stream?index=1&id=0&codec=truehd&type=audio (16 live) 0ms 195 bytes (pipelined: 18) (range: bytes=0-) 
Jul 25, 2021 13:09:26.490 [0x7f0811bc5b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/streamDetail?index=0&id=0&codec=h264&type=video&profile=High&language=eng&width=1920&height=1080&interlaced=0&level=41&frameRate=29.970&disp_default=1 (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.491 [0x7f0812b4bb38] Debug — Completed: [10.42.1.179:52482] 200 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/streamDetail?index=0&id=0&codec=h264&type=video&profile=High&language=eng&width=1920&height=1080&interlaced=0&level=41&frameRate=29.970&disp_default=1 (16 live) 0ms 195 bytes (pipelined: 19) (range: bytes=0-) 
Jul 25, 2021 13:09:26.491 [0x7f08124c8b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/streamDetail?index=1&id=0&codec=truehd&type=audio&language=eng&channels=8&layout=7.1&sampleRate=48000&bitDepth=24&disp_default=1 (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.492 [0x7f0812b6eb38] Debug — Completed: [10.42.1.179:52482] 200 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress/streamDetail?index=1&id=0&codec=truehd&type=audio&language=eng&channels=8&layout=7.1&sampleRate=48000&bitDepth=24&disp_default=1 (16 live) 0ms 195 bytes (pipelined: 20) (range: bytes=0-) 
Jul 25, 2021 13:09:26.495 [0x7f0811bc5b38] Debug — Request: [10.42.1.179:52482 (WAN)] PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?duration=94.493000 (16 live) Signed-in Token (brandon099) (range: bytes=0-) 
Jul 25, 2021 13:09:26.495 [0x7f0811ba2b38] Debug — [Transcode] Started session successfully: kxhszaist38dg2y4thwqyaoh
Jul 25, 2021 13:09:26.495 [0x7f0812b6eb38] Debug — Completed: [10.42.1.179:52482] 204 PUT /video/:/transcode/session/kxhszaist38dg2y4thwqyaoh/9c05327e-59f2-4392-954e-2f6b7394d0a3/progress?duration=94.493000 (16 live) 0ms 203 bytes (pipelined: 21) (range: bytes=0-) 
Jul 25, 2021 13:09:26.495 [0x7f0812b6eb38] Debug — Completed: [10.0.30.175:60788] 200 GET /video/:/transcode/universal/start.mpd?hasMDE=1&path=%2Flibrary%2Fmetadata%2F1&mediaIndex=0&partIndex=0&protocol=dash&fastSeek=1&directPlay=0&directStream=1&subtitleSize=100&audioBoost=100&location=lan&addDebugOverlay=0&autoAdjustQuality=0&directStreamAudio=1&mediaBufferSize=102400&session=kxhszaist38dg2y4thwqyaoh&subtitles=burn&Accept-Language=en (16 live) GZIP 5223ms 953 bytes (pipelined: 8)
Jul 25, 2021 13:09:28.129 [0x7f0811ba2b38] Debug — Request: [10.0.30.175:60788 (WAN)] GET /video/:/transcode/universal/session/kxhszaist38dg2y4thwqyaoh/1/header (16 live) GZIP Signed-in
Jul 25, 2021 13:09:28.131 [0x7f08124c8b38] Debug — Request: [10.0.30.175:60790 (WAN)] GET /video/:/transcode/universal/session/kxhszaist38dg2y4thwqyaoh/1/0.m4s (16 live) GZIP Signed-in
Jul 25, 2021 13:09:28.131 [0x7f08124c8b38] Debug — [Transcode/kxhszaist38dg2y4thwqyaoh] Asked for segment 0 from session.
Jul 25, 2021 13:09:28.898 [0x7f0811bc5b38] Debug — Request: [10.0.30.175:60798 (WAN)] GET /video/:/transcode/universal/session/kxhszaist38dg2y4thwqyaoh/0/0.m4s (16 live) GZIP Signed-in
Jul 25, 2021 13:09:28.898 [0x7f081250eb38] Debug — Request: [10.0.30.175:60796 (WAN)] GET /video/:/transcode/universal/session/kxhszaist38dg2y4thwqyaoh/0/header (16 live) GZIP Signed-in
Jul 25, 2021 13:09:28.899 [0x7f0811bc5b38] Debug — [Transcode/kxhszaist38dg2y4thwqyaoh] Asked for segment 0 from session.
Jul 25, 2021 13:09:29.001 [0x7f0812122b38] Error — [Transcoder] [truehd_eae @ 0x7ff5f692d580] EAE timeout! EAE not running, or wrong folder? Could not read '/transcode/Convert to WAV (to 8ch or less)/kxhszaist38dg2y4thwqyaoh_13-0-0.wav'
Jul 25, 2021 13:09:29.003 [0x7f0812122b38] Error — [Transcoder] [truehd_eae @ 0x7ff5f692d580] error reading output
Jul 25, 2021 13:09:29.004 [0x7f0812122b38] Error — [Transcoder] Error while decoding stream #0:1: I/O error
Jul 25, 2021 13:09:32.000 [0x7f0812122b38] Error — [Transcoder] [truehd_eae @ 0x7ff5f692d580] EAE timeout! EAE not running, or wrong folder? Could not read '/transcode/Convert to WAV (to 8ch or less)/kxhszaist38dg2y4thwqyaoh_13-0-1.wav'

That sounds pretty similar to what I used to see.. I assumed that it was due to the fact that the Plex process is trying to write to the EAE root and things are timing out because the files aren't visible on both sides.

We could try to mount a shared volume in the EAE root location in tmp.. but that might get complex rather quickly

Yeah looks like this is what leads up to the failure (according to this post):

1. Transcoder tells EAE where to write the file
2. EAE opens and starts writing
3. Transcoder, expecting to see the open() , doesn’t get one.
4. This is where the logic breaks down, and thinks the EAE has died.

To get this working, we'll probably need to get really creative. Either that, or just accept it would require Plex to make an internal change to support a custom EAE_ROOT, which is very unlikely.

I did some more looking at this, including what other similar projects are doing, and there are only two implementations I am seeing.

First being the bypass if EAE is needed, so the primary takes the request like KubePlex does today (e.g. Plex-Remote-Transcoder).

The other, which is far more complex and essentially still relying on a local EAE service is what Unicorn Transcoder does. There's a load balancer in-front of requests to Plex, and from what I've gathered essentially it manages it's own EAE instance (It downloads codecs and EAE from plex.tv, extracts them, so they are locally available).

So if we downloaded and ran the EAE in the transcoder pod as well (still relying on the EAE_ROOT environment variable (serving it up to the pod similar to how codecs are shared/served up), to convert the .mlp files to their desired output? When testing I am seeing the .mlp files show up in the EAE_ROOT I specified, but because EAE on the primary isn't able to look in the correct place. That might get us closer to being able to share those files with the primary pod.

The other thought I had was having a shim that replaces the "Plex EAE Service" binary (/config/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-1445-linux-x86_64/EasyAudioEncoder/EasyAudioEncoder), exactly the same way we do for the Transcoder for possibly more control, that spawns an EAE server with our overrides.

All of these might be adding more complexity than is worth, but I'll keep poking at it and your thoughts and insight are great 😀, so thank you for entertaining these ideas with your time!

There's a load balancer in-front of requests to Plex, and from what I've gathered essentially it manages it's own EAE instance (It downloads codecs and EAE from plex.tv, extracts them, so they are locally available).

This is similar to what we are already doing. Instead of downloading the binaries from plex, we copy the existing blobs from the plex container to the transcoder container. Since Plex already downloads the files, they can be quickly copied over (within a few milliseconds) so there is no concerns about startup delay or anything like that.

The other thought I had was having a shim that replaces the "Plex EAE Service" binary (/config/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-1445-linux-x86_64/EasyAudioEncoder/EasyAudioEncoder), exactly the same way we do for the Transcoder for possibly more control, that spawns an EAE server with our overrides.

This seems like a good idea. I never even considered that there is a separate Plex EAE Service that is involved with EAE. It would be rather trivial to move that to the transcoder pod. Considering that we already should have everything else in place (the binary blobs, backchannel to Plex, files and mounts etc..)

I'm in general not too concerned about adding more complexity. This considering that I've been thinking about solutions to preload the transcoder pods. Currently there is a few seconds (I think I average at around 5 seconds) of delay when scheduling the pods. So preloading the pods would reduce that delay... That by itself will increase complexity, so as long as complexity is justified, I think it's fine.

I spent some time looking at this again. It seems that when I initially created the launcher, I was smart and used the original command line as the base. So replacing an additional binary would be as simple as just copying the shim over the required binary and things would "just work".

On the other hand, it doesn't seem like there is a separate binary for the Plex EAE Service, so it's not possible to just move it to the container. I think I need to take a closer look at how Unicorn approaches the problem. But at least for now, I didn't make any meaningful progress.

ressu / kube-plex

Introduce EAE transcoding support for elastic transcoding jobs #5