Closed treverehrfurth closed 9 months ago
One issue I'm seeing is that your PMS_IP
is set to localhost, which won't work. Try setting that to that node's actual IP.
Because the way this works is, workers will connect to that PMS_IP on the Relay port to report transcoding progress, and from that relay port traffic is then locally forwarded to the real Plex port. That's to have plex believe the progress report is coming from itself and not a remote location (which makes it reject it).
Ahh, I made that change and its looking like the nodes are communicating with the orchestrator/plex and finally passing along transcodes. Thanks again! This is an awesome job you've done here, allowing me to use my various home lab servers to handle a worst case scenario of 40 streams (in my use case) without having to shell out for a super expensive moba(s) and gpu's! Appreciate the help.
Do you suggest 1 or 2 worker replicas per node and why? And do I need a worker job on the main node? Or will the plex service handle those transcodes itself? Because I think I only see it registering 2 workers, or maybe it just doesn't log the worker running on the server of pms and orchestrator?
As always with technology, it depends lol. No, but seriously, in a default scenario of just doing cpu transcoding, one worker per node would be enough. Since you're sharing the same physical cpu resource there isn't much benefit to running multiple. Now, if you manage to get hardware transcoding working and you have let's say an igpu plus an Nvidia gpu, you could probably run one worker for each dedicated hardware and make use of them all. But I haven't actually tried that setup configuration. Regarding the running of a worker along with pms, it's not required but if you want that physical machine to also transcode you could have it. Now, you should be seeing it registering in the orchestrator though. If you're not then something is not properly configured on that worker.
Meaning, the way it currently works (affected by the operating_mode) is that it tries to first do a remote transcode and if that fails or there are no workers available it falls back to a local transcode on pms.
That would be really cool to use both igpu and gpu! I played around for a day trying to get Proxmox > Ubuntu > Docker > Plex/Container to get GPU access and man... was that a struggle of outdated tutorials. I went down that route thinking i'd never get this clusterplex configured for my scenario right. I had lots of service restarts when I was running plex on a NFS share from Truenas Scale. The second I ran plex config on the local VM storage.... it was golden. No crashes or reboots anymore. I really wanted both Proxmox HA and Docker Swarm HA but it was such a headache. And adding GPU and PCIE passthrough into the mix was also a nightmare.
I got GPU in docker and Plex to work really easy in a baremetal instance of ubuntu but missed the oversight in proxmox. But now that I have clusterplex running smoothly, there's no need for those additional headaches of gpu passthrough.
Per your explanation, I have reduced my replicas down to 1 on all nodes including the main. I do see in orchestrator it registering all the workers and handing off transcodes to them!
And again, I can't thank you enough for this, an awesome project well done. I do video work on the side and once I get some time, I definitely want to make a tutorial vid on how to get this setup properly so more people can make use of this repo!
These are my logs as of recent from my workers, when transcoding. There isn't many people on so transcoding so not sure if this is just because of something else or if something is wrong but lots of refused connection messages:
Worker Node 1:
[tcp @ 0x7f48538dcec0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48538dcec0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48538dcec0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[dash @ 0x7f484fe1b440] Opening 'chunk-stream1-00033.m4s.tmp' for writing
[tcp @ 0x7f48537ea640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537ea640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537ea640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[dash @ 0x7f484fe1b440] Opening 'chunk-stream0-00033.m4s.tmp' for writing
[tcp @ 0x7f485388fe80] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485388fe80] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485388fe80] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[AVIOContext @ 0x7f48538db200] Statistics: 9743 bytes written, 0 seeks, 1 writeouts
[tcp @ 0x7f48537d3440] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537d3440] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537d3440] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[dash @ 0x7f484fe1b440] Representation 0 media segment 34 written to: chunk-stream0-00033.m4s
[tcp @ 0x7f48537ea640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537ea640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537ea640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[AVIOContext @ 0x7f484cd51380] Statistics: 571 bytes written, 0 seeks, 1 writeouts
[tcp @ 0x7f485206b240] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b240] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485206b240] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[dash @ 0x7f484fe1b440] Representation 1 media segment 34 written to: chunk-stream1-00033.m4s
[tcp @ 0x7f485388fe80] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485388fe80] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485388fe80] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[dash @ 0x7f484fe1b440] Opening 'http://192.168.1.147:32499/video/:/transcode/session/ryicsm9212kgtnz535bc1o61/b8960e81-822a-46a2-9329-6fdcfa85ff65/manifest?X-Plex-Http-Pipeline=infinite' for writing
[tcp @ 0x7f4852085680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f4852085680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f4852085680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[tcp @ 0x7f48537d3440] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b240] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b240] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485206b240] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[tcp @ 0x7f48537d3440] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537ea640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537ea640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537ea640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[tcp @ 0x7f48537d3440] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[tcp @ 0x7f485206b280] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b280] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485206b280] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[dash @ 0x7f484fe1b440] Unable to open http://192.168.1.147:32499/video/:/transcode/session/ryicsm9212kgtnz535bc1o61/b8960e81-822a-46a2-9329-6fdcfa85ff65/manifest?X-Plex-Http-Pipeline=infinite for writing: Connection refused
[tcp @ 0x7f48537ea680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537ea680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537ea680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[tcp @ 0x7f4852085640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f4852085640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f4852085640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
frame= 239 fps=0.0 q=-1.0 Lsize=N/A time=00:00:09.86 bitrate=N/A speed=17.2x
[tcp @ 0x7f48537d9f00] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537d9f00] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537d9f00] Connection to tcp://192.168.1.147:32499 failed: Connection refused
video:1045kB audio:160kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[tcp @ 0x7f485206b240] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b240] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485206b240] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Input file #0 (/tv/It's Always Sunny in Philadelphia/Season 08/It's Always Sunny in Philadelphia - S08E08 - Charlie Rules the World.mkv):
[tcp @ 0x7f4852085680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f4852085680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f4852085680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Input stream #0:0 (video): 239 packets read (1070396 bytes);
[tcp @ 0x7f48537d9ec0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537d9ec0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537d9ec0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Input stream #0:1 (audio): 466 packets read (163452 bytes);
[tcp @ 0x7f485206b280] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b280] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485206b280] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Total: 705 packets (1233848 bytes) demuxed
[tcp @ 0x7f48531a1840] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48531a1840] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48531a1840] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Output file #0 (dash):
[tcp @ 0x7f4852085640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f4852085640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f4852085640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Output stream #0:0 (video): 239 packets muxed (1070396 bytes);
[tcp @ 0x7f4852448640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f4852448640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f4852448640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Output stream #0:1 (audio): 466 packets muxed (163452 bytes);
[tcp @ 0x7f485206b240] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485206b240] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485206b240] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Total: 705 packets (1233848 bytes) muxed
[tcp @ 0x7f485388fe80] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f485388fe80] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f485388fe80] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[AVIOContext @ 0x7f48538db040] Statistics: 1300736 bytes read, 3 seeks
[tcp @ 0x7f48537d3440] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537d3440] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537d3440] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Conversion failed!
[tcp @ 0x7f48537ea640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7f48537ea640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7f48537ea640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Completed transcode
Removing process from taskMap
Worker Node 2:
[tcp @ 0x7feae1de41c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de41c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Input stream #0:1 (audio): 467 packets read (159494 bytes); 467 frames decoded (478208 samples);
[tcp @ 0x7feae1de45c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de45c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de45c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Total: 707 packets (1238149 bytes) demuxed
[tcp @ 0x7feae1de4680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Output file #0 (dash):
[tcp @ 0x7feae1de45c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de45c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de45c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Output stream #0:0 (video): 213 frames encoded; 193 packets muxed (153627 bytes);
[tcp @ 0x7feae1de4680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Output stream #0:1 (audio): 431 frames encoded (441344 samples); 432 packets muxed (123474 bytes);
[tcp @ 0x7feae1de41c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de41c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de41c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Total: 625 packets (277101 bytes) muxed
[tcp @ 0x7feae1de45c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de45c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de45c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] frame I:4 Avg QP:23.05 size: 10282
[tcp @ 0x7feae1de4680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] frame P:56 Avg QP:26.62 size: 1444
[tcp @ 0x7feae1de4740] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4740] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4740] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] frame B:153 Avg QP:31.10 size: 240
[tcp @ 0x7feae1de41c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de41c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de41c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] consecutive B-frames: 2.8% 1.9% 7.0% 88.3%
[tcp @ 0x7feae1de45c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de45c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de45c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] mb I I16..4: 2.9% 9.7% 87.4%
[tcp @ 0x7feae1de4680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] mb P I16..4: 0.5% 2.8% 3.9% P16..4: 37.5% 17.8% 10.1% 0.0% 0.0% skip:27.5%
[tcp @ 0x7feae1de4740] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4740] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4740] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] mb B I16..4: 0.1% 0.6% 0.3% B16..8: 20.8% 6.1% 0.5% direct: 2.0% skip:69.7% L0:36.1% L1:43.1% BI:20.8%
[tcp @ 0x7feae1de41c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de41c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de41c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] 8x8 transform intra:30.1% inter:26.6%
[tcp @ 0x7feae1de45c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de45c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de45c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] coded y,uvDC,uvAC intra: 84.5% 86.8% 54.0% inter: 7.5% 3.7% 0.6%
[tcp @ 0x7feae1de4680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] i16 v,h,dc,p: 34% 44% 7% 14%
[tcp @ 0x7feae1de4740] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4740] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4740] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 18% 12% 6% 4% 5% 5% 8% 9%
[tcp @ 0x7feae1de41c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de41c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de41c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 16% 10% 6% 7% 8% 8% 8% 8%
[tcp @ 0x7feae1de45c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de45c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de45c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] i8c dc,h,v,p: 42% 22% 26% 11%
[tcp @ 0x7feae1de4680] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4680] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4680] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] Weighted P-Frames: Y:0.0% UV:0.0%
[tcp @ 0x7feae1de4740] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de4740] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de4740] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[libx264 @ 0x7feae5341080] kb/s:142.93
[tcp @ 0x7feae1de41c0] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae1de41c0] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae1de41c0] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[aac @ 0x7feae5341b00] Qavg: 678.630
[tcp @ 0x7feae9099640] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae9099640] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae9099640] Connection to tcp://192.168.1.147:32499 failed: Connection refused
[AVIOContext @ 0x7feae8cbc580] Statistics: 1299411 bytes read, 3 seeks
[tcp @ 0x7feae9443e40] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae9443e40] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae9443e40] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Conversion failed!
[tcp @ 0x7feae915af40] Starting connection attempt to 192.168.1.147 port 32499
[tcp @ 0x7feae915af40] Connection attempt to 192.168.1.147 port 32499 failed: Connection refused
[tcp @ 0x7feae915af40] Connection to tcp://192.168.1.147:32499 failed: Connection refused
Completed transcode
Removing process from taskMap
Main server:
2023-09-14 12:21:33 ERROR 404: Not Found.
Codec libxan_wc3_decoder.so does not exist. Downloading...
--2023-09-14 12:21:33-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxan_wc3_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 104.18.41.153, 172.64.146.103, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|104.18.41.153|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:33 ERROR 404: Not Found.
Codec libxan_wc4_decoder.so does not exist. Downloading...
--2023-09-14 12:21:33-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxan_wc4_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:33 ERROR 404: Not Found.
Codec libxbin_decoder.so does not exist. Downloading...
--2023-09-14 12:21:33-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxbin_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 104.18.41.153, 172.64.146.103, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|104.18.41.153|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:33 ERROR 404: Not Found.
Codec libxbm_decoder.so does not exist. Downloading...
--2023-09-14 12:21:33-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxbm_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::ac40:9267, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:34 ERROR 404: Not Found.
Codec libxface_decoder.so does not exist. Downloading...
--2023-09-14 12:21:34-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxface_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 104.18.41.153, 172.64.146.103, 2606:4700:4400::ac40:9267, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|104.18.41.153|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:34 ERROR 404: Not Found.
Codec libxl_decoder.so does not exist. Downloading...
--2023-09-14 12:21:34-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxl_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 104.18.41.153, 172.64.146.103, 2606:4700:4400::ac40:9267, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|104.18.41.153|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:34 ERROR 404: Not Found.
Codec libxma1_decoder.so does not exist. Downloading...
--2023-09-14 12:21:34-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxma1_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:34 ERROR 404: Not Found.
Codec libxma2_decoder.so does not exist. Downloading...
--2023-09-14 12:21:34-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxma2_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::ac40:9267, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:34 ERROR 404: Not Found.
Codec libxwd_decoder.so does not exist. Downloading...
--2023-09-14 12:21:34-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libxwd_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:35 ERROR 404: Not Found.
Codec libyop_decoder.so does not exist. Downloading...
--2023-09-14 12:21:35-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libyop_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 104.18.41.153, 172.64.146.103, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|104.18.41.153|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:35 ERROR 404: Not Found.
Codec libzerocodec_decoder.so does not exist. Downloading...
--2023-09-14 12:21:35-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libzerocodec_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:35 ERROR 404: Not Found.
Codec libzlib_decoder.so does not exist. Downloading...
--2023-09-14 12:21:35-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libzlib_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 172.64.146.103, 104.18.41.153, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|172.64.146.103|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:35 ERROR 404: Not Found.
Codec libzmbv_decoder.so does not exist. Downloading...
--2023-09-14 12:21:35-- https://downloads.plex.tv/codecs//linux-x86_64-standard/libzmbv_decoder.so
Resolving downloads.plex.tv (downloads.plex.tv)... 104.18.41.153, 172.64.146.103, 2606:4700:4400::6812:2999, ...
Connecting to downloads.plex.tv (downloads.plex.tv)|104.18.41.153|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-09-14 12:21:35 ERROR 404: Not Found.
EAE_SUPPORT => 1
EAE_EXECUTABLE => /codecs/-linux-x86_64-standard/EasyAudioEncoder/EasyAudioEncoder/EasyAudioEncoder
FFMPEG_HWACCEL => false
ON_DEATH: debug mode enabled for pid [916]
Computed CPU ops => 1253573
Initializing Worker 47a8b11c-93f8-4b83-bd29-8c371916f307|plex-worker-{{.Node.Hostname}}
Worker listening on port 3501
And my pms instance is showing these .run line 4 over and over
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
Calling external transcoder: /app/transcoder.js
ON_DEATH: debug mode enabled for pid [182429]
Local Relay enabled, traffic proxied through PMS local port 32499
Setting VERBOSE to ON
Sending request to orchestrator on: http://localhost:3500
cwd => "/transcode/Transcode/Sessions/plex-transcode-8d7ok3b6te9ih211tdiqwic1-c3bad9b9-e909-431f-87f7-748c4bbef02d"
args => ["-codec:0","hevc","-ss","0","-noaccurate_seek","-analyzeduration","20000000","-probesize","20000000","-i","/tv/It's Always Sunny in Philadelphia/Season 08/It's Always Sunny in Philadelphia - S08E08 - Charlie Rules the World.mkv","-map","0:0","-codec:0","copy","-map","0:1","-codec:1","copy","-f","dash","-seg_duration","5","-dash_segment_type","mp4","-init_seg_name","init-stream$RepresentationID$.m4s","-media_seg_name","chunk-stream$RepresentationID$-$Number%05d$.m4s","-window_size","5","-delete_removed","false","-skip_to_segment","1","-time_delta","0.0625","-manifest_name","http://192.168.1.147:32499/video/:/transcode/session/8d7ok3b6te9ih211tdiqwic1/c3bad9b9-e909-431f-87f7-748c4bbef02d/manifest?X-Plex-Http-Pipeline=infinite","-avoid_negative_ts","disabled","-map_metadata","-1","-map_chapters","-1","dash","-start_at_zero","-copyts","-vsync","cfr","-y","-nostats","-loglevel","verbose","-loglevel_plex","verbose","-progressurl","http://192.168.1.147:32499/video/:/transcode/session/8d7ok3b6te9ih211tdiqwic1/c3bad9b9-e909-431f-87f7-748c4bbef02d/progress"]
env => {"PUID":"1000","PLEX_ARCH":"amd64","HOSTNAME":"plex","S6_CMD_WAIT_FOR_SERVICES_MAXTIME":"0","LANGUAGE":"en_US.UTF-8","TRANSCODE_OPERATING_MODE":"both","ORCHESTRATOR_URL":"http://localhost:3500","PWD":"/transcode/Transcode/Sessions/plex-transcode-8d7ok3b6te9ih211tdiqwic1-c3bad9b9-e909-431f-87f7-748c4bbef02d","PLEX_DOWNLOAD":"https://downloads.plex.tv/plex-media-server-new","PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS":"6","NVIDIA_DRIVER_CAPABILITIES":"compute,video,utility","PMS_IP":"192.168.1.147","TZ":"America/Chicago","PLEX_MEDIA_SERVER_USER":"abc","HOME":"/root","LANG":"en_US.UTF-8","PGID":"1000","VIRTUAL_ENV":"/lsiopy","S6_VERBOSITY":"1","S6_STAGE2_HOOK":"/docker-mods","TERM":"xterm","PLEX_MEDIA_SERVER_INFO_VENDOR":"Docker","PLEX_MEDIA_SERVER_HOME":"/usr/lib/plexmediaserver","DOCKER_MODS":"ghcr.io/pabloromeo/clusterplex_dockermod:latest","X_PLEX_TOKEN":"local-72fd516c-42d2-4bf1-9dc0-7a066ff7f89d","PLEX_MEDIA_SERVER_INFO_MODEL":"x86_64","SHLVL":"0","LD_LIBRARY_PATH":"/usr/lib","PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION":"5.15.0-83-generic","LIBVA_DRIVERS_PATH":"/config/Library/Application Support/Plex Media Server/Cache/va-dri-linux-x86_64","LSIO_FIRST_PARTY":"true","PMS_PORT":"32400","PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR":"/config/Library/Application Support","PATH":"/command:/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VERSION":"docker","LOCAL_RELAY_PORT":"32499","LOCAL_RELAY_ENABLED":"1","DEBIAN_FRONTEND":"noninteractive","PLEX_MEDIA_SERVER_INFO_DEVICE":"Docker Container (LinuxServer.io)","FFMPEG_EXTERNAL_LIBS":"/config/Library/Application\\ Support/Plex\\ Media\\ Server/Codecs/55f77b3-4598-linux-x86_64/","TRANSCODER_VERBOSE":"1"}
JobPoster connected, announcing
Orchestrator requesting pending work
Sending request to orchestrator on: http://localhost:3500
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
Distributed transcoder failed, calling local
Calling external transcoder: /app/transcoder.js
ON_DEATH: debug mode enabled for pid [182460]
Local Relay enabled, traffic proxied through PMS local port 32499
Setting VERBOSE to ON
Sending request to orchestrator on: http://localhost:3500
cwd => "/transcode/Transcode/Sessions/plex-transcode-8d7ok3b6te9ih211tdiqwic1-fe5bc8c9-e336-4316-9639-18231dfcfb03"
args => ["-codec:0","hevc","-ss","195","-noaccurate_seek","-analyzeduration","20000000","-probesize","20000000","-i","/tv/It's Always Sunny in Philadelphia/Season 08/It's Always Sunny in Philadelphia - S08E08 - Charlie Rules the World.mkv","-map","0:0","-codec:0","copy","-map","0:1","-codec:1","copy","-f","dash","-seg_duration","5","-dash_segment_type","mp4","-init_seg_name","init-stream$RepresentationID$.m4s","-media_seg_name","chunk-stream$RepresentationID$-$Number%05d$.m4s","-window_size","5","-delete_removed","false","-skip_to_segment","40","-time_delta","0.0625","-manifest_name","http://192.168.1.147:32499/video/:/transcode/session/8d7ok3b6te9ih211tdiqwic1/fe5bc8c9-e336-4316-9639-18231dfcfb03/manifest?X-Plex-Http-Pipeline=infinite","-avoid_negative_ts","disabled","-map_metadata","-1","-map_chapters","-1","dash","-start_at_zero","-copyts","-y","-nostats","-loglevel","verbose","-loglevel_plex","verbose","-progressurl","http://192.168.1.147:32499/video/:/transcode/session/8d7ok3b6te9ih211tdiqwic1/fe5bc8c9-e336-4316-9639-18231dfcfb03/progress"]
env => {"PUID":"1000","PLEX_ARCH":"amd64","HOSTNAME":"plex","S6_CMD_WAIT_FOR_SERVICES_MAXTIME":"0","LANGUAGE":"en_US.UTF-8","TRANSCODE_OPERATING_MODE":"both","ORCHESTRATOR_URL":"http://localhost:3500","PWD":"/transcode/Transcode/Sessions/plex-transcode-8d7ok3b6te9ih211tdiqwic1-fe5bc8c9-e336-4316-9639-18231dfcfb03","PLEX_DOWNLOAD":"https://downloads.plex.tv/plex-media-server-new","PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS":"6","NVIDIA_DRIVER_CAPABILITIES":"compute,video,utility","PMS_IP":"192.168.1.147","TZ":"America/Chicago","PLEX_MEDIA_SERVER_USER":"abc","HOME":"/root","LANG":"en_US.UTF-8","PGID":"1000","VIRTUAL_ENV":"/lsiopy","S6_VERBOSITY":"1","S6_STAGE2_HOOK":"/docker-mods","TERM":"xterm","PLEX_MEDIA_SERVER_INFO_VENDOR":"Docker","PLEX_MEDIA_SERVER_HOME":"/usr/lib/plexmediaserver","DOCKER_MODS":"ghcr.io/pabloromeo/clusterplex_dockermod:latest","X_PLEX_TOKEN":"local-72fd516c-42d2-4bf1-9dc0-7a066ff7f89d","PLEX_MEDIA_SERVER_INFO_MODEL":"x86_64","SHLVL":"0","LD_LIBRARY_PATH":"/usr/lib","PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION":"5.15.0-83-generic","LIBVA_DRIVERS_PATH":"/config/Library/Application Support/Plex Media Server/Cache/va-dri-linux-x86_64","LSIO_FIRST_PARTY":"true","PMS_PORT":"32400","PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR":"/config/Library/Application Support","PATH":"/command:/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VERSION":"docker","LOCAL_RELAY_PORT":"32499","LOCAL_RELAY_ENABLED":"1","DEBIAN_FRONTEND":"noninteractive","PLEX_MEDIA_SERVER_INFO_DEVICE":"Docker Container (LinuxServer.io)","FFMPEG_EXTERNAL_LIBS":"/config/Library/Application\\ Support/Plex\\ Media\\ Server/Codecs/55f77b3-4598-linux-x86_64/","TRANSCODER_VERBOSE":"1"}
JobPoster connected, announcing
Orchestrator requesting pending work
Sending request to orchestrator on: http://localhost:3500
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
Distributed transcoder failed, calling local
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
./run: line 4: /usr/sbin/nginx: No such file or directory
I think I got it under wraps, pulled new, redeployed and changed around some of the localhost/ip addresses of the main instance and seems like its sending out jobs again and all are connecting fine. Will continue to monitor
Any idea what the solution for this was? I have essentially the same docker-compose file as you except:
BTW, /usr/sbin/nginx does not exist on the any of the containers, though nginx is running in the main plex container launched by s6-supervise.
root 328 0.4 0.0 208 76 ? S 09:04 0:28 s6-supervise nginx
@pabloromeo or anyone... any ideas?
Thanks
Hi, I'm currently away for the next few days. I'll take a look again when I'm close to a computer again. However, if nginx isn't present it may have failed during the initialization and install stage, which would show something in the logs. You can try deleting the container and running it again while taking a look at the logs. If I remember correctly you should see entries for when it tries to install nginx along with other stuff. Maybe a networking error is preventing the download and install through apt.
@pabloromeo Thanks for responding. After working on this more, I noticed that startup of pms showed lots of errors when running the scripts from the dockermod. I decided to instead use your custom images for everything which already include the docker mods and got past this error. All containers started up and plex is running.
Now, I'm having a transcoding issue, which I believe is because start.sh on the worker doesn't have some of the variables set, such as: CLUSTERPLEX_PLEX_CODECS_VERSION, CLUSTERPLEX_PLEX_EAE_VERSION I'll open a separate ticket for that.
I wanted to reach out here because while I got clusterplex to run in docker containers across 3 machines, it's not looking like the other machines are picking up any transcoding jobs. I get
distributed transcoder failed, calling local
on the plex server but not sure if that means its not receiving or if because my LOCAL_RELAY_ENABLED = 1, it will always fail and is still sending?Here is a pic of the log from clusterplex-plex-1:
in proxmox for the vm, it looks like the one where clusterplex is on and the orchestrator, that is getting hit but on my other 2 nodes where I just have the worker jobs running in docker containers, those vm's aren't getting much of any cpu hit with 5 people streaming and transcoding.
Here is my docker Container on the main node:
Here is my worker docker compose running on the other 2 nodes:
This is one of my node worker logs:
Here is the plex service on the main node:
Here is the orchestrator on the main node:
Let me know if anything seems wrong.... or if it all does. I have no idea whats right or wrong in those.