pabloromeo / clusterplex

ClusterPlex is an extended version of Plex, which supports distributed Workers across a cluster to handle transcoding requests.
MIT License
452 stars 35 forks source link

Orchestrator errors and so does not use worker container #337

Closed stellyuk closed 4 weeks ago

stellyuk commented 4 weeks ago

Describe the bug Error calling orchestrator To Reproduce Steps to reproduce the behavior:

  1. Install clusterplex using helm deployment
  2. Add films
  3. play film
  4. using PMS and not worker containers

Expected behavior use the worker container that have been spun up

Screenshots No screenshots needed attached error log from orchestrator

Desktop (please complete the following information):

Additional context Local Relay enabled, traffic proxied through PMS local port 32499 Setting VERBOSE to ON Sending request to orchestrator on: http://plex-cluster-orchestrator:3500 cwd => "/config/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-3xkymojgtlbtzdbs82gwvm0q-f1eae8ee-4f2e-47bc-be2e-3215389e340c" args => ["-codec:0","h264","-codec:1","eac3_eae","-eaeprefix:1","3xkymojgtlbtzdbs82gwvm0q","-ss","0","-noaccurate_seek","-analyzeduration","20000000","-probesize","20000000","-i","/media/Movies/The Union (2024)/The Union (2024).mkv","-map","0:0","-codec:0","copy","-filter_complex","[0:1] aresample=async=1:ochl='stereo':rematrix_maxval=60.000000dB:osr=48000[0]","-map","[0]","-metadata:s:1","language=eng","-codec:1","aac","-b:1","256k","-f","dash","-seg_duration","5","-dash_segment_type","mp4","-init_seg_name","init-stream$RepresentationID$.m4s","-media_seg_name","chunk-stream$RepresentationID$-$Number%05d$.m4s","-window_size","5","-delete_removed","false","-skip_to_segment","1","-time_delta","0.0625","-manifest_name","http://plex-cluster-pms:32499/video/:/transcode/session/3xkymojgtlbtzdbs82gwvm0q/f1eae8ee-4f2e-47bc-be2e-3215389e340c/manifest?X-Plex-Http-Pipeline=infinite","-avoid_negative_ts","disabled","-map_metadata","-1","-map_chapters","-1","dash","-start_at_zero","-copyts","-vsync","cfr","-y","-nostats","-loglevel","verbose","-loglevel_plex","verbose","-progressurl","http://plex-cluster-pms:32499/video/:/transcode/session/3xkymojgtlbtzdbs82gwvm0q/f1eae8ee-4f2e-47bc-be2e-3215389e340c/progress"] env => {"KUBERNETES_SERVICE_PORT_HTTPS":"443","PLEX_CLUSTER_PMS_PORT_32400_TCP_PORT":"32400","KUBERNETES_SERVICE_PORT":"443","PMS_SERVICE":"plex-cluster-pms","PLEX_CLUSTER_ORCHESTRATOR_PORT":"tcp://10.43.141.42:3500","PUID":"1026","PLEX_ARCH":"amd64","HOSTNAME":"plex-cluster-pms-fc55d4fd6-s5f6x","S6_CMD_WAIT_FOR_SERVICES_MAXTIME":"0","LANGUAGE":"en_US.UTF-8","PLEX_CLUSTER_WORKER_SERVICE_HOST":"10.43.215.138","TRANSCODE_OPERATING_MODE":"both","PLEX_CLUSTER_WORKER_PORT_3501_TCP_PROTO":"tcp","ORCHESTRATOR_URL":"http://plex-cluster-orchestrator:3500","PLEX_CLUSTER_PMS_PORT_32400_TCP_PROTO":"tcp","PLEX_CLUSTER_ORCHESTRATOR_SERVICE_PORT":"3500","PLEX_CLUSTER_ORCHESTRATOR_PORT_3500_TCP_ADDR":"10.43.141.42","PLEX_CLUSTER_PMS_PORT":"tcp://10.43.250.69:32400","PLEX_CLUSTER_ORCHESTRATOR_SERVICE_PORT_HTTP":"3500","PWD":"/config/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-3xkymojgtlbtzdbs82gwvm0q-f1eae8ee-4f2e-47bc-be2e-3215389e340c","PLEX_DOWNLOAD":"https://downloads.plex.tv/plex-media-server-new","PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS":"6","PLEX_CLUSTER_PMS_PORT_32499_TCP_PORT":"32499","PLEX_CLUSTER_PMS_PORT_32499_TCP":"tcp://10.43.250.69:32499","NVIDIA_DRIVER_CAPABILITIES":"compute,video,utility","PLEX_CLUSTER_ORCHESTRATOR_PORT_3500_TCP_PROTO":"tcp","PLEX_CLUSTER_PMS_SERVICE_PORT_PLEX":"32400","TZ":"Europe/London","PLEX_MEDIA_SERVER_USER":"abc","HOME":"/root","LANG":"en_US.UTF-8","KUBERNETES_PORT_443_TCP":"tcp://10.43.0.1:443","PLEX_CLUSTER_WORKER_PORT_3501_TCP_PORT":"3501","PLEX_CLUSTER_PMS_PORT_32400_TCP_ADDR":"10.43.250.69","PGID":"101","VIRTUAL_ENV":"/lsiopy","PLEX_CLUSTER_WORKER_PORT_3501_TCP":"tcp://10.43.215.138:3501","S6_VERBOSITY":"1","S6_STAGE2_HOOK":"/docker-mods","PLEX_CLUSTER_PMS_SERVICE_PORT":"32400","PLEX_CLUSTER_PMS_SERVICE_HOST":"10.43.250.69","PLEX_CLAIM":"claim-bUWq6xZnSLqU6QfKpoeM","TERM":"xterm","PLEX_MEDIA_SERVER_INFO_VENDOR":"Docker","PLEX_MEDIA_SERVER_HOME":"/usr/lib/plexmediaserver","PLEX_CLUSTER_ORCHESTRATOR_PORT_3500_TCP_PORT":"3500","PLEX_CLUSTER_PMS_PORT_32499_TCP_PROTO":"tcp","PLEX_CLUSTER_PMS_SERVICE_PORT_RELAY":"32499","DOCKER_MODS":"ghcr.io/pabloromeo/clusterplex_dockermod:1.4.13","X_PLEX_TOKEN":"local-3777e151-94b2-47fa-8674-576aa27c515b","PLEX_MEDIA_SERVER_INFO_MODEL":"x86_64","SHLVL":"0","PLEX_CLUSTER_ORCHESTRATOR_SERVICE_HOST":"10.43.141.42","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_PORT_443_TCP_ADDR":"10.43.0.1","PLEX_CLUSTER_WORKER_PORT":"tcp://10.43.215.138:3501","LD_LIBRARY_PATH":"/usr/lib","PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION":"5.15.0-113-generic","PLEX_CLUSTER_WORKER_PORT_3501_TCP_ADDR":"10.43.215.138","LIBVA_DRIVERS_PATH":"/config/Library/Application Support/Plex Media Server/Cache/va-dri-linux-x86_64","KUBERNETES_SERVICE_HOST":"10.43.0.1","LSIO_FIRST_PARTY":"true","KUBERNETES_PORT":"tcp://10.43.0.1:443","KUBERNETES_PORT_443_TCP_PORT":"443","PLEX_CLUSTER_WORKER_SERVICE_PORT":"3501","PMS_PORT":"32400","PLEX_CLUSTER_PMS_PORT_32499_TCP_ADDR":"10.43.250.69","PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR":"/config/Library/Application Support","PLEX_CLUSTER_ORCHESTRATOR_PORT_3500_TCP":"tcp://10.43.141.42:3500","PLEX_CLUSTER_WORKER_SERVICE_PORT_HTTP":"3501","PATH":"/command:/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VERSION":"docker","LOCAL_RELAY_PORT":"32499","LOCAL_RELAY_ENABLED":"1","EAE_ROOT":"/tmp/pms-7689d69b-d98b-4bd1-a2d5-53ccd5a9ab56/EasyAudioEncoder","PLEX_CLUSTER_PMS_PORT_32400_TCP":"tcp://10.43.250.69:32400","DEBIAN_FRONTEND":"noninteractive","PLEX_MEDIA_SERVER_INFO_DEVICE":"Docker Container (LinuxServer.io)","FFMPEG_EXTERNAL_LIBS":"/config/Library/Application\ Support/Plex\ Media\ Server/Codecs/27d3929-731e70e17b964ba367f4016a-linux-x86_64/","TRANSCODERVERBOSE":"1","":"/usr/local/bin/node"} JobPoster connected, announcing Orchestrator requesting pending work Sending request to orchestrator on: http://plex-cluster-orchestrator:3500 Distributed transcoder failed, calling local Main process is no longer running

pabloromeo commented 4 weeks ago

Do you have the logs from the orchestrator and from the worker?

stellyuk commented 4 weeks ago

@pabloromeo I have attached them mate - thanks alot for looking at this - probably something I'm missing in the helm deployment. plex-cluster-orchestrator.log plex-cluster-worker.log

pabloromeo commented 4 weeks ago

Found this in the worker logs:

cwd => /tmp/pms-7689d69b-d98b-4bd1-a2d5-53ccd5a9ab56/EasyAudioEncoder EAE Support - Writing PID file CWD path doesn't seem to exist. Plex should have created this path before-hand, so you may have an issue with your shares => "/config/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-2zvdx7ax89ea3jvktd8xo6hx-bfd765a3-3270-4dfa-a109-b4f222408203" Transcoding failed

It seems your worker and PMS are not sharing transcode path through some form of network storage. PMS initially creates this path where the transcoding will take place. If the worker can't find that same path then that error occurs.

stellyuk commented 4 weeks ago

@pabloromeo I think they are sir - any ideas on this? As you can see from the image attached both pods are using the same PVC entitles transcode

image

stellyuk commented 4 weeks ago

What is supposed to create this path the worker or PMS?

stellyuk commented 4 weeks ago

@pabloromeo Okay - I missed a step in the readme and had to set my transcode path to /transcode - my bad mate - looks to be working a treat now.... really appreciate the help alot

pabloromeo commented 4 weeks ago

Ah, no worries! Glad it's working now!!