Remote transcode failing, but showing it works in the logs.

[konSMASH]

Describe the bug Hello! I have been running ClusterPlex for a few months with a few small quirks here and there. However, today I noticed that my remote transcoding has actually been failing and I didn't realize it. Everything was just falling back to local. Currently what I am seeing is Plex sees a video started, sends the request to Orchestrator, Orchestrator then send the request to a worker, and it shows that it's working. However, the video will never end up playing. I can sometimes get it to work if I skip ahead 30 seconds. I'm not sure what is

I have a single server within my Docker Swarm handling PMS and Orchestrator, and 6 separate servers within the swarm acting as my workers. They are all labelled so I can direct my services on the servers I'd like them to be on.

Please let me know of any logs you'd like me to pull, I just didn't want to flood the initial report with them if they aren't necessary.

This is my current docker-compose.yml file:

version: '3.8'

services: plex: image: ghcr.io/linuxserver/plex:latest deploy: placement: constraints:

• node.labels.master == true environment: DOCKER_MODS: "ghcr.io/pabloromeo/clusterplex_dockermod:latest" VERSION: docker PUID: 1000 PGID: 1000 TZ: America/Los_Angeles ORCHESTRATOR_URL: http://plex-orchestrator:3500 PMS_SERVICE: plex # This service. If you disable Local Relay then you must use PMS_IP instead PMS_PORT: "32400" TRANSCODE_OPERATING_MODE: remote #(local|remote|both) TRANSCODER_VERBOSE: "1" # 1=verbose, 0=silent LOCAL_RELAY_ENABLED: "1" LOCAL_RELAY_PORT: "32499" healthcheck: test: curl -fsS http://localhost:32400/identity > /dev/null || exit 1 interval: 15s timeout: 15s retries: 5 start_period: 30s volumes:

  • /root/docker-containers/clusterplex/config:/config
  • /mnt/plex_tvshows:/tvshows
  • /mnt/plex_movies:/movies
  • /mnt/plex_tvshows/Plex/Transcode:/transcode ports:
  • 32499:32499 # LOCAL_RELAY_PORT
  • 32400:32400
  • 3005:3005
  • 8324:8324
  • 1900:1900/udp
  • 32410:32410/udp
  • 32412:32412/udp
  • 32413:32413/udp
  • 32414:32414/udp

  plex-orchestrator: image: ghcr.io/pabloromeo/clusterplex_orchestrator:latest deploy: placement: constraints:

• node.labels.orchestrator == true update_config: order: start-first healthcheck: test: curl -fsS http://localhost:3500/health > /dev/null || exit 1 interval: 15s timeout: 15s retries: 5 start_period: 30s environment: TZ: America/Los_Angeles LISTENING_PORT: 3500 WORKER_SELECTION_STRATEGY: "LOAD_RANK" # RR | LOAD_CPU | LOAD_TASKS | LOAD_RANK (default) volumes:

  • /etc/localtime:/etc/localtime:ro ports:
  • 3500:3500

  plex-worker: image: ghcr.io/linuxserver/plex:latest hostname: "plex-worker-{{.Node.Hostname}}" deploy: mode: replicated replicas: 6 placement: constraints:

• node.labels.worker == true environment: DOCKER_MODS: "ghcr.io/pabloromeo/clusterplex_worker_dockermod:latest" VERSION: docker PUID: 1000 PGID: 1000 TZ: America/Los_Angeles LISTENING_PORT: 3501 # used by the healthcheck STAT_CPU_INTERVAL: 2000 # interval for reporting worker load metrics ORCHESTRATOR_URL: http://plex-orchestrator:3500 EAE_SUPPORT: "1" healthcheck: test: curl -fsS http://localhost:3501/health > /dev/null || exit 1 interval: 15s timeout: 15s retries: 5 start_period: 240s volumes:
  • /mnt/plex_tvshows:/tvshows
  • /mnt/plex_movies:/movies
  • /mnt/plex_tvshows/Plex/Transcode:/transcode

[konSMASH]

I will add that I have gone the route of disabling the local relay and adding the PMS_IP in and it still doesn't seem to work. I am using CIFS to mount a NAS for my transcode location.

[pabloromeo]

Hi! One thing I'm seeing, is that you have TRANSCODE_OPERATING_MODE set to local, which means it will always only transcode in your PMS and not on workers. Try setting that to either remote or both to see if remote transcodes start working. Also, you can try the current experimental build instead of latest.

Meaning in the dockermods for both pms and the worker, change :latest for :experimental. That build is using a more up to date EAE as well as some changes regarding how EAE is triggered and how child processes are monitored.

[konSMASH]

Sorry, I updated my yml file to the correct settings when I was having issues. local was there as it was the only way to make it so Plex functioned.

These are the settings I have tried so far to no avail:

• Changing branch to experimental
• Swapping between remote and both for the operating mode.
• Changing PMS_SERVICE from plex to PMS_IP and adding the Plex server IP.
• Changing anything related to plex-orchestrator as a service name to the IP of the orchestrator server. (Since I have it set to the same server as the Plex server)
• Bashing into the Plex container and making sure I can get to plex-orchestrator on port 3500 by running the command "nc -v -z -w 2 plex-orchestrator 3500"
• Completely disabling the local relay and using the PMS_IP, disabling "Enable Relay" within Plex, and adding the worker servers under "List of IP addresses and network that are allowed without auth"
• Changing around the deployment constraints. (The only time this worked was when I used replicas and it replicated a worker to the PMS database server. Basically all that did was a backdoor local transcode)
• Moved codecs to a shared path between all servers within the swarm.

I am at a loss for anything else to test as my Linux knowledge isn't amazing by any means. ClusterPlex was the sole thing that actually got me into Linux. :)

If you have any other ideas I could try please let me know.

Here is my current setup (total of 7 physical servers within a docker swarm): master01 (label "master = true" and "orchestrator=true") - PMS database & Orchestrator worker01 (label "worker = true") - Worker worker02 (label "worker = true") - Worker worker03 (label "worker = true") - Worker worker04 (label "worker = true") - Worker worker05 (label "worker = true") - Worker worker06 (label "worker = true") - Worker

[pabloromeo]

Can you scale the worker replicas to 1, and then review logs when attempting to transcode? There should be something valuable in the worker logs and in plex's UI Console logs. Nothing should be stopping CPU transcoding other than a config issue.

[konSMASH]

Made the changes you requested to the yml file, and here are the logs. I attached the worker log as it has a lot of lines in it for the 30 seconds I tried to play the video.

PMS/Orchestrator server:

clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | Local Relay enabled, traffic proxied through PMS local port 32499 clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | Setting VERBOSE to ON clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | Sending request to orchestrator on: http://plex-orchestrator:3500 clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | cwd => "/transcode/Transcode/Sessions/plex-transcode-20413B97-DFF1-4494-9D29-C16D00900BE1-23ec0144-4894-4ee8-9afa-1fcc4dfe2565" clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | args => ["-codec:0","h264","-noaccurate_seek","-analyzeduration","20000000","-probesize","20000000","-i","/movies/Movies/Expend4bles.2023.1080p.WEBRip.x264.koN.mkv","-map","0:0","-codec:0","copy","-map","0:1","-metadata:s:1","language=eng","-codec:1","copy","-segment_format","matroska","-f","ssegment","-individual_header_trailer","0","-flags","+global_header","-segment_header_filename","header","-segment_time","10","-segment_start_number","0","-segment_copyts","1","-segment_time_delta","0.0625","-segment_list","http://plex:32499/video/:/transcode/session/20413B97-DFF1-4494-9D29-C16D00900BE1/23ec0144-4894-4ee8-9afa-1fcc4dfe2565/manifest?X-Plex-Http-Pipeline=infinite","-segment_list_type","csv","-segment_list_size","5","-segment_list_separate_stream_times","1","-segment_list_unfinished","1","-segment_format_options","output_ts_offset=10","-max_delay","5000000","-avoid_negative_ts","disabled","-map_metadata:g","-1","-map_metadata:c","-1","-map_chapters","-1","media-%05d.ts","-start_at_zero","-copyts","-vsync","cfr","-y","-nostats","-loglevel","verbose","-loglevel_plex","verbose","-progressurl","http://plex:32499/video/:/transcode/session/20413B97-DFF1-4494-9D29-C16D00900BE1/23ec0144-4894-4ee8-9afa-1fcc4dfe2565/progress"] clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | env => {"PMS_SERVICE":"plex","PUID":"1000","PLEX_ARCH":"amd64","HOSTNAME":"c8742d1cb16f","S6_CMD_WAIT_FOR_SERVICES_MAXTIME":"0","LANGUAGE":"en_US.UTF-8","TRANSCODE_OPERATING_MODE":"remote","ORCHESTRATOR_URL":"http://plex-orchestrator:3500","PWD":"/transcode/Transcode/Sessions/plex-transcode-20413B97-DFF1-4494-9D29-C16D00900BE1-23ec0144-4894-4ee8-9afa-1fcc4dfe2565","PLEX_DOWNLOAD":"https://downloads.plex.tv/plex-media-server-new","PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS":"6","NVIDIA_DRIVER_CAPABILITIES":"compute,video,utility","TZ":"America/Los_Angeles","PLEX_MEDIA_SERVER_USER":"abc","HOME":"/root","LANG":"en_US.UTF-8","PGID":"1000","VIRTUAL_ENV":"/lsiopy","S6_VERBOSITY":"1","S6_STAGE2_HOOK":"/docker-mods","TERM":"xterm","PLEX_MEDIA_SERVER_INFO_VENDOR":"Docker","PLEX_MEDIA_SERVER_HOME":"/usr/lib/plexmediaserver","DOCKER_MODS":"ghcr.io/pabloromeo/clusterplex_dockermod:latest","X_PLEX_TOKEN":"local-5f4a39cd-3aeb-4ecc-a7a1-5374345b2c83","PLEX_MEDIA_SERVER_INFO_MODEL":"x86_64","SHLVL":"0","LD_LIBRARY_PATH":"/usr/lib","PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION":"5.10.0-23-amd64","LIBVA_DRIVERS_PATH":"/config/Library/Application Support/Plex Media Server/Cache/va-dri-linux-x86_64","LSIO_FIRST_PARTY":"true","PMS_PORT":"32400","PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR":"/config/Library/Application Support","PATH":"/command:/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VERSION":"docker","LOCAL_RELAY_PORT":"32499","LOCAL_RELAY_ENABLED":"1","DEBIAN_FRONTEND":"noninteractive","PLEX_MEDIA_SERVER_INFO_DEVICE":"Docker Container (LinuxServer.io)","FFMPEG_EXTERNAL_LIBS":"/config/Library/Application\\ Support/Plex\\ Media\\ Server/Codecs/a31c019-4614-linux-x86_64/","TRANSCODER_VERBOSE":"1"} clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | JobPoster connected, announcing clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | Orchestrator requesting pending work clusterplex_plex.1.lmo3aluv3ltd@kon-master01.sevenzerotwo.net | Sending request to orchestrator on: http://plex-orchestrator:3500

Worker server: worker.log

[pabloromeo]

From what I can see in the worker logs, it's actually transcoding fine and it seems the entire transcode finished.

For some reason PMS doesn't seem to be picking up on the segments being written to the shared transcode path.

How is that being shared, NFS? Or CIFS/SMB?

[konSMASH]

From what I can see in the worker logs, it's actually transcoding fine and it seems the entire transcode finished.

For some reason PMS doesn't seem to be picking up on the segments being written to the shared transcode path.

How is that being shared, NFS? Or CIFS/SMB?

Shared via a CIFS mount.

[pabloromeo]

Any chance you could try sharing it over NFS? I've seen other issues reported in the past where CIFS wasn't playing very nice with the way plex monitors the file system for new transcoding segments.

[konSMASH]

Any chance you could try sharing it over NFS? I've seen other issues reported in the past where CIFS wasn't playing very nice with the way plex monitors the file system for new transcoding segments.

I'll see if I can get that done this evening or early tomorrow morning and test. Once I test I'll let you know.

[konSMASH]

Got NFS going and looks to be having the same issues. I can get remote transcoding to work kind of, depending on the player I am using. iPhone, Roku, Windows Plex app, etc do not work and end up just timing out. If I use Google Chrome, it takes about 30 seconds and will eventually start working.

When I start one of the non-working players and let it sit, I end up with the following in the worker logs:

Transcoder close: child process exited with code 255

I think that is just an error showing when the player fails and stops retrying to play the video.

[pabloromeo]

Can you go into Plex's Server Settings -> Manage -> Console, and see what messages come up there during your attempt at transcoding? Also if no errors pop up in that test you can enable Verbose logging in Plex, in Plex's Server Settings -> Settings -> General where you have both Debug, and Verbose logging that can be enabled. They are quite noisy in the Console log, but might shed some light on what's going on.

[konSMASH]

Verbose logging is enabled. You are right, it is very noisy, lol. I am seeing no errors and a lot of text showing transcode requests and completions.

I really am at a loss at what is going on. This issue just doesn't make sense. Transcoding is technically working remotely, but it just isn't pushing to the player.

[pabloromeo]

And you also mentioned that it works fine in Chrome, but not other devices, right? Very odd. We don't do anything specific depending on devices, just forward the transcoding parameters to a remote ffmpeg and let Plex do the rest.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.