plex.service always coredumps when stopped

tavi-vi commented 2 years ago

Describe the bug

Every time the plex.service systemd service is stopped, it coredumps. It does this because the service is set to send SIGQUIT, which looks like it was changed to do in #24900.

Steps To Reproduce

Steps to reproduce the behavior:

set services.plex.enable = true;
run systemctl stop plex
run systemctl status plex, and check the exit code and logs

Expected behavior

The plex service should be terminated with sigterm and systemd should consider it to have exited successfully.

Additional context

This was originally done to prevent database corruption, but doing my own tests, by running systemctl start plex ; sleep 20 ; systemctl kill --signal=SIGTERM plex repeatedly, it doesn't seem like plex corrupts the database, it still functions as expected, and the logs (jouralctl and plex/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.system.log) don't reveal any unusual errors. I think at this point, it's probably safer to kill plex with SIGTERM than SIGQUIT.

I'd appreciate input from @pjones, if you remember what specifically the original problem was. If we can't switch back to using the default SIGTERM stop signal, maybe we could add a SuccessExitStatus to the service so it doesn't show as failed when it's stopped.

Notify maintainers

@pjones @thoughtpolice @Forkk @MayNiklas @maxeaubrey

Metadata

[tavi@daemons:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.102, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

hurricanehrndz commented 1 year ago

In my testing I have come to believe this is due to bubblewrap not forwarding kill signals to child processes.

Plex additionally has a shutdown service that takes sometime, but quits this process if the parent process is terminated:

Dec 05 11:21:18 DeepThought systemd[1]: Stopping Plex Media Server...
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Shutting down with signal 15 (Terminated)
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Ordered to stop server.
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Stopping server...
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: HttpServer: Stopping server.
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: MetadataItemClusterRequestHandler: a shutdown event has been received
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: TranscodeJobManager: [Running] onServerShutdown
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: TranscodeJobManager: [Running] -> [Shutdown]
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Server stopped.
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: NetworkService: Stopping advertisement.
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Network Service: Error in advertiser handle read: 125 (Operation canceled) socket=73
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Server is already in shutdown mode, ignoring this signal...
Dec 05 11:21:18 DeepThought PlexMediaServer[199099]: Killing plug-ins.
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: Received SIGTERM: Software termination signal.
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: Received SIGTERM: Software termination signal.
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: Parent process no longer present; exiting
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] device_manager_t::term
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] device_manager_t::term. Stop all devices
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] device_manager_t::term. Shut down all device auxes
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] tvbutler_aux_t::term
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] web_server_t::stop. Web server is stopped
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] device_manager_t::term
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] device_manager_t::term. Stop all devices
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] device_manager_t::term. Shut down all device auxes
Dec 05 11:21:18 DeepThought PlexMediaServer[199291]: [I] broker_t::shutdown. Shutdown is completed.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: State 'final-sigterm' timed out. Killing.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Killing process 199217 (Plex Script Hos) with signal SIGKILL.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Killing process 199327 (Plex Script Hos) with signal SIGKILL.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Killing process 199271 (Plex Script Hos) with signal SIGKILL.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Killing process 199332 (Plex Script Hos) with signal SIGKILL.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Killing process 199333 (Plex Script Hos) with signal SIGKILL.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Killing process 199338 (n/a) with signal SIGKILL.
Dec 05 11:21:23 DeepThought systemd[1]: plex.service: Failed with result 'timeout'.
Dec 05 11:21:23 DeepThought systemd[1]: Stopped Plex Media Server.

I was able to work around these issues by writing a script

{ pkgs, lib, config, ... }:

let
  pidFile = "${config.services.plex.dataDir}/Plex Media Server/plexmediaserver.pid";
in
{
  services.plex = {
    enable = true;
    openFirewall = true;
  };
  systemd.services.plex.serviceConfig = {
    KillSignal = lib.mkForce "SIGKILL";
    Restart = lib.mkForce "no";
    TimeoutStopSec = 10;
    ExecStop = pkgs.writeScript "plex-stop" ''
      #!${pkgs.bash}/bin/bash

      ${pkgs.procps}/bin/pkill --signal 15 --pidfile "${pidFile}"
      ${pkgs.coreutils}/bin/sleep 5
    '';
    PIDFile = lib.mkForce "";
  };
}

chfoidl commented 3 weeks ago

I improved @hurricanehrndz solution a bit by actually checking if the plex service has been shutdown. This is achieved by checking the existence of the PID file instead of the flat 5 second sleep:


systemd.services.plex.serviceConfig = let
  pidFile = "${config.services.plex.dataDir}/Plex Media Server/plexmediaserver.pid";
in {
  KillSignal = lib.mkForce "SIGKILL";
  Restart = lib.mkForce "no";
  TimeoutStopSec = 10;
  ExecStop = pkgs.writeShellScript "plex-stop" ''
    ${pkgs.procps}/bin/pkill --signal 15 --pidfile "${pidFile}"

    # Wait until plex service has been shutdown
    # by checking if the PID file is gone
    while [ -e "${pidFile}" ]; do
      sleep 0.1
    done

    ${pkgs.coreutils}/bin/echo "Plex shutdown successful"
  '';
  PIDFile = lib.mkForce "";
};
``

NixOS / nixpkgs