TrueNAS SCALE - can't use script due to the healthcheck

[jrelax]

Hello just like This user, I can't use the tool because the TrueNAS plex chart health-check restarts the container. I tried to defeat it by modifying healthcheck.sh inside the container but that is not what it's using. It is using a kubernetes check

Please type a number: 3
 _   _                        _____           _       _   
| | | |                      /  ___|         (_)     | | 
| |_| | ___  __ ___   ___   _\ `--.  ___ _ __ _ _ __ | |_
|  _  |/ _ \/ _` \ \ / / | | |`--. \/ __| '__| | '_ \ __|
| | | |  __/ (_| |\ V /| |_| /\__/ / (__| |  | | |_) | |_ 
\_| |_/\___|\__,_| \_/  \__, \____/ \___|_|  |_| .__/ \__|
                         __/ |                 | |        
                        |___/                  |_|        
main

App Name:  plex-snorlax
Pod:       plex-snorlax-849c97b84d-pwgp6
Container: plex-snorlax

If everything looks correct press enter/spacebar, or press ctrl+c to exit
 _   _                        _____           _       _   
| | | |                      /  ___|         (_)     | | 
| |_| | ___  __ ___   ___   _\ `--.  ___ _ __ _ _ __ | |_
|  _  |/ _ \/ _` \ \ / / | | |`--. \/ __| '__| | '_ \ __|
| | | |  __/ (_| |\ V /| |_| /\__/ / (__| |  | | |_) | |_ 
\_| |_/\___|\__,_| \_/  \__, \____/ \___|_|  |_| .__/ \__|
                         __/ |                 | |        
                        |___/                  |_|        
main

root@plex-snorlax-849c97b84d-pwgp6:/# 
root@plex-snorlax-849c97b84d-pwgp6:/# 
root@plex-snorlax-849c97b84d-pwgp6:/# 
root@plex-snorlax-849c97b84d-pwgp6:/# ls /
bin   config  dev  healthcheck.sh  init              lib    lib64    libx32  mnt  plex-common.sh   proc  run   shared  sys  transcode  var
boot  data    etc  home            installBinary.sh  lib32  libexec  media   opt  plex_service.sh  root  sbin  srv     tmp  usr        version.txt
root@plex-snorlax-849c97b84d-pwgp6:/# 
root@plex-snorlax-849c97b84d-pwgp6:/# 
root@plex-snorlax-849c97b84d-pwgp6:/# /plex_service.sh -d
root@plex-snorlax-849c97b84d-pwgp6:/# cd /config/
root@plex-snorlax-849c97b84d-pwgp6:~# chmod +x DBRepair.sh 
root@plex-snorlax-849c97b84d-pwgp6:~# ./DBRepair.sh 

      Plex Media Server Database Repair Utility (Docker)
                       Version v1.0.3

Select

  1 - 'stop'      - Stop PMS
  2 - 'automatic' - database check, repair/optimize, and reindex in one step.
  3 - 'check'     - Perform integrity check of database
  4 - 'vacuum'    - Remove empty space from database
  5 - 'repair'    - Repair/Optimize  databases
  6 - 'reindex'   - Rebuild database database indexes
  7 - 'start'     - Start PMS

  8 - 'import'    - Import watch history from another database independent of Plex. (risky)
  9 - 'replace'   - Replace current databases with newest usable backup copy (interactive)
 10 - 'show'      - Show logfile
 11 - 'status'    - Report status of PMS (run-state and databases)
 12 - 'undo'      - Undo last successful command

 99 -  exit

Enter command # -or- command name (4 char min) : 3

Checking the PMS databases
Check complete.  PMS main database is OK.
Check complete.  PMS blobs database is OK.

Select

  1 - 'stop'      - Stop PMS
  2 - 'automatic' - database check, repair/optimize, and reindex in one step.
  3 - 'check'     - Perform integrity check of database
  4 - 'vacuum'    - Remove empty space from database
  5 - 'repair'    - Repair/Optimize  databases
  6 - 'reindex'   - Rebuild database database indexes
  7 - 'start'     - Start PMS

  8 - 'import'    - Import watch history from another database independent of Plex. (risky)
  9 - 'replace'   - Replace current databases with newest usable backup copy (interactive)
 10 - 'show'      - Show logfile
 11 - 'status'    - Report status of PMS (run-state and databases)
 12 - 'undo'      - Undo last successful command

 99 -  exit

Enter command # -or- command name (4 char min) : command terminated with exit code 129
This container does not accept shell access, try a different one.
root@snorlax:~# 

describing the pod shows the command it using to check

root@snorlax:~# plexpod=`k3s kubectl get pods -n ix-plex-snorlax | awk 'FNR == 2 {print $1}'` && k3s kubectl describe pod -n ix-plex-snorlax $plexpod
...
    Liveness:              tcp-socket :32400 delay=10s timeout=5s period=10s #success=1 #failure=5
    Readiness:             tcp-socket :32400 delay=10s timeout=5s period=10s #success=2 #failure=5
    Startup:               tcp-socket :32400 delay=30s timeout=2s period=5s #success=1 #failure=180
...

[ChuckPa]

Nothing I can do about that.

Another user had the same problem and submitted an enhancement request to TrueNAS to allow turning healthcheck on/off temporarily .

If there's a solution, it will be there.

Sorry

[ChuckPa]

Who is plex-snorlax ? did someone clone my script?

[jrelax]

Who is plex-snorlax ? did someone clone my script?

lol no. that's the name of my plex container on TrueNAS. Heavyscript is another script that helps you manage the containers and in this case I used it to get a shell on the plex-snorlax container

[ChuckPa]

ok.. LOL

I've seen heavyscript in use. There is yet another person who I helped.
That dialog is likely in the Plex forums.

He also uses HeavyScript.

I do remember that to get a proper resolution here, TrueNAS needs to, and agreed to, provide a method of turning off HealthCheck because of how they actually check to see if PMS is running.

The Plex-supported method is whether the container is running (independent of PMS itself) because the container is set to exit when PMS does.

This whole thing is a mess. I'm trying to now write hard C/C++ code for this but a change in Plex SQLite will KILL it if invoked too soon after PMS is stopped (either a bug or they don't like me HAHAHAHAHA)

[jrelax]

if you had an offline DB repair option where the user takes all responsibility that plex is not running that would work.

[ChuckPa]

I have thought about that. Implementing automatic "Stand Alone" (outside the container) is difficult.

My only problem is ensuring the runtime library environment is sane.

1. Making sure I identify Plex and don't allow anything to happen if running
2. Containers assume Ubuntu runtime libraries
3. Other hosts have RPATH set at link time.

If I implement -p (programs) option and -d (databases) option

DBRepair.sh -p /usr/lib/plexmediaserver -d "/some/crazydir/Plex Media Server/Plug-in Support/Databases"

I can do this but the only risk (unknown) is rpath.

I did speak to someone on the Engineering team. My current bug (being killed) sounds like a PMS bug and I'll take that up with the engineer in charge.

[jrelax]

I agree it is not ideal but maybe a docker image with the sane runtime env plus some flags like —force —database “db” pops up a warning. You continue and it skips all checks for pms running

[ChuckPa]

That's a lot of responsibility. I'm NOT a docker person.

I will look at it but healthchecks will still be problematic because, AFAICT, there's no "process" to ping other than a shell script (which doesn't respond to a socket).

See here.. It's a PITA.
https://docs.docker.com/build/building/base-images/

You can be certain, whatever I do, the databases will be held sacred at all cost.

[jrelax]

If you include a requirements.txt I am sure we can figure it out. If it had that option. I would stop the container. Copy the DB then run the repair against it. That way theres no chance plex is running

[ChuckPa]

I just had a wild hair idea hit me.

Your words about fully manual mode knocked a couple cobwebs loose. I'm about 1/2 way through codifying manual config

example: (using Linux native installation paths)

DBRepair.sh --sqlite /usr/lib/plexmediaserver --databases "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases"  

I will still support the -i, -f, and -p options.

I will use -- for options which take arguments.

Thoughts ??

[ChuckPa]

Thoughts? ( a little cleanup needed)

root@lizum:/home/chuck/git/chuck/PlexDBRepair# ./DBRepair.sh --sqlite /usr/lib/plexmediaserver --databases "/usb/plex/Plug-in Support/Databases" -p
==== Setting DBDIR = "/usb/plex/Plug-in Support/Databases"

      Plex Media Server Database Repair Utility ()
                       Version v1.06.00

      PlexSQLite = '/usr/lib/plexmediaserver/Plex SQLite'
      Databases  = '/usb/plex/Plug-in Support/Databases'

Select

  1 - 'stop'      - (Not available. Stop manually.)
  2 - 'automatic' - Check, Repair/Optimize, and Reindex Database in one step.
  3 - 'check'     - Perform integrity check of database.
  4 - 'vacuum'    - Remove empty space from database without optimizing.
  5 - 'repair'    - Repair/Optimize databases.
  6 - 'reindex'   - Rebuild database database indexes.
  7 - 'start'     - (Not available. Start manually)

  8 - 'import'    - Import watch history from another database independent of Plex. (risky).
  9 - 'replace'   - Replace current databases with newest usable backup copy (interactive).
 10 - 'show'      - Show logfile.
 11 - 'status'    - Report status of PMS (run-state and databases).
 12 - 'undo'      - Undo last successful command.

 21 - 'prune'     - Prune (remove) old image files (jpeg,jpg,png) from PhotoTranscoder cache.
 42 - 'ignore'    - Ignore duplicate/constraint errors.

 88 - 'update'    - Check for updates.
 99 - 'quit'      - Quit immediately.  Keep all temporary files.
      'exit'      - Exit with cleanup options.

Enter command # -or- command name (4 char min) :

[jrelax]

Yes that is exactly what we could use in cases where we run plex on a platform your script doesn’t recognize or doesn’t support for whatever reason!

[ChuckPa]

Come on over to the DBRepair Development thread on the Plex Forum

:sunglasses:

[jrelax]

Found it. Thanks for this!