Add app_restart command

[porst17]

... to server and station scripts

[malex984]

plan

1. detect the currently running (or exited or crashed) local app and restart it
2. otherwise - start the default one

[elondaits]

Even if you add this (which might be OK) I probably won't use it in the dashboard...

... I need to block the UI until the app is up, but I have no way to detect the progress of the restart process and I might see the new process up and think it's the original one (unless I was able to track process via pid).

So it's easier if I stop it myself, wait until checkMK tells me it's down, and then start it again and wait for checkMK to tell me it's up.

[porst17]

When app_restart exits, the new process will be running or at least the container that contains the app, but that's everything the top1 service check checks anyway. For the dashboard, you probably need some kind of additional metadata to tell apart the different instances of the same app. The timestamp of the service check alone is not enough as we already noticed, but if we add the container id (or some other unique identifier) to the output of top1 it should be enough.

But that shouldn't be discussed here, because hilbert-cli knows nothing about monitoring.

[elondaits]

I'm discussing this here just in the spirit of not adding a feature if the only intention is that I use it because maybe I won't.

Originally I was told (and there was never any documentation to the contrary) that I should treat the hilbert cli scripts (now hilbert-cli) as "async calls". Even though now I could consider that the app is in effect restarted once the call to hilbert-cli ends I always understood that perhaps I should wait for it to report back through checkMK because:

• Is it possible that after starting the app CheckMK might not see it as top with an OK status for a couple of seconds? (if the app startup is multi-stage / has a launcher process or something)
• Is it possible that the app takes some time to start and I should wait until trying to kill it / restart it / etc?
• Does launching the app ever do anything in terms of deployment / updating docker images / etc. which might take some time after the script finished?

So basically:

• I'd need some way to track the process
• and some assurance of what are the possible outcomes of an app start / restart in terms of the system state so that we can prevent the user from breaking things. Semantics are the KEY part of the API.

[porst17]

app_restart is necessary on the CLI as well, regardless of the dashboard. Same with reboot.

[elondaits]

Some time ago we had kind of said that users should only use the hilbert-ui cli, which uses HTTP calls to the Hilbert server and only performs operations if its safe and keeping the UI server as informed as possible of the state of stations. Is this no longer true?

[porst17]

hilbert-cli is used a lot during development and debugging - also during installation in the museum. This is done by tech-savvy people and turn-around time and instant debug output matters more then "safe operation". During normal operation of the museum, however, people should not use hilbert-cli but only the hilbert-ui.

Unfortunately, the hilbert-ui cli is currently not exposed to users. It is contained inside the mng container, so regular users without docker knowledge can't access it.