search

joe-at-startupmedia / pmon3

Golang Production Process Manager
10 stars 0 forks source link

new function suggestion #17

Closed BleethNie closed 6 months ago

BleethNie commented 6 months ago

  1. Add a parameter flag to send an alert to email or other platforms if the process is automatically restarted

  2. When the task is restarted, a callback event is added, which is a script

joe-at-startupmedia commented 6 months ago

I could see where that would be useful, however it just barely exceeds the scope of this project: a process manager (not a process alert notifier). As such, I would delegate such notification tasks to a separate module outside of this repository. With that being said, implementing something to perform this is still possible in the current state of the project. Currently events are logged and as such an alerting tool could simply watch for particular events in the log.

1) There are two ways a process can be restarted: user-initiated and pmond. It would only be useful for most to be notified of pmond-initiated restarts.

2) We emit both start and restart events to our process log files. Currently this is only emitted with Info-level verbosity which is currently the default log level. As indicated by the string builder, the pattern to watch for would be "restarting process:" https://github.com/joe-at-startupmedia/pmon3/blob/3fcb9023944385ae830196ac615da5802973957b/pmond/process/process.go#L128

3) Utilize a module like logalert to watch for such events logged to the process file(s). Currently there are individual log files per each process which would have to be monitored using a wildcard like the following: https://github.com/jhuckaby/logalert#multiple-files

Improvements

1) Implement more robust logging to capture other critical events (e.g. process failure) using the proper log-levels as necessary: https://github.com/sirupsen/logrus?tab=readme-ov-file#level-logging

2) Currently the log-level is info by default and debug if the PMON3_DEBUG flag is provided. Making this configurable would probably be ideal: https://github.com/joe-at-startupmedia/pmon3/blob/3fcb9023944385ae830196ac615da5802973957b/pmond/init.go#L21

joe-at-startupmedia commented 6 months ago

~> 3. Currently there are individual log files per each process which would have to be monitored using a wildcard like the following:~

If pmond is managed by systemd the event gets emitted to /var/log/messages so there is no need to monitor each individual process file located in /var/log/pmond. Example: 

[root@node-1]# tail /var/log/messages | grep pmond
Apr 20 19:40:04 node-1 pmond: level=info msg="restarting process: golang-admin /home/git/go/src/golang-admin/golang-admin\n"
joe-at-startupmedia commented 6 months ago

2. When the task is restarted, a callback event is added, which is a script

I've tested the commit above after doing the following.

  1. Specify the script to run for the on_process_restart_exec value. Obviously it must be executable. pmond will pass a json-escaped string of the process details as the first argument.

/etc/pmond/config/config.yaml

on_process_restart_exec: "/etc/pmon3/bin/on_restart.bash"

  1. create the script to accept the json-escaped process details:

    /etc/pmon3/bin/on_restart.bash

    PROCESS_JSON="$1"
PROCESS_ID=$(echo "${PROCESS_JSON}" | jq '.id')
PROCESS_NAME=$(echo "${PROCESS_JSON}" | jq '.name')
echo "process restarted: ${PROCESS_ID} - ${PROCESS_NAME}" >> /var/log/pmond/output.log

  2. start pmond in debug mode

    $ PMON3_DEBUG=true pmond
INFO/vagrant/go_src/pmon3/pmond/observer/observer.go:29 pmon3/pmond/observer.HandleEvent() Received event: &{restarted 0xc0001da630}
WARN/vagrant/go_src/pmon3/pmond/observer/observer.go:47 pmon3/pmond/observer.onRestartEvent() restarting process: happac3 (3)
DEBU/vagrant/go_src/pmon3/pmond/observer/observer.go:70 pmon3/pmond/observer.onEventExec() Attempting event executor(restarted): /etc/pmon3/bin/on_restart.bash "{\"id\":3,\"created_at\":\"2024-05-03T05:44:25.114957302Z\",\"updated_at\":\"2024-05-03T06:09:18.71222185Z\",\"pid\":4952,\"log\":\"/var/log/pmond/acf3f83.log\",\"name\":\"happac3\",\"process_file\":\"/usr/local/bin/happac\",\"args\":\"-h startup-patroni-1.node.consul -p 5557 -r 5002\",\"status\":2,\"auto_restart\":true,\"uid\":1000,\"username\":\"vagrant\",\"gid\":1000}"

  3. confirm the script executed successfully

    $ tail /var/log/pmond/output.log
process restarted: 4 - "happac4"
  1. Add a parameter flag to send an alert to email or other platforms if the process is automatically restarted

You can delegate email sending to the bash script provided in the following parameters:

# a script to execute when a process is restarted which accepts the process details json as the first argument
on_process_restart_exec: ""
# a script to execute when a process fails (--no-autorestart) which accepts the process details json as the first argument
on_process_failure_exec: ""