asgs-mon next steps

[wwlwpd]

Integration with asgs_main.sh

• [x] asgs_main.sh PID find:

ps aux | grep [a]sgs_main.sh | grep "$ASGS_CONFIG" | awk '{ print $2 }'

• [ ] implement support for the following

# snippet of my proposed $ASGS_CONF, with some context variables to demonstrate what I am copying
OPENDAPPOST=opendap_post2.sh
POSTPROCESS=( includeWind10m.sh createOPeNDAPFileList.sh $OPENDAPPOST )
ASGSMON=( 000-asgs_main-pid-check 001-instance-status-check 002-hook-status-check 003-syslog-progress 004-instance-status-progress 005-hook-status-progress 006-rundir-du 007-failed-dir )

Triage the following issues as they related,

• [ ] https://github.com/StormSurgeLive/asgs/issues/1291
• [x] https://github.com/StormSurgeLive/asgs/issues/1214
• [ ] https://github.com/StormSurgeLive/asgs/issues/1162
• [x] https://github.com/StormSurgeLive/asgs/issues/1103
• [ ] https://github.com/StormSurgeLive/asgs/issues/897
• [ ] https://github.com/StormSurgeLive/asgs/issues/55

Additional plugins to create:

• [ ] "is this thing still running" for long running modes (daily heartbeat email)
• [ ] SU budget check (will be machine specific most likely)
• [ ] quota based checked
• [ ] ASGS directory "age" check (for systems with a purge)
• [ ] basic check on jobs stuck in queue

Future check ideas:

• [ ] deep inspection of SYSLOG
• [ ] deep inspection instance JSON
• [ ] deep inspection of hooks JSON
• [ ] availability of TDS servers, verify delivery of output on remote servers
• [ ] integration with a heartbeat dashboard
• [ ] check status of meteorological data and just report when new data or NHC forecast is ready/update
• [ ] integrate notifications with Slack, Discord, ntfy.sh

Additional asgs_main.sh integration could be:

1. start asgs-mon with run command, pass directly --pid $$
2. fork and detach from parent process so that asgs-mon doesn't go away
3. output a log that can be tail'd for output
4. ASGSH command that can tail -f this output log so it can be observed at will

[wwlwpd]

Just adding a note here from my experience using the monitor.

1. each asgs-mon instance itself should be tied to an asgs_main.sh
2. it can guess it (the usual case) or can be givin a --pid to watch
3. if the ASGS_PID goes away (asgs-mon should actually watch for this), then it should pause until one is found (it can check periodically)

The issue I am finding is that asgs-mon is happily continuing along when the ASGS_PID goes away, and this is pretty useless.

[wwlwpd]

Another thought after experiencing this for a few days now and monitoring 4 different systems, it'd be nice to get one summary email per notification window (1-3 hours) that had the subject summary:

$HPCENVSHORT $PROFILE: M Critical, N Warnings, P Notifications, Q Unknowns

... emumerated summary of warnings

Example,

subject: qbd HSOFS_nam: 1 Critical, 2 Warning, 1 Notification

Body:

Summary,
...

That decide upfront that only actionable things get emails; e.g.,

1. failed directories detected
2. asgs_main.sh PID went away

I am also not finding the "still alive" heart beat emails super useful.

[wwlwpd]

Another idea, make a plugin for asgs-lint

[wwlwpd]

I am going to peel the monitor off into it's own repo