ENH: tail *all* the EPICS IOC log files

[klauer]

TL;DR

This works, overall.

Notes

• It doesn't use filebeats
  • We didn't even pin a core with ~1000 IOC log files (ls /reg/d/iocData/ioc-*/iocInfo/ioc.log|wc -l -> 997) open on ctl-logdev01 according to an informal check with htop
  • So... anecdotally that additional layer doesn't seem necessary here?
  • I don't yet know how to use them either, so perhaps there's some big benefit I'm missing.
• Grafana dashboards are very handy especially with setting up filtering options (see below)
• Choosing latin-1 for string encoding seemed right
• There's additional filtering for removing ANSI color colors and prepended timestamps in the messages (replacing them with when logstash tails the file)
• This looks to be the only index that really needs rolling over at the end of the month

Screenshots

[klauer]

I think if we end up going this route:

• We may need to look more into how much of a burden the queries will be rather than data insertion (which appears minimal). 100% on 4 VM cores during some dashboard reloading.
• It will obviate the need for EPICS errlog (which seems nigh useless anyway...)

[klauer]

Ran out of dev database space pretty quickly with this one. It will absolutely require rollover and perhaps an adjustment to the retention policy. Cleared bi-monthly or so?

May consider paring down the document details as well. No real need to duplicate into the sub-dict log, it just follows conventions of the others.

[klauer]

Thought I had a few months (a year?) ago:

• Filter log messages through Python or some other script
• Buffer messages for a certain period
• Generate message hashes, removing identical/duplicate messages
• Detect very similar messages coming in at a high rate (timestamp differs, etc) and summarize these
• Pipe filtered messages to logstash along with reduced summaries of the spam messages

[klauer]

I can't find a way to do this as dynamically as I want natively in logstash itself. Though you can run arbitrary ruby code in a pipeline, there does not appear to be a different scope on a per-file basis or anything along those lines.

I'm not comfortable enough with ruby to make an attempt at something like this, and Python will be more easily maintainable for our team in the future. So, I think an okayish solution would include:

• Python daemon/helper script with per-log file asyncio tasks
  • Watches for new log files via a simple glob
  • Tails all log files and tracks their state
  • State includes a rolling buffer of recently-logged lines (on a per-file basis, as in the task)
  • Periodically outputs duplicate summary information so that users are aware it's spammy
  • Using a single TCP socket connection, dump de-duplicated/summarized log data (JSON output straightaway?)
• New Logstash pipeline to intercept the summarized, pre-processed data

[klauer]

Start of this is here: https://github.com/pcdshub/epics-log-squasher

[klauer]

Closing this as we'll use the Loki-based solution @jbellister-slac is working on