CIF Health Monitoring

[GoogleCodeExporter]

1. Every script that process anything should keep track of how long
it took, how many table entries it generated, etc., and log them to
syslog.  That way syslog can be used by a SEIM/SEM/SIEM/SIM/wtf?
to know when something fails to run, and provide some data to draw
nice graphs of processing time, tables sizes, etc. over time.  From
this, some nice linear regression code can predict when resources
will be exhausted well ahead of time, etc.

2. Table sizes, disk space, the same stuff above, ... could all be
stuck into a table and turned into a feed itself, so someone with
the right API key can poll the system for stats and draw the
pretty graphs from a script on a monitoring system, a user
portal, etc. Just having the ability to use the cif command
line tool to give table size counts alone would be a big help
in debugging things with feed parsers, etc.

Right now, the debug output has no time stamps, so it easy
to see when things are happening in the moment, but hard
to look at a log file and tell when something stopped working,
or figure out how long it took/is taking when you make subtle
changes in configuration settings.

ref: https://groups.google.com/d/topic/ci-framework/h7bjP-aFRxE/discussion

Original issue reported on code.google.com by saxjazm...@gmail.com on 17 Jul 2012 at 12:39

[GoogleCodeExporter]

https://groups.google.com/d/topic/ci-framework/TStuPM16IHI/discussion

Original comment by saxjazm...@gmail.com on 17 Jul 2012 at 12:48

• Changed title: CIF Health Monitoring

[GoogleCodeExporter]

Original comment by saxjazm...@gmail.com on 20 Jul 2012 at 11:06

• Added labels: Milestone-v2
• Removed labels: Milestone-v1.1

[GoogleCodeExporter]

https://github.com/collectiveintel/cif-v2/issues/10

Original comment by saxjazm...@gmail.com on 5 Apr 2013 at 2:26

• Changed state: Invalid