cedadev / wflogger

Workflow logging utilities (for JASMIN and LOTUS)
BSD 2-Clause "Simplified" License
0 stars 0 forks source link

Ingest workflow logger data from application logs #15

Closed niallmcc closed 1 year ago

niallmcc commented 1 year ago

When logging in realtime to postgres isn't possible, another option is to write entries to the application's log file and later, ingest the entries from the log file(s) into the database, using log_ingestor.py

Example usage:

python log_ingestor.py --verbose /tmp/test1/*.log

Where log files contain WFL entries starting with a WFL_START token

... WFL_START <user_id> | <hostname> | <workflow> | <tag> | <stage_number> | <stage> | <iteration> | <date_time> | <flag> | <comment>

For example:

Lorem Ipsum etc ... WFL_START fred | compute1 | modeler.py | v14.3 | 2 | model | 1 | 2022-01-01 12:23:05.927111 |Comment| 10
agstephens commented 1 year ago

@niallmcc: what an excellent idea. I've merged.