Extract new message field from Heroku system logs

[sjparkinson]

We currently extract component and dyno from Heroku system logs.

The messages seem to all be free form text, so it might be helpful to extract that into a field called message.

Documentation for field extraction using the props.conf file is at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/Propsconf#Field_extraction_configuration.

Example logs

2022-06-27T11:04:23.307185+00:00 heroku logplex - Error L10 (output buffer overflow): 1 messages dropped since 2022-06-27T11:04:23.261691+00:00.
2022-06-27T10:47:10.956921+00:00 heroku web.1 - State changed from starting to up
2022-06-27T10:47:06.981137+00:00 heroku web.1 - Starting process with command `node dist/server/index.js`
2022-06-27T10:46:57.055996+00:00 heroku web.1 - Stopping all processes with SIGTERM
2022-06-27T10:46:57.357378+00:00 heroku web.1 - Process exited with status 0
2022-06-26T23:32:05.979915+00:00 heroku web.1 - Cycling
2022-06-26T18:22:48.719893+00:00 heroku worker.1 - Starting process with command `node dist/file-feed-worker.js`
2022-06-26T00:00:23.415949+00:00 heroku scheduler.2891 - Starting process with command `node dist/cron.js`
2022-06-26T00:00:23.999033+00:00 heroku scheduler.2891 - State changed from starting to up
2022-06-26T03:52:39.943758+00:00 heroku scheduler.2891 - Process exited with status 0
2022-06-26T03:52:40.07917+00:00 heroku scheduler.2891 - State changed from up to complete

Source type

https://github.com/Financial-Times/splunk-heroku/blob/fbbd353b910231cb9d0e9c251e71227f3b2b6f1d/app/default/props.conf#L22-L26

[sjparkinson]

Bonus field... could we also extract a state field from messages such as "State change from up to crashed"?

I know teams are likely to build alerts based of the crashed messages in this source type.

And also exit_code from "Process exited with status 0" messages?

[sjparkinson]

Closed by #34 🎉