mozilla-services / foxsec-pipeline

Log analysis pipeline utilizing Apache Beam
Mozilla Public License 2.0
25 stars 9 forks source link

add parser payload metrics #502

Closed kkleemola closed 3 years ago

kkleemola commented 3 years ago

Adds metrics to the parser functions to give us a summary of what payloads are being ingested so we can improve insight into if for example log messages change and we begin parsing as a raw payload instead of the expected event.

kkleemola commented 3 years ago

Awesome. Should we set up an alert in Stackdriver on an increase in RAW payload types or is this more for investigation purposes?

Yep! I'm hoping it's something that we should be able to alert on pretty easily.