Normalization of ingested syslog data

[ehashman]

We need to determine a standardized schema for all our ingested data from syslog.

In particular, we want to ensure that each log entry/alert (hereafter "event") is annotated with:

• timestamp
• the client generating the event
• the customer id of the client generating the event
• the type of event (e.g., Alert Logic, Falconhose...)
• source IP/port
• destination IP/port

[lvh]

Source and destination IPs/ports seem like they'd be specific to some kinds of events.

I'm not sure how to make this ticket actionable. It seems like it would be a per-source bit of mangling.

[lvh]

I am making this about syslog to make it actionable, and adding it to durable storage because that's where we really care.

[lvh]

This might be a convenient location to pick roughly the same things Metron picks.