Managed log source for AWS VPC Flow logs

[timoguin]

Add support for managing VPC Flow logs.

Considerations

Flow logs can now be published to Kinesis Firehose. We should look at implementing the current transformer Lambda in a way that it can be utilized by a Firehose Delivery Stream for transformation. This is probably as close to real-time as we can get. It would allow us to handle the normalization and delivery to S3 (in parquet) in one step, bypassing the data batcher Lambda.

Flow logs can also be published directly to S3 in parquet format. Since we already need to do record-based ECS normalization, these may not be as useful as the text-based delivery. Now that I think of it, would it actually be useful to process the raw parquet files simply due to the smaller file sizes? It would also save on up-front storage costs for the ingestion buckets. Flow logs get big fast.

Flow logs can also be delivered to CloudWatch Logs, although anyone with real volume is probably not doing this because CloudWatch Logs gets expensive. However, streaming CloudWatch Logs to Firehose is a rather nice experience when such high volumes are not a concern.

Tasks

• [x] ECS normalization for VPC Flow logs
• [x] VRL transforms
• [ ] ~Investigate Firehose transforms~

References

• VPC Flow logs: https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html
• VPC Flow logs to Firehose: https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-firehose.html
• Firehose transforms: https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

[shaeqahmed]

Added in #70 with support for ingesting AWS VPC Flow logs in the Text-based format