Add CloudFront log convertor before sending to OpenSearch

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.

https://opensearch.org/docs/latest/clients/data-prepper/index/

Apache License 2.0

265 stars 203 forks source link

Add CloudFront log convertor before sending to OpenSearch #2304

Open YikaiHu opened 1 year ago

YikaiHu commented 1 year ago

Is your feature request related to a problem? Please describe. It would be nice to have a convertor for CloudFront logs before sending to OpenSearch. If we want to build some Dashboard.

Describe the solution you'd like

For fields: "sc-content-len", "sc-range-start", "sc-range-end", we need to transfer them to 0 if the raw log is '-'.
Use parse.unquote_plus to parse "cs-user-agent" field.

dlvenable commented 1 year ago

@YikaiHu , Data Prepper provides an S3 source. You can configure an S3 bucket with an SQS queue for your CloudFront logs to retrieve these. The csv codec should be able to decode the CloudFront files.

Is there anything else in particular you would be looking for?