search

awslabs / athena-glue-service-logs

Glue scripts for converting AWS Service Logs for use in Athena
Apache License 2.0
142 stars 45 forks source link

Feature/alb fixes #9

Closed dacort closed 5 years ago

dacort commented 5 years ago

This incorporates the changes from PR #2 as well as a few others:

  1. Moves conversion logic from RawCatalog class to ConvertedCatalog class - it belongs in the latter. This isn't well documented.
  2. Removes custom timestamp parsing logic. Glue is able to do this natively since source timestamp string is ISO8601.
  3. Use dynamic_frame.apply_mapping to perform the above type conversions.
  4. Make the ALB Grok pattern backwards compatible by using optional fields.

I don't really like the existing approach of having to reimplement the type conversion logic (we do this in VPC logs as well), so at some point this will need to be refactored as it's pretty confusing.