search

awslabs / athena-glue-service-logs

Glue scripts for converting AWS Service Logs for use in Athena
Apache License 2.0
142 stars 46 forks source link

Support for multiple jobs for a service type? #29

Open taschmidt opened 3 years ago

taschmidt commented 3 years ago

We have multiple CloudFront distributions (across a couple different AWS accounts). It seems like the job definition file only allows a single job per service type. It seems like allowing the job definition file to be overridden could help both those situations.

Like maybe changing this line https://github.com/awslabs/athena-glue-service-logs/blob/master/Makefile#L6 to: JOB_DEFINITION_FILE?=scripts/glue_jobs.json ?

dacort commented 3 years ago

Hey @taschmidt

You've got a good point here - easiest thing is probably just adding a line as you mentioned, but in reality the glue_jobs.json file should probably just be a list of job definitions rather than a dictionary with the key being hard-coded to the service type.

I've been experimenting with Glue blueprints as a replacement for big chunks of this script so will take that into consideration.