awslabs / athena-glue-service-logs

Glue scripts for converting AWS Service Logs for use in Athena
Apache License 2.0
142 stars 45 forks source link

Partition subset of S3 logs based on date #12

Open bregard opened 5 years ago

bregard commented 5 years ago

I am working with a large set of S3 access logs (90 days). The first 60 days have been moved to Glacier. I really only need the last 30 days. Its still a large amount of objects (~1.5 million). Could you point me to the easiest way to modify this code base to run for smaller date range? Even one day at a time would suffice. Thanks.