Feature: Check for contents of archive prefix before dispatching to ETL parser.

m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.

Apache License 2.0

13 stars 5 forks source link

Feature: Check for contents of archive prefix before dispatching to ETL parser. #302

Open gfr10598 opened 4 years ago

gfr10598 commented 4 years ago

Since different experiments were launched at different times, some date prefixes have no archive files. Rather than dispatch these to ETL and then handle errors, Gardener should check for the number of archive files in the prefix, and suppress the dispatch if there are no archives.

It could also record the number of archives, and compare this to the number of archives reported by the parser.