m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Add job source bucket to output path #407

Closed stephen-soltesz closed 2 years ago

stephen-soltesz commented 2 years ago

Today, when we change the archive source bucket for a datatype in the gardener config, the output data mixes results from two different sources. We can work around this by deleting the bucket directory, but this is slow. Better for them to remain separate.

This change includes the job source bucket in the output path so that the gardener system preserves the separation between the two directories.

Both the parser and gardener must agree on this path. So, this change must be deployed with its companion in etl https://github.com/m-lab/etl/pull/1101

Part of:


This change is Reviewable

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 3565


Totals Coverage Status
Change from base Build 3553: 0.0%
Covered Lines: 1233
Relevant Lines: 1588

💛 - Coveralls