m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add job source bucket to output path #1101

Closed stephen-soltesz closed 1 year ago

stephen-soltesz commented 1 year ago

Today, when we change the archive source bucket for a datatype in the gardener config, the output data mixes results from two different sources. We can work around this by deleting the bucket directory, but this is slow. Better for them to remain separate.

This change includes the job source bucket in the output path so that the gardener system preserves the separation between the two directories.

Both the parser and gardener must agree on this path. So, this change must be deployed with its companion in etl-gardener https://github.com/m-lab/etl-gardener/pull/407

Part of:


This change is Reviewable

coveralls commented 1 year ago

Pull Request Test Coverage Report for Build 7402


Changes Missing Coverage Covered Lines Changed/Added Lines %
storage/rowwriter.go 0 1 0.0%
<!-- Total: 1 2 50.0% -->
Files with Coverage Reduction New Missed Lines %
active/active.go 2 90.63%
<!-- Total: 2 -->
Totals Coverage Status
Change from base Build 7397: 0.04%
Covered Lines: 3321
Relevant Lines: 4940

💛 - Coveralls
stephen-soltesz commented 1 year ago

Thank you!