Today, when we change the archive source bucket for a datatype in the gardener config, the output data mixes results from two different sources. We can work around this by deleting the bucket directory, but this is slow. Better for them to remain separate.
This change includes the job source bucket in the output path so that the gardener system preserves the separation between the two directories.
Today, when we change the archive source bucket for a datatype in the gardener config, the output data mixes results from two different sources. We can work around this by deleting the bucket directory, but this is slow. Better for them to remain separate.
This change includes the job source bucket in the output path so that the gardener system preserves the separation between the two directories.
Both the parser and gardener must agree on this path. So, this change must be deployed with its companion in etl https://github.com/m-lab/etl/pull/1101
Part of:
This change is