m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Allow parsing third-party archives #1029

Closed stephen-soltesz closed 2 years ago

stephen-soltesz commented 2 years ago

This change updates the etl archive path parsing regular expression so that it will match the third-party archives generated by the synthetic uuid annotation export process. The synthetic archives reuse the "machine" field (normally mlab[1-4]) and the "site" field (normally [a-z]{3}\d[0-9t]) to also match "third" and "party" respectively.

With this change, thirdparty archives copied to the public archive will be natively parseable by the etl+gardener system.

For example:

gs://archive-mlab-sandbox/ndt/annotation/2019/07/12/20211107T143543.161376Z-annotation-third-party-ndt.tgz

Tested using the local output mode.


This change is Reviewable

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 6721


Files with Coverage Reduction New Missed Lines %
active/active.go 2 90.63%
<!-- Total: 2 -->
Totals Coverage Status
Change from base Build 6707: 0.02%
Covered Lines: 3766
Relevant Lines: 5902

💛 - Coveralls