m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Remove live v1 annotation logic from data pipeline #1087

Closed stephen-soltesz closed 2 years ago

stephen-soltesz commented 2 years ago

Now that all network and geographic annotations are static, produced either by the uuid-annotator at measurement time, or historical exports from synthetic uuid annotations, there is no longer any need for the real time annotation logic left from the v1 pipeline.

This change removes almost all references to the annotation parameters and interfaces. The row package and references to row.NullAnnotator in all current schemas remain to be removed in a future PR.

Part of:


This change is Reviewable

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 7335


Changes Missing Coverage Covered Lines Changed/Added Lines %
parser/ndt.go 3 4 75.0%
parser/ss.go 2 3 66.67%
parser/parser.go 0 9 0.0%
<!-- Total: 33 44 75.0% -->
Files with Coverage Reduction New Missed Lines %
parser/ndt.go 72 48.24%
<!-- Total: 72 -->
Totals Coverage Status
Change from base Build 7325: -1.09%
Covered Lines: 3405
Relevant Lines: 5284

💛 - Coveralls
stephen-soltesz commented 2 years ago

I believe the coverage lost in ndt.go will be regained with the removal of the row package and what is essentially dead code in the ndt parser, and sidestream & pt schema files. I could include these changes in this PR too if desired.. but I suspected this will involve interface updates that propagate into more meaningful code changes than simple removals in this PR.

stephen-soltesz commented 2 years ago

Okay - actually, the row package is the newer replacement for the parser.Base interface; the next PR would remove parser.Base. All the same, that change is still too much for this PR, I think.