m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add parser for hopannotation1 datatype #1015

Closed cristinaleonr closed 3 years ago

cristinaleonr commented 3 years ago

Test data under gs://archive-mlab-testing/ndt/hopannotation1/2021/08/18/20210818T174432Z-hopannotation1-mlab1-arn03-ndt.tar

Sample output

{
  "ID":"20210818_1e0b318cf3c2_72.14.203.143",
  "Parser":  {
    "Version":"local development",
    "Time":"2021-08-19T01:04:37.447687391Z",
    "ArchiveURL":"gs://archive-mlab-testing/ndt/hopannotation1/2021/08/18/20210818T174432Z-hopannotation1-mlab1-arn03-ndt.tar",
    "Filename":"hopannotation1/2021/08/18/20210818T174429Z_1e0b318cf3c2_72.14.203.143.json",
    "Priority":0,
    "GitCommit":"uninitialized"
  },
  "Date":"2021-08-18",
  "Raw":  {
    "ID":"20210818_1e0b318cf3c2_72.14.203.143",
    "Timestamp":"2021-08-18T17:44:29Z",
    "Annotations":  {
      "Geo":  {
        "ContinentCode":"NA",
        "CountryCode":"US",
        "CountryName":"United States",
        "Latitude":37.751,
        "Longitude":-97.822,"AccuracyRadiusKm":1000
      },
      "Network":  {
        "CIDR":"72.14.192.0/18",
        "ASNumber":15169,
        "ASName":"Google LLC",
        "Systems": [{"ASNs":[15169]}]
      }
    }
  }
}

This change is Reviewable

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 6638


Changes Missing Coverage Covered Lines Changed/Added Lines %
parser/parser.go 0 2 0.0%
schema/hopannotation1.go 9 11 81.82%
parser/hopannotation1.go 56 75 74.67%
<!-- Total: 66 89 74.16% -->
Files with Coverage Reduction New Missed Lines %
active/active.go 4 88.54%
<!-- Total: 4 -->
Totals Coverage Status
Change from base Build 6632: 0.1%
Covered Lines: 3643
Relevant Lines: 5762

💛 - Coveralls
cristinaleonr commented 3 years ago

@SaiedKazemi, can you please take a look at the schema definitions https://github.com/m-lab/etl/pull/1015/files#diff-667275c909e96a34d4817e97f3b68c37fccc55da30ddd20f6bb229bafc0bc570R15 and descriptions https://github.com/m-lab/etl/pull/1015/files#diff-e82f937a1142bc2a12b7e30b30b39dddca417ac068593221b1f0006bc5da56fa? Thanks!