m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add parser for scamper1 datatype #1019

Closed cristinaleonr closed 3 years ago

cristinaleonr commented 3 years ago

Sample output

{
  "ID": "ndt-vfhzg_1630559948_0000000000024BD6",
  "Parser": {
    "Version": "local development",
    "Time": "2021-09-15T02:02:53.944790299Z",
    "ArchiveURL": "gs://archive-mlab-staging/ndt/scamper1/2021/09/06/20210906T003000.489493Z-scamper1-mlab4-bru01-ndt.tgz",
    "Filename": "2021/09/06/20210906T000000Z_ndt-vfhzg_1630559948_0000000000024BD6.jsonl",
    "Priority": 0,
    "GitCommit": "uninitialized"
  },
  "Date": "2021-09-06",
  "Raw": {
    "Metadata": {
      "UUID": "ndt-vfhzg_1630559948_0000000000024BD6",
      "TracerouteCallerVersion": "d84ea3d",
      "CachedResult": true,
      "CachedUUID": "ndt-vfhzg_1630559948_0000000000024B27"
    },
    "CycleStart": {
      "type": "cycle-start",
      "list_name": "default",
      "id": 0,
      "hostname": "ndt-vfhzg",
      "start_time": 1630885980
    },
    "Tracelb": {
      "Type": "tracelb",
      "Version": "0.1",
      "Userid": 0,
      "Method": "icmp-echo",
      "Src": "195.89.146.178",
      "Dst": "34.139.254.27",
      "Start": {
        "sec": 1630885980,
        "usec": 569320
      },
      "ProbeSize": 44,
      "Firsthop": 1,
      "Attempts": 3,
      "Confidence": 95,
      "Tos": 0,
      "Gaplimit": 0,
      "WaitTimeout": 5,
      "WaitProbe": 150,
      "Probec": 133,
      "ProbecMax": 3000,
      "Nodec": 11,
      "Linkc": 11,
      "Nodes": [
        {
          "UUID": "20210905_ndt-vfhzg_195.89.146.129",
          "Addr": "195.89.146.129",
          "Name": "xe-11-0-1-xcr1.brx.cw.net",
          "QTTL": 1,
          "Linkc": 1,
          "Links": [
            {
              "Links": [
                {
                  "addr": "195.2.16.254",
                  "probes": [
                    {
                      "tx": {
                        "sec": 1630885980,
                        "usec": 719881
                      },
                      "replyc": 1,
                      "ttl": 2,
                      "attempt": 0,
                      "flowid": 1,
                      "replies": [
                        {
                          "rx": {
                            "sec": 1630885980,
                            "usec": 723936
                          },
                          "ttl": 253,
                          "rtt": 4.055,
                          "icmp_type": 11,
                          "icmp_code": 0,
                          "icmp_q_tos": 0,
                          "icmp_q_ttl": 1
                        }
                      ]
                    },
                  ]
                }
              ]
            }
          ]
        },
      ]
    },
    "CycleStop": {
      "type": "cycle-stop",
      "list_name": "default",
      "id": 0,
      "hostname": "ndt-vfhzg",
      "stop_time": 1630886350
    }
  }
}

This change is Reviewable

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 6665


Changes Missing Coverage Covered Lines Changed/Added Lines %
parser/parser.go 0 2 0.0%
schema/scamper1.go 9 11 81.82%
parser/scamper1.go 110 126 87.3%
<!-- Total: 120 140 85.71% -->
Files with Coverage Reduction New Missed Lines %
active/active.go 4 88.54%
<!-- Total: 4 -->
Totals Coverage Status
Change from base Build 6639: 0.5%
Covered Lines: 3763
Relevant Lines: 5902

💛 - Coveralls
cristinaleonr commented 3 years ago

Opening smaller PR chunks