Update test runners to combine metadata and test data in test output

Murakami test runners currently use the default summary output of each test binary, but upstream tests don't necessarily format their output in basic formats suitable for ingestion into a database or conversion to CSV. The visualization service needs to either access test results directly or to ingest them into a database.

Because we want to add Murakami metadata (environment variables) to each test row, and to ensure that Murakami's data visualization service doesn't break by upstream dependencies on specific test binary formats, we will:

produce a spec for each test output that is a simple, non-nested, single JSON object
- [x] #32
- [x] #33
- [x] #34
add functions to each test runner that parse the tests' output and adds Murakami metadata values as columns in the test output to provide a simple, non-nested, single JSON object for each test
- [ ] #36
- [x] #37
- [x] #38

With these additions, the Murakami data vis service will be able to more easily parse the values needed for each test.

The only remaining work on this meta issue is the updates to the Murakami DASH test runner for our output format spec. in #32. This work will be completed once the DASH client itself has been updated by the upstream developer.

To recap so that our Murakami JSON specs are in this active issue, I've added the final specs below. @jheretic you can use these specs for the visualization service's processes that bring test data into the service database.

ndt5 Murakami JSON output specification

{
    "ServerName": "ndt-iupui-mlab1-iad03.measurement-lab.org",
    "ServerIP": "38.90.140.139"
    "ClientIP": "104.145.221.196",
    "MurakamiLocation": "<MURAKAMI 'LOCATION' ENV VAR>",
    "MurakamiNetworkType": "<MURAKAMI 'NETWORK_TYPE' ENV VAR>",
    "MurakamiConnectionType": "<MURAKAMI 'CONNECTION_TYPE' ENV VAR>",
    "TestUUID": "NULL",
        "TestProtocol": "ndt5",
        "TestError": "recvResultsAndLogout failed: cannot get message: read tcp 192.168.178.28:58788-\u003e77.67.115.11:3001: i/o timeout",
    "DownloadTestStartTime": "2020-02-18 20:32:52.639720903 +0000 UTC m=+5825.539350534",
    "DownloadTestEndTime": "2020-02-18 20:33:02.640686711 +0000 UTC m=+5835.540316330",
    "DownloadValue": "NULL", 
    "DownloadUnit": "Mbit/s", 
        "DownloadError": "download failed: cannot get TestPrepare message: read tcp 192.168.178.28:58788-\u003e77.67.115.11:3001: i/o timeout",
    "UploadValue": "NULL", 
    "UploadUnit": "Mbit/s", 
        "UploadError": "upload failed: cannot get TestMsg message: read tcp 192.168.178.28:58788-\u003e77.67.115.11:3001: i/o timeout",
    "DownloadRetransValue": "NULL", 
    "DownloadRetransUnit": "pct", 
    "MinRTTValue": "NULL",
    "MinRTTUnit": "ms"
}

ndt7 Murakami JSON output specification

Successful test:

{
  "SchemaVersion": 1,
  "TestName": "ndt7",
  "TestStartTime": "2020-02-25T17:02:40.918022",
  "TestEndTime": "2020-02-25T17:03:01.754734",
  "MurakamiLocation": "Corciano",
  "MurakamiConnectionType": "wifi",
  "MurakamiNetworkType": "home",
  "ServerName": "ndt-iupui-mlab3-mil04.measurement-lab.org",
  "ServerIP": "213.242.77.165",
  "ClientIP": "93.188.101.116",
  "DownloadUUID": "ndt-cz99j_1580820576_000000000003708B",
  "DownloadValue": 405.32368416128276,
  "DownloadUnit": "Mbit/s",
  "DownloadError": null,
  "UploadValue": 26.85925099645699,
  "UploadUnit": "Mbit/s",
  "UploadError": null,
  "DownloadRetransValue": 2.3442441873334583,
  "DownloadRetransUnit": "%",
  "MinRTTValue": 29.108,
  "MinRTTUnit": "ms"
}

Failed Test:

{
  "SchemaVersion": 1,
  "TestName": "ndt7",
  "TestStartTime": "2020-02-21T14:45:05.960649",
  "TestEndTime": "2020-02-21T14:45:05.965838",
  "MurakamiLocation": "Corciano",
  "MurakamiConnectionType": "wifi",
  "MurakamiNetworkType": "home",
  "DownloadError": "Get https://locate.measurementlab.net/ndt7: dial tcp: lookup locate.measurementlab.net: Temporary failure in name resolution",
  "UploadError": "Get https://locate.measurementlab.net/ndt7: dial tcp: lookup locate.measurementlab.net: Temporary failure in name resolution",
  "ServerName": null,
  "ServerIP": null,
  "ClientIP": null,
  "DownloadUUID": null,
  "DownloadValue": null,
  "DownloadUnit": null,
  "UploadValue": null,
  "UploadUnit": null,
  "DownloadRetransValue": null,
  "DownloadRetransUnit": null,
  "MinRTTValue": null,
  "MinRTTUnit": null
}

DASH Murakami JSON output specification

{
    "ServerIP": "194.116.85.211",     # "remote_address": "194.116.85.211",
    "ClientIP": "79.40.147.190",        #  "real_address": "79.40.147.190",
    "MurakamiLocation": "<MURAKAMI 'LOCATION' ENV VAR>",
    "MurakamiNetworkType": "<MURAKAMI 'NETWORK_TYPE' ENV VAR>",
    "MurakamiConnectionType": "<MURAKAMI 'CONNECTION_TYPE' ENV VAR>",    
    "TestUUID": "4364c5da-a07b-4188-9cbd-ad79e5a7c2f8",   
    "TestProtocol": "dash",
    "TestError": "NULL",
    "TestRuntime": 33.1672148704529,
    "TestRuntimeUnits": "seconds",
    "TestStartTime": "2017-07-24 09:44:53",
    "ProbeASN": "AS1234",
    "ProbeCC": "IT",
    "ConnectLatency": 0.0268912315368652,
    "ConnectLatencyUnits": "seconds" 
    "MedianBitrate": 5786.0,
    "MedianBitrateUnits": "Kbit/s",
    "MinPlayoutDelay": 0.214933633804321
    "MinPlayoutDelayUnits": "seconds"
}

speedtest-cli Murakami JSON output specification

{
  "TestName": "speedtest-cli-multi-stream",
  "TestStartTime": "2020-02-26T17:31:09.294212",
  "TestEndTime": "2020-02-26T17:31:32.438628",
  "MurakamiLocation": "<MURAKAMI 'LOCATION' ENV VAR>",
  "MurakamiNetworkType": "<MURAKAMI 'NETWORK_TYPE' ENV VAR>",
  "MurakamiConnectionType": "<MURAKAMI 'CONNECTION_TYPE' ENV VAR>",   
  "Download": 205429128.65829697,
  "DownloadUnits": "Bit/s",
  "Upload": 62332520.98624364,
  "UploadUnits": "Bit/s",
  "Ping": 20.556,
  "PingUnits": "ms",
  "BytesSent": 78004224,
  "BytesReceived": 258037852,
  "Share": null,
  "Timestamp": "2020-02-26T17:31:09.912341Z",
  "ServerURL": "http://speedtest.netandwork.net:8080/speedtest/upload.php",
  "ServerLat": "44.7709",
  "ServerLon": "10.7819",
  "ServerName": "Correggio",
  "ServerCountry": "Italy",
  "ServerCountryCode": "IT",
  "ServerSponsor": "NETandWORK s.r.l.",
  "ServerID": "20372",
  "ServerHost": "speedtest.netandwork.net:8080",
  "ServerDistance": 53.7642143944149,
  "ServerLatency": 20.556,
  "ServerLatencyUnits": "ms",
  "ClientIP": "93.188.101.116",
  "ClientLat": "44.4938",
  "ClientLon": "11.3387",
  "Isp": "Ehinet Srl",
  "IspRating": "3.7",
  "Rating": "0",
  "IspDownloadAvg": "0",
  "IspUploadAvg": "0",
  "LoggedIn": "0",
  "Country": "IT"
}

@critzo questions as I'm refining the validation schema for the test output in Murakami Viz:

Which fields are required for a test to be considered valid?
I assume MurakamiLocation, MurakamiNetworkType, and MurakamiConnectionType are arbitrary strings, but are other strings like TestName meant to be checked against a specific set of strings?
It seems like ndt7 and speedtest-cli use TestName and ndt5 & dash use TestProtocol. Is this accurate, or is only one of these correct?
Is there a uniform timestamp format? It seems like Timestamp can include any kind of date data.
There seems to be broadly overlapping use of ServerName, ServerHost, and ServerURL. In some kinds of test, ServerName seems to be a FQDN, and in some it seems to be a proper noun. Is there a consistent usage of these across different kinds of tests, or are they all in use interchangeably for different tests?

MurakamiLocation, MurakamiNetworkType and MurakamiConnectionType are arbitrary strings. They should just be validated to not contain unsafe strings.
TestName comes from each test runner's code. I'm not sure if we need to validate these against a specific set of strings. Since we would like to accommodate new test runners in the future, I would hesitate about adding validation for the current set of test runners.
For the other questions, @robertodauria could you provide more information for @jheretic ?

Hi, I had a look at the current Murakami outputs. It looks like they are pretty consistent, with a static set of fields -- the same for every test -- and a dynamic one depending on the specific working of each test.

The static fields are:

TestName: string
TestStartTime: string ('%Y-%m-%dT%H:%M:%S.%f')
TestEndTime: string ('%Y-%m-%dT%H:%M:%S.%f')
MurakamiLocation: string
MurakamiConnectionType: string
MurakamiNetworkType: string

The only outlier in the above examples is DASH, which has not been implemented yet and thus won't be used in Murakami in this phase.

I've also changed the speedtest (both multi and single-stream) outputs so that the fields containing the download and upload speed have the same name as ndt5/7:

DownloadValue
DownloadUnit
UploadValue
UploadUnit

Please don't consider those mandatory fields, as different test outputs in future (e.g. DASH) may not contain them.

Also, please note that the Ping field in speedtest-cli's output is comparable with MinRTT from ndt5 and ndt7 directly -- i.e., we can show them on the same graph -- but they're not the same, so I believe it would be wrong to rename Ping -> MinRTT, or MinRTT -> Ping.

cc @critzo @jheretic

m-lab / murakami

Update test runners to combine metadata and test data in test output #39