dmioscope: allow data logging in JSON and CSV formats

[bmr-cymru]

The dmioscope script should allow logging of histogram data points, in order to do more thorough analysis, or more sophisticated plotting, after a run has finished.

E.g.:

# dmioscope -nb 8 --output="data.csv"

Since someone is bound to want data in JSON format today, we should also support:

# dmioscope -nb 8 --output="data.json"

This can be achieved by handing an IOScope instance a log writer class that accepts an array of data points and writes them into the appropriately formatted output file.

In case data is going to a pipe, a --format may be required:

# dmioscope -nb 8 --output - --format csv

Suggested data formats for CSV and JSON:

CSV has a header row with N=nr_bins width values, followed by M repetitions of data rows:

0,0,<width1>,<width2>,...<widthN>\n
<iteration>,<timestamp>,<data1>,<data2>,...<dataN>\n
...

{
    "start_time": time_stamp,
    "nr_bins": number,
    widths: [ width1, width2, ... , widthN ],
    values: [
       [ data1, data2, ..., dataN ],
       [ data1, data2, ..., dataN ]
    ]
}

E.g.:

{
    "start_time": 1475149102,
    "nr_bins": 4,
    "widths": [ 8192, 8192, 8192, 8192 ],
    "values": [
        [ 0, 567, 80, 1112],
        [10, 400, 23, 987]
    ]
}

Adaptive histograms may need further thought: emitting a new header row, or starting a new JSON object, will correctly set the new width parameters, but may not produce data in a form easily used for plotting.

[bmr-cymru]

JSON output is implemented using JSON Time Seried notation.

This is very similar in structure to the proposed dmioscope JSON format:

{
    "docType": "jts",
    "version": "1.0",
    "header": {
        "startTime": "2014-08-16T02:00:00.000Z",
        "endTime": "2014-08-16T02:20:43.000Z",
        "recordCount": 5,
        "columns": {
            "0": {
                "id": "541a5a129bc9b4035f906d70",
                "name": "Temperature",
                "dataType": "NUMBER",
                "renderType": "VALUE",
                "format": "0.###",
                "aggregate": "NONE"
            }
        }
    },
    "data": [
        {
            "ts": "2014-08-16T02:00:39.000Z",
            "f": { "0": {"v": 28.21, "q": 100 } }
        },
        {
            "ts": "2014-08-16T02:05:40.000Z",
            "f": { "0": {"v": 28.22 } }
        },
        {
            "ts": "2014-08-16T02:10:41.000Z",
            "f": { "0": {"v": 28.7 } }
        },
        {
            "ts": "2014-08-16T02:15:42.000Z",
            "f": { "0": {"v": 29.2, "q": 100 } }
        },
        {
            "ts": "2014-08-16T02:20:43.000Z",
            "f": { "0": {"v": 29.18 } }
        }
    ]
}