(MAINT) handle large stream frame payloads which are split across multiple Reads

[johnmccabe]

Prior to this PR the FilterDockerStream function wasn't handling large Docker stream frames where a single Read wasn't returning the whole buffer (as observed in #20).

This commit switches to using io.LimitReader with ioutil.ReadAll to read the entire frame payload.

It refactors FilterDockerStream adding local functions for reading the frame headers and payloads.

It also includes some minor log message tweaks in the apk, dpkg and rpm capabilities.

[johnmccabe]

mccabe@john:~/workspaces/go/src/github.com/puppetlabs/lumogon% docker ps | grep jenkins
edb234bb2cb8        jenkinsci/workflow-demo            "/bin/tini -- /usr..."   10 hours ago        Up 10 hours         8080-8081/tcp, 9418/tcp, 50000/tcp                                                   jenkins

mccabe@john:~/workspaces/go/src/github.com/puppetlabs/lumogon% docker run --rm -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan jenkins | jq
{
  "$schema": "http://puppet.com/lumogon/core/draft-01/schema#1",
  "generated": "2017-05-15 23:19:45.86572164 +0000 UTC",
  "owner": "default",
  "group": [
    "default"
  ],
  "client_version": {
    "BuildVersion": "development",
    "BuildTime": "2017-05-15 11:15:19 UTC",
    "BuildSHA": "41f2b6a081d22e5a99a920a415855ce426756184"
  },
  "reportid": "6acb40f1-7f11-4d11-a8fd-a5f61484260c",
  "containers": {
    "edb234bb2cb8fd72f520148c8099f4381b9e762c563eda23260925cdb46fd546": {
      "$schema": "http://puppet.com/lumogon/containerreport/draft-01/schema#1",
      "generated": "2017-05-15 23:19:45.012074417 +0000 UTC",
      "container_report_id": "b9722ed4-84fe-4448-b807-96a242164b93",
      "container_id": "edb234bb2cb8fd72f520148c8099f4381b9e762c563eda23260925cdb46fd546",
      "container_name": "/jenkins",
      "capabilities": {
        "dpkg": {
          "$schema": "http://puppet.com/lumogon/capability/label/draft-01/schema#1",
          "title": "Packages (DPKG)",
          "type": "dockerapi",
          "harvestid": "03cc9082-7f8a-46a0-87b2-431601c1bfbf",
          "payload": {
            "acl": "2.2.52-2",
            "adduser": "3.113+nmu3",
            "apt": "1.0.9.8.4",
            "base-files": "8+deb8u7",
            "base-passwd": "3.5.37",
... snip ...
            "xtrans-dev": "1.3.4-1",
            "xz-utils": "5.1.1alpha+20120614-2+b3",
            "zlib1g": "1:1.2.8.dfsg-2+b1"
          }
        },
        "host": {
          "$schema": "http://puppet.com/lumogon/capability/host/draft-01/schema#1",
          "title": "Host Information",
          "type": "attached",
          "harvestid": "7572c08d-d04b-473a-b442-e7a82648dca1",
          "payload": {
            "hostname": "1c32255362be",
            "os": "linux",
            "procs": "0",
            "uptime": "0"
          }
        },
        "label": {
          "$schema": "http://puppet.com/lumogon/capability/label/draft-01/schema#1",
          "title": "Labels",
          "type": "dockerapi",
          "harvestid": "e96d614f-15ef-4aef-825f-da686ea4dc1b"
        }
      }
    }
  }
}

[rick]

Ran a few reports through, with debugging on and off and things look :+1:

[johnmccabe]

@rick I've pushed some updates above, could you have a look and let me know what you think

[rick]

Other than one comment on a comment, :+1:

[johnmccabe]

@rick don't merge !! hold on I need to check something

[johnmccabe]

@rick ok, its fine, I'd scared myself with the test logging in Read error tests - the TimeoutReader is based on ioutil.TimeoutReader but lets you specify the number of successful reads before returning an error of "timeout" which I wondering is maybe misleading? Its what ioutil.TimeoutReader returns..