search

cirruslabs / cirrus-ci-agent

Agent to execute Cirrus CI tasks
Mozilla Public License 2.0
13 stars 6 forks source link

Executor: log short stats about received metrics #305

Closed edigaryev closed 1 year ago

edigaryev commented 1 year ago

I was able to reproduce https://github.com/cirruslabs/cirrus-ci-docs/issues/1217 but the results are still seemingly impossible if you look at the agent's code: we should either get a data point for both CPU or memory, or an error that should be logged.

In case of getting context.Canceled or context.DeadlineExceeded from the underlying CPU/memory source, we'd stop collecting metrics, which is not the case, because the memory is collected just fine.

Another interesting detail is that I was only able to reproduce this when using matrix:

  freebsd_instance:
    matrix:
      image_family: freebsd-12-4-snap
      image_family: freebsd-13-2-snap

Using only freebsd-12-4-snap or freebsd-13-2-snap results in a correct CPU graph.

The data points in the API look like this:

{
  "data": {
    "task": {
      "executionInfo": {
        "agentNotifications": [],
        "events": [],
        "cpuChart": {
          "minValue": 0,
          "maxValue": 3.2,
          "points": [
            {
              "value": 0,
              "secondsFromStart": 0
            },
            {
              "value": 0.64,
              "secondsFromStart": 60
            },
            {
              "value": 1.56,
              "secondsFromStart": 120
            },
            {
              "value": 3.2,
              "secondsFromStart": 180
            }
          ]
        },
        "memoryChart": {
          "minValue": 788.921875,
          "maxValue": 2760.77734375,
          "points": [
            {
              "value": 788.921875,
              "secondsFromStart": 1
            },
            {
              "value": 838.9609375,
              "secondsFromStart": 2
            },
            {
              "value": 890.20703125,
              "secondsFromStart": 3
            },
            {
              "value": 906.87109375,
              "secondsFromStart": 4
            },
            {
              "value": 962.3828125,
              "secondsFromStart": 5
            },
            {
              "value": 993.70703125,
              "secondsFromStart": 6
            },
            {
              "value": 1076.0859375,
              "secondsFromStart": 7
            },
            [...]
            {
              "value": 2276.6875,
              "secondsFromStart": 250
            },
            {
              "value": 2231.41796875,
              "secondsFromStart": 260
            },
            {
              "value": 2214.5,
              "secondsFromStart": 270
            },
            {
              "value": 2285.1484375,
              "secondsFromStart": 280
            }
          ]
        }
      }
    }
  }
}

Also bump github.com/shirou/gopsutil just in case.