gardener / autoscaler

Customised fork of cluster-autoscaler to support machine-controller-manager
Apache License 2.0
16 stars 25 forks source link

Add logging in case we have minimum memory recommendations #299

Closed voelzmo closed 8 months ago

voelzmo commented 8 months ago

What this PR does / why we need it: Add logging in case we have minimum memory recommendations. This happens from time to time and we need more information to understand why. This gives us all the internal recommender state, such that we can derive from there, how to fix this.

This is what the debug output currently looks like

W0311 09:54:11.860225       1 estimator.go:157] Computed resources for memory were below minimum! Computed 5886, minimum is 262144000.
W0311 09:54:11.860234       1 estimator.go:180] Here's the checkpoint/state: {
  "lastUpdateTime": "2024-03-11T09:54:11Z",
  "version": "v3",
  "cpuHistogram": {
    "referenceTimestamp": "2024-03-11T00:00:00Z",
    "bucketWeights": {
      "26": 10000
    },
    "totalWeight": 0.26620445628136635
  },
  "memoryHistogram": {
    "referenceTimestamp": "2024-03-12T00:00:00Z",
    "bucketWeights": {
      "0": 10000
    },
    "totalWeight": 2.6620445628136635
  },
  "firstSampleStart": "2024-03-11T09:54:02Z",
  "lastSampleStart": "2024-03-11T09:54:04Z",
  "totalSamplesCount": 2
}
W0311 09:54:11.860235       1 estimator.go:157] Computed resources for cpu were below minimum! Computed 0, minimum is 25.
W0311 09:54:11.860241       1 estimator.go:180] Here's the checkpoint/state: {
  "lastUpdateTime": "2024-03-11T09:54:11Z",
  "version": "v3",
  "cpuHistogram": {
    "referenceTimestamp": "2024-03-11T00:00:00Z",
    "bucketWeights": {
      "26": 10000
    },
    "totalWeight": 0.26620445628136635
  },
  "memoryHistogram": {
    "referenceTimestamp": "2024-03-12T00:00:00Z",
    "bucketWeights": {
      "0": 10000
    },
    "totalWeight": 2.6620445628136635
  },
  "firstSampleStart": "2024-03-11T09:54:02Z",
  "lastSampleStart": "2024-03-11T09:54:04Z",
  "totalSamplesCount": 2
}

Release note:

Add logging in case we have minimum memory recommendations
voelzmo commented 8 months ago

/invite @andrerun

gardener-robot commented 8 months ago

@voelzmo Command "/invite @andrerun" failed with "Reviews may only be requested from collaborators. One or more of the users or teams you specified is not a collaborator of the gardener/autoscaler repository.".

Additional Information ``` Redacted in public. Check backend logs. ```