search

sustainable-computing-io / susql-operator

a Kubernetes operator that aggregates energy and CO2 emission data for tagged resources
http://susql.org
Apache License 2.0
10 stars 1 forks source link

Investigate periodic pod eviction #140

Open trent-s opened 3 weeks ago

trent-s commented 3 weeks ago

pods are occasionally ejected with an errors such as:

The node was low on resource: ephemeral-storage. Threshold quantity: 24068434671, available: 23201836Ki. Container kube-rbac-proxy was using 8Ki, request is 0, has larger consumption of ephemeral-storage. Container manager was using 106848Ki, request is 0, has larger consumption of ephemeral-storage.

and

The node was low on resource: ephemeral-storage. Threshold quantity: 32121498671, available: 30114200Ki. Container kube-rbac-proxy was using 4Ki, request is 0, has larger consumption of ephemeral-storage. Container manager was using 1920Ki, request is 0, has larger consumption of ephemeral-storage.
trent-s commented 3 weeks ago

But... This might be as simple as just reducing stdout logging. As we are in development, we are someone verbose at the moment...

trent-s commented 3 weeks ago

Also, possibly reduce sampling rate from 2 seconds to something a bit longer.... 30 seconds? 60 seconds?