search

m-lab / scraper

Scrape experiment data off of MLab nodes and upload it to Google Cloud Storage
Apache License 2.0
5 stars 3 forks source link

node exporter needs more RAM #274

Closed stephen-soltesz closed 6 years ago

stephen-soltesz commented 6 years ago

It looks like the node exporter is OOM'ing

    Image:         prom/node-exporter:v0.13.0
    State:          Running
      Started:      Sun, 25 Mar 2018 12:15:23 -0400
    Last State:     Terminated
      Reason:       OOMKilled
      Started:      Tue, 13 Mar 2018 16:46:50 -0400
      Finished:     Sun, 25 Mar 2018 12:15:22 -0400
stephen-soltesz commented 6 years ago

This command extracts the lastState.teriminated.reason for all pods:

$ kubectl -n scraper get pods -o go-template="{{range .items}}{{(index .status.containerStatuses 0).lastState.terminated.reason}} {{.metadata.name}}:{{end}}" | tr ":" "\n"

2444 of 3043 have state OOMKilled.

stephen-soltesz commented 6 years ago

Should be fixed by: https://github.com/m-lab/scraper/pull/276