search

sacloud / sakuracloud_exporter

Prometheus exporter for SakuraCloud metrics.
Apache License 2.0
7 stars 2 forks source link

Exporter is OOMkilled #18

Closed TakumaNakagame closed 5 years ago

TakumaNakagame commented 5 years ago

Exporter will be regularly OOMkilled. For Pod, resource.limits.memory is set to 100 MiB. Pod will periodically use up to 100MB of memory and will be killed.

If you check the memory usage of Pod in Prometheus, the memory usage continues to increase. And it was OOMkilled around 100 MiB.

Memory Usage Graph

container_memory_usage_bytes {container =" sakuracloud-exporter "}

image

Pod Details

kubectl get pod -o yaml sakuracloud-exporter

 - containerID: docker://a6e8c9743262f2b50bdfeefa8bca83778445fd28ffa21707e2825feaa40fb3f1
   image: sacloud/sakuracloud_exporter:0.6.0
   imageID: docker-pullable://sacloud/sakuracloud_exporter@sha256:9a6d5c8c0af0ba1aba692136f8f73baefba3a28e44287be5c07612784157579d
   lastState:
     terminated:
       containerID: docker://58ab86ade1cf59c4b5e01c130e8ee2f29ee27b77d43b484409868f7f1490f199
       exitCode: 137
       finishedAt: "2019-07-26T01:39:23Z"
       reason: OOMKilled
       startedAt: "2019-07-25T21:17:11Z"
yamamoto-febc commented 5 years ago

@TakumaNakagame

Thanks for reporting this. It seems like goroutine leak. I'll fix this ASAP.

TakumaNakagame commented 5 years ago

@yamamoto-febc , Thank you for the response. I see! understood. Please fix it. I am always saved!

yamamoto-febc commented 5 years ago

I merged #19. It includes the improvement of goroutines handling.

The following image is the comparison of number of goroutines before and after merging.

sakuracloud_exporter_goroutines_count

After merging, the number of leaked goroutines decreased. (and it probably reduces memory usage)

UPDATE

Latest master has some unfixed bugs. So please wait for a while until #20 is fixed.

yamamoto-febc commented 5 years ago

@TakumaNakagame

We released v0.7.0. https://github.com/sacloud/sakuracloud_exporter/releases/tag/0.7.0

It includes some improvements for handling leaked goroutines. Please try it.

TakumaNakagame commented 5 years ago

@yamamoto-febc

Thank you Release! I will use it immediately! It was very helpful!

TakumaNakagame commented 5 years ago

@yamamoto-febc

I tried it. Increase in memory is suppressed!

Thank you for the correction! This issue is okay as a close.

image

yamamoto-febc commented 5 years ago

I'm going to close this. Feel free to re-open if this problem happens again.