status-im / infra-status-legacy

Infrastructure for old Status fleet
https://github.com/status-im/nim-waku
1 stars 3 forks source link

Memory issue on prod fleet #7

Closed jakubgs closed 1 year ago

jakubgs commented 2 years ago

Prod nodes are running out of memory regularly recently:

image

https://grafana.infra.status.im/d/QCTZ8-Vmk/single-host-dashboard?orgId=1&var-host=node-02.do-ams3.status.prod&from=1652555352670&to=1653592152670

This is most probably related to the recent changes in store capacity:

It appears even 200k is too much considering everything is loaded into memory, and there's a memory leak:

jakubgs commented 2 years ago

I'm going to attempt to lower the store capacity to 100k to see if that makes a difference: https://github.com/status-im/infra-status/commit/dd76f2c2

jakubgs commented 1 year ago

I don't believe this is relevant any longer:

image