search

mozilla-services / services-engineering

Services engineering core repo - Used for issues/docs/etc that don't obviously belong in another repo.
2 stars 1 forks source link

syncstorage-rs memory consumption #66

Open erkolson opened 3 years ago

erkolson commented 3 years ago

We are seeing syncstorage pods killed/evicted in production due to consuming too much memory and starving the host. It has been difficult to pin down when/why this is happening, as the memory utilization reported by monitoring stats is always below 75%. The application must be allocating a large amount of memory in a very short period of time, otherwise, we would see the 100% memory utilization in monitoring stats.

Is it possible to bound the memory footprint of rust applications?

Are there metrics available from the application concerning memory allocation?

erkolson commented 3 years ago

We've had 6 pods evicted in the last 12 hours. This time, the metrics show abrupt memory utilization around the time the pods get evicted:

Screen Shot 2020-08-19 at 7 43 13 AM