hashicorp / consul-esm

External service monitoring for Consul
Mozilla Public License 2.0
262 stars 41 forks source link

Memory Leak in ESM 0.7.X #257

Open reskin89 opened 6 months ago

reskin89 commented 6 months ago

I've tested both 0.7.0 as well as 0.7.1, and I tested them as I was experiencing the issue in 0.6.2.

It seems ESM slowly eats memory until it crashes. The duration it takes depends on the amount of external services.

I am running ESM in ECS Fargate (I just installed ESM on a base alpine container), and over the course of a week, with 1 external service (using nick jackson to test, with a 50% error rate so that ESM has to update its health status) memory utilization went from 20% to start, to now 98%.

I tested this initially with no external services, memory remained stable, I then put a test service in (a nick jackson container in fargate behind an ALB) with no ERROR_RATE, and it was fine, but as soon as ESM has to start modifying the status of external services in the consul catalog, memory usage slowly climbs.

This issue exists in: