Ambassador memory grows indefinitely, causes config throttling

[etotten]

Describe the bug We have multiple environments running Ambassador with the ConsulResolver sending to a Consul Connect mesh. In each of these environments, we see that Ambassador's memory appears to grow indefinitely until it gets to about 98-99%, where it hovers. While this might be acceptable alone, it causes a behavior in Ambassador where it starts throttling config changes, showing error messages like this:

time="2021-07-30T08:26:04Z" level=warning msg="Memory Usage: throttling reconfig v14739 due to constrained memory with 30 stale reconfigs (30 max)"

This throttling coincides with clients getting lots of 503 responses from Ambassador when they try to access services due to unhealthy upstreams.

We understand that the throttling is working as intended, but the issue here is that the memory leak combined with that throttling behavior causes much instability with requests.

Here's an example of mem usage and mem % in one of these environments:

To Reproduce

• Memory leak:
  • Deploy Ambassador 1.13.0 using the ConsulResolver with a Consul Connect Mesh
  • Wait a while and see the pod's memory grow; wait a few days and see that continues to grow fairly linearly
• Config Throttling under high memory:
  • Deploy Ambassador 1.13.0 using the ConsulResolver with a Consul Connect Mesh, having a test app with a Mapping and having pods in upstream
  • terminate a test app pod, see there are no issues with getting 503's
  • Simulate the memory leak by using a python shell to waste pod memory:
  • hop onto Ambassador pod, run "python", enter:
  • foo = []
  • foo.append(['xxxxxxxxxx' for i in range(1, 10000000)]) # repeat and watch pod memory usage go up
  • terminate a pod
  • See the "throttling reconfig" log messages happening
  • see that clients get 503's on some of the requests to the test app
  • terminate the rest of the original test app pods
  • see that clients get all 503's on requests to the test app

Versions (please complete the following information):

• Ambassador: 1.13.0
• Kubernetes environment: AWS EKS
• Version 1.18
• Consul Connect: 1.9.3

Additional Context Upon inspection, we can see that under these circumstances Ambassador does not keep up with changes to upstream pods. Specifically, if I hop onto an ambassador pod and run:

curl -s http://127.0.0.1:8001/clusters | grep

...we can see the IPs for pods in the upstream, but when we terminate a pod under throttling conditions, we can see that the IP of the terminated pod lingers in the cluster, increasing latency (when using retries) and 503's soon after.

We noticed that there is a new feature in 1.13.10, AMBASSADOR_AMBEX_NO_RATELIMIT, which can suppress the throttling behavior; however we are having a different, eventual cluster-upstreams issue using that...see: https://github.com/emissary-ingress/emissary/issues/3680

We have recently given the pods more memory to create a longer period of time before the config throttling begins; however, the memory keeps growing and it will just be matter of time before the config throttling happens again.

[etotten]

In order to get some sense of which processes are growing, I did take these two snapshots in a test environment about 28 hours apart. It looks as if the "busyambassador entrypoint" process may be the one that is growing.

[etotten]

There is at least one other who sees this problem as well - sounds like it is still something they're working-around: https://github.com/emissary-ingress/emissary/issues/3414#issuecomment-896437505

[juanjoku]

Hi, we also observe an increase in memory usage with Ambassador v1.13.10 (did not occur with v1.11.1).

We use Kubernetes Endpoint Resolver, not Consul. The increase occurs on an Ambassador that is receiving minimal traffic, and is not being reconfigured (the mappings do not change).

For example, last two days:

[ppeble]

Just to bump, I believe this might be happening to us as well. We just noticed the Memory Usage: throttling reconfig error messages with high memory usage. We are running 1.14.1.

We are running an istio mesh, not using consul.

[dwgillies-bluescape]

We are seeing the same issue @Bluescape, in our alphatest cluster (50+ users, 1.13.3, running istio). We have 6 ambassador pods, agent, and redis. Agent grows without bounds and crashes, often.

[kareem-elsayed]

We are also facing this issue with Emissary v2.3.1 and noticed that memory consumption increased almost double more than the old version of Ambassador v1.7.4 We tried to follow recommendations on documentation but we didn't find a more significant effect on resources consumption