search

mercari / tortoise

Tortoise: Shell-Shockingly-Good Kubernetes Autoscaling
MIT License
316 stars 14 forks source link

more appropriate GOMEMLIMIT modification #386

Open sanposhiho opened 3 months ago

sanposhiho commented 3 months ago

What happened seemingly

When Tortoise is configured to vertically scale up/down resource, Tortoise basically calculates recommended resource requests by {VPA's recommendation} *1.1 (1.1 is configurable through BufferRatioOnVerticalResource)

Also, Tortoise keeps the ratio of GOMEMLIMIT to memory request. It means, if GOMEMLIMIT is configured to be 80% of the memory request right now, tortoise keeps the ratio even after it changes the memory request.

GOMEMLIMIT is a soft limit, it means, obviously, "GOMEMLIMIT: 1MiB" doesn't mean the memory consumption does never go beyond 1MiB. Based on this fact, one service encounters this issue seemingly:

  1. GOMEMLIMIT is configured around 90% of memory request. Meaning the memory consumption would be always around 90%+.
  2. VPA suggests nearly the current memory request.
  3. Tortoise calculates the recommendation by VPA's value * 1.1. Because of (2), it increases the memory request a bit.
  4. go back to (1) and never stops.

What Tortoise should do instead

We don't want to stop multiplying 1.1 to VPA's value for safety. We can just stop modifying GOMEMLIMIT, at least for now.

The downside to do that would be that, if the application is changed and starts to consume memory a lot more, GC runs more often and it negatively impacts the CPU consumption.

sanposhiho commented 3 months ago

The quick patch is done with https://github.com/mercari/tortoise/pull/387.