search

anycable / anycable-rails

AnyCable for Ruby on Rails applications
https://anycable.io
MIT License
495 stars 35 forks source link

RPC memory usage - pods OOMkilled with 4GB allocated per pod #177

Closed vitobotta closed 1 year ago

vitobotta commented 1 year ago

Tell us about your environment

Ruby version: 3.1.2

Rails version: 7.0.4

anycable gem version: 1.2.3

anycable-rails gem version: 1.3.3

grpc gem version: 1.45.0

What did you do?

I have been using AnyCable for a while already, but at the moment I am doing general load tests for our app with Locust.

What did you expect to happen?

RPC process to function correctly with 4GB of RAM allocated per process/pod (Kubernetes).

What actually happened?

The RPC processes get OOMKilled by Kubernetes very quickly and keep restarting, despite there isn't much activity really.

Can you advise on how much memory each RPC process is supposed to use? I think that 4GB of RAM are already a lot considering that it's a single Rails process, and I have tried lowering the threads from 40 to 20, with no success. The RPC processes get OOMKilled in a very short time from restarting, so it doesn't happen after a while. Seems like a memory leak or something like that.

Thanks in advance for your help!

Envek commented 1 year ago

That's weird. In my experience I've never seen RPC processes taking much memory, usually ≈300 MB almost constantly. Anyway, usually they only handles connects and disconnects.

Just in case, do you use plain AnyCable or maybe GraphQL with graphql-anycable? Any other plugins on top of ActionCable?

vitobotta commented 1 year ago

Hi @Envek , no I don't use any plugins etc. But you know what is really weird? After a while that behaviour stopped and I hadn't changed anything at all. No idea what happened but I guess we can close this one. Thanks!

palkan commented 1 year ago

For the record: we experienced similar problems a while ago, and it turned out to be a memory management problem in the google-protobuf gem: https://twitter.com/any_cable/status/1367761452740866048

So, I would suggest looking at their issue tracker for Ruby memory related issues (this one looks interesting, though need more time to investigate whether AnyCable's protobuf usage could be affected).

vitobotta commented 1 year ago

Thanks @palkan , if it happens again I will try a different version of that gem.