Process memory reporting uses Erlang memory estimates by default

[jbaublitz]

This bug report is for version 3.6.12 of rabbitmq-server/rabbitmq-common.

This information was gathered using rabbitmqctl eval on the same RabbitMQ test server that showed performance issues under load.

There seems to be an issue in vm_memory_monitor:get_process_memory() where it actually reports erlang:memory(total) if the config option vm_memory_calculation_strategy is unset. As soon as this config option is set, the get_process_memory() function begins to report consistently with get_ps_memory(). However rabbitmqctl status does report that vm_memory_calculation_strategy is set to rss even when unset which is the default behavior but not consistent with the memory reporting values. In my glance at the code, this seems to mean that it is falling back on Erlang due to a ps output parsing issue but cannot verify this due to the function not being exported to the main application. Please let me know if you need any more information or would like me to test anything else.

[michaelklishin]

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. This assumes two things:

1. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team)
2. We have a certain amount of information to work with

We get at least a dozen of questions through various venues every single day, often quite light on details. At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because of that questions, investigations, root cause analysis, discussions of potential features are all considered to be mailing list material by our team. Please post this to rabbitmq-users.

Getting all the details necessary to reproduce an issue, make a conclusion or even form a hypothesis about what's happening can take a fair amount of time. Our team is multiple orders of magnitude smaller than the RabbitMQ community. Please help others help you by providing a way to reproduce the behavior you're observing, or at least sharing as much relevant information as possible on the list:

• Server, client library and plugin (if applicable) versions used
• Server logs
• A code example or terminal transcript that can be used to reproduce
• Full exception stack traces (not a single line message)
• rabbitmqctl status (and, if possible, rabbitmqctl environment output)
• Other relevant things about the environment and workload, e.g. a traffic capture

Feel free to edit out hostnames and other potentially sensitive information.

When/if we have enough details and evidence we'd be happy to file a new issue.

Thank you.

[michaelklishin]

The defaults are set in rabbitmq-server (via the rabbit.vm_memory_calculation_strategy key). You haven't provided any specific steps to reproduce but I have a node that includes #227 running with mostly defaults and observe the following.

rabbit.vm_memory_calculation_strategy is what it is expected to be:

rabbitmqctl environment
 {rabbit,
      …
      {vm_memory_calculation_strategy,allocated}
      …

rabbitmqctl eval is not meant to be used for monitoring, so any side effects on it are considered to be a low priority for our team.

[michaelklishin]

Also:

rabbitmqctl eval "vm_memory_monitor:get_memory_calculation_strategy()."
allocated

So either #227 takes care of that on top of other things or I don't understand how to reproduce the issue.

[jbaublitz]

@michaelklishin I'm happy to provide steps to reproduce this but is that better suited for the mailing list? I'm a little confused because I was told to open an issue here.

[michaelklishin]

Yes. When/if we have enough details and conclude it is something that should be fixed, we will either update this issue or file a new one.

Our snapshot builds are currently not produced for unrelated reasons, so I don't think there is a snapshot that includes #227 but we can test the steps against a node running from source.

[gerhard]

@jbaublitz ping https://groups.google.com/d/msg/rabbitmq-users/Dw11uk9TPdc/-cgfkvTdAwAJ