CLI tools using same JVM options as logstash itself

[dilchenko]

This results in attempts to install plugins (or using any other cmd tools that source logstash.lib.sh) to throw a 'cannot allocate memory' exception:

[root@logstash03.dal05 logstash]# ./bin/logstash-plugin update logstash-input-beats
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000640000000, 697892864, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 697892864 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/share/logstash/hs_err_pid3936.log

Proposed solution:

• pass custom LS_JAVA_OPTS from cli scripts

[dilchenko]

Even better option would be to not set the LS_JAVA_OPTS from lib.sh or have a flag for it, or another function in it - like setup_cli - that would avoid reading LS_JAVA_OPTS but do everything else.

Case in point: I am trying to automate installation of x-pack plugin. We are using chef, and it allows specifying environment variables for a bash resource. However, I believe logstash.lib.sh overrides whatever I set, and attempts to use LS_JAVA_OPTS, which generates the error message from description.

[kovyrin]

+1 on this. In almost any setup where there are dedicated instances/servers for running logstash, having logstash configured to use anything close to 50% of RAM would result in logstash cli being unusable.

[richard-mauri]

I encountered this also.

In our case, we have logstash jvm.options -Xmx12g -Xms12g, but on our host with 16g memory we get OOM running logstash-plugin.

The found the same jvm options are used by logstash and small cli tools.

I tested and recommend the small cli tools like logstash-plugin internally override like using LS_JAVA_OPTS="-Xms1g"

Do not burden the community by forcing them to develop,support wrapper scripts.

[yaauie]

I think the reason that this ticket hasn't seen much action, is that it is hard to predict how these configurations are being used "in the wild", which makes it hard to reason about changes we would propose and hard to ensure that a small change that is beneficial for some won't be catastrophic for others.

My personal way of handling this has been to export high memory values to LS_JAVA_OPTS in my service manager's configuration for the logstash service, overriding a low value set in the config/jvm.options. This would ensure that helper tools got the lower values from the config/jvm.options without any extra manual intervention, and that the service would get the higher memory limits requested by the service manager.

[richard-mauri]

Thank you for your response @yaauie My first reaction was "yes that would work" My second reaction was "How will elastic document this or will they let their customers fall into the trap and let them dig and ultimately find this post?"

Might theris be an optional, separate cli-jvm.options file that would be used by cli tools and fallback to using jvm.options if not available?