Dead references to LS_HEAP_SIZE

[jarpy]

Some references to the LS_HEAP_SIZE environment variable remain in the repo (particularly in the docs), but the variable looks to have been deprecated in favour of LS_JAVA_OPTS.

This may have to led to some user confusion. See: https://github.com/elastic/logstash-docker/issues/27

[nicknameforever]

+1 It is not only confusing. It leads to serious issues (OutOfMemory exceptions) in migration from Logstash 2 to Logstash 5, because the documented option (LS_HEAP_SIZE) does not work anymore.

[colinsurprenant]

Thanks @jarpy for the heads up. Two issues here:

• correct the docs /cc @dedemorton
• fix the Windows scripts to get rid of LS_HEAP_SIZE to be consistent with Linux startup scripts.

[colinsurprenant]

created issue #7567 for the Windows startup script, will keep this one specifically for the docs inconsistencies.

[dedemorton]

@colinsurprenant Can you check the following references and confirm that replacing the name is all that's required?

https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html The pipeline.batch.size setting defines the maximum number of events an individual worker thread collects before attempting to execute filters and outputs. Larger batch sizes are generally more efficient, but increase memory overhead. Some hardware configurations require you to increase JVM heap size by setting the LS_HEAP_SIZE variable to avoid performance degradation with this option. Values of this parameter in excess of the optimum range cause performance degradation due to frequent garbage collection or JVM crashes related to out-of-memory exceptions. Output plugins can process each batch as a logical unit. The Elasticsearch output, for example, issues https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[bulk requests] for each batch received. Tuning the pipeline.batch.size setting adjusts the size of bulk requests sent to Elasticsearch.

Also in that topic:

The total number of inflight events is determined by the product of the pipeline.workers and pipeline.batch.size settings. This product is referred to as the inflight count. Keep the value of the inflight count in mind as you adjust the pipeline.workers and pipeline.batch.size settings. Pipelines that intermittently receive large events at irregular intervals require sufficient memory to handle these spikes. Configure the LS_HEAP_SIZE variable accordingly.

https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html

-b, --pipeline.batch.size SIZE*:: Size of batches the pipeline is to work in. This option defines the maximum number of events an individual worker thread will collect from inputs before attempting to execute its filters and outputs. The default is 125 events. Larger batch sizes are generally more efficient, but come at the cost of increased memory overhead. You may have to increase the JVM heap size by setting the LS_HEAP_SIZE variable to effectively use the option.

https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html pipeline.batch.size The maximum number of events an individual worker thread will collect from inputs before attempting to execute its filters and outputs. Larger batch sizes are generally more efficient, but come at the cost of increased memory overhead. You may have to increase the JVM heap size by setting the LS_HEAP_SIZE variable to effectively use the option. 125

[colinsurprenant]

I suggest we wait for #7727 and simply link to this topic to replace references to LS_HEAP_SIZE

[dedemorton]

@karenzone #7727 hasn't been resolved yet. I keep pinging people about it, but haven't gotten an answer. This is pretty old, so I'd get this sorted out one way or another soonish.

[karenzone]

@colinsurprenant If we can remove user confusion by replacing the name (LS_HEAP_SIZE -> LS_JAVA_OPTS) in the Reference Guide, I'd like to do that as a stop gap while we get it together on 7727. (See DeDe's notes/questions in an earlier comment.)

[colinsurprenant]

As per our discussion, each mention of LS_HEAP_SIZE can be replaced by something like

[...] by changing the related option in the jvm.options config file [...]

and add a link to https://www.elastic.co/guide/en/logstash/6.2/config-setting-files.html for now and once we have #7727 then we can link there.

[karenzone]

Fixed with #9335