Purpose of LOGSTASH_HEAP_SIZE

[sadovnikov]

What is the purpose of LOGSTASH_HEAP_SIZE or --logstash.heapSize?

• this parameter is being processed by the process (or container), which is already started. Therefore it does not influence resources, which are allocated to the scheduler process
• in the code LOGSTASH_HEAP_SIZE is being summed with EXECUTOR_HEAP_SIZE as requirement for executor.

    private boolean isNotEnoughRAM(ClusterState clusterState, Protos.Offer offer) {
        return !hasEnoughOfResourceType(offer.getResourcesList(), "mem", executorConfig.getHeapSize() + logstashConfig.getHeapSize() + executorConfig.getOverheadMem());
    }

Isn't it confusing? Should we remove this parameter?

[jhftrifork]

Setting --logstash.heapSize=H means we set environment variable LS_HEAP_SIZE=Hm passed to the Logstash process. This LS_HEAP_SIZE is undocumented, but you can see here that it's used to set the -Xmx in the JAVA_OPTS for the Logstash process.

So --logstash.heapSize is the maximum size, in megabytes, of the memory allocation pool of the Logstash process.

However, it's unclear to me why this should be a user-configurable option. We the developers should know what the heap size limit should be.

[sadovnikov]

OK!! Now it actually makes sense!!

I got confused because there is also --executor.heap-size and they used somehow together to determining resource requirements of the executor. And there are FIXME comment in README.md.

Could you please update the description?

[mwl]

@sadovnikov Does #106 make it clearer?

[jhftrifork]

@mwl I think so - I basically copied over the description above into the README

[sadovnikov]

106 does make it clearer

[mwl]

Thanks @sadovnikov. #106 has been merged already.