ES_HEAP_SIZE set to default on shell script all-in-one

using the easy-button I believe we are leaving the ES_HEAP_SIZE variable unset which defaults to 1GB.

On a machine with 16 GB of ram i was seeing this issue:

$ cif --otype fqdn -c 85 --limit 160000 -d -f csv > /dev/null
[2016-02-22T08:40:52,547Z][INFO][main:268]: starting up client...
[2016-02-22T08:40:52,547Z][INFO][main:303]: running search...
[2016-02-22T08:40:52,547Z][DEBUG][CIF::SDK::Client:170]: uri created: https://localhost/observables?confidence=85&gzip=1&otype=fqdn&limit=160000
[2016-02-22T08:40:52,548Z][DEBUG][CIF::SDK::Client:171]: making request...
[2016-02-22T08:40:53,024Z][INFO][CIF::SDK::Client:175]: status: 401
[2016-02-22T08:40:53,024Z][INFO][CIF::SDK::Client:181]: response size: < 1MB
[2016-02-22T08:40:53,024Z][DEBUG][CIF::SDK::Client:184]: decoding content..
[2016-02-22T08:40:53,024Z][DEBUG][CIF::SDK::Client:187]: decompressing...
[2016-02-22T08:40:53,024Z][DEBUG][CIF::SDK::Client:193]: Data input to gunzip is not in gzip format at /usr/local/share/perl/5.18.2/CIF/SDK/Client.pm line 189.
unauthorized at /usr/local/bin/cif line 310.

cif-router log

[2016-02-22T08:36:15,865Z][32181][ERROR]: [Request] ** [http://localhost:9200]-[500] {"error":"OutOfMemoryError[Java heap space]","status":500}, called from sub Search::Elasticsearch::Transport::try {...}  at /usr/share/perl5/Try/Tiny.pm line 81. With vars: {'status_code' => 500,'request' => {'path' => '/cif.observables-%2A/_search','mime_type' => 'application/json','ignore' => [],'body' => {'query' => {'filtered' => {'filter' => {'and' => [{'or' => [{'term' => {'group' => ['everyone']}}]},{'or' => [{'term' => {'otype' => ['fqdn']}}]},{'range' => {'confidence' => {'gte' => '85'}}}]}}},'sort' => [{'@timestamp' => {'order' => 'desc'}}]},'method' => 'GET','qs' => {'size' => 160000},'serialize' => 'std'}}

after setting the ES_HEAP_SIZE to 8GB, the heap size issue was gone.

head /etc/default/elasticsearch 
# Run Elasticsearch as this user ID and group ID
#ES_USER=elasticsearch
#ES_GROUP=elasticsearch

# Heap Size (defaults to 256m min, 1g max)
ES_HEAP_SIZE=8g

$ cif --otype fqdn -c 85 --limit 260000 -d -f csv > /dev/null
[2016-02-22T08:42:33,361Z][INFO][main:268]: starting up client...
[2016-02-22T08:42:33,361Z][INFO][main:303]: running search...
[2016-02-22T08:42:33,361Z][DEBUG][CIF::SDK::Client:170]: uri created: https://localhost/observables?confidence=85&otype=fqdn&limit=260000&gzip=1
[2016-02-22T08:42:33,361Z][DEBUG][CIF::SDK::Client:171]: making request...
[2016-02-22T08:43:05,546Z][INFO][CIF::SDK::Client:175]: status: 200
[2016-02-22T08:43:05,547Z][INFO][CIF::SDK::Client:181]: response size: 24.2860841751099MB
[2016-02-22T08:43:05,547Z][DEBUG][CIF::SDK::Client:184]: decoding content..
[2016-02-22T08:43:05,547Z][DEBUG][CIF::SDK::Client:187]: decompressing...
[2016-02-22T08:43:06,871Z][INFO][main:363]: search returned, formatting..
[2016-02-22T08:43:09,402Z][INFO][main:410]: done...

should we set the heap size to a large value using the easy_button? we've got some python code to dynamically determine a sane value.

...
if results:

    # turn result into an int
    mem_total_kilobytes = int(results[0])

    #1048576 kB = 1GB
    if mem_total_kilobytes < 1048576:
        # mem less than 1G of mem, set ES_HEAP_SIZE to 
        # 256mb
        print("256m")
    # if total mem between 1gb and 2gb, use 25% for ES        
    elif mem_total_kilobytes >= 1048576 and mem_total_kilobytes <= 2097152:
        calculated_value = ( int(round((mem_total_kilobytes * .25) / 1024)))
        print("{0}m".format(calculated_value))
    # if total mem between 2gb and 59.5 gb, use 50% for ES
    elif mem_total_kilobytes >= 2097153 and mem_total_kilobytes <= 62390272:
        calculated_value = ( int(round((mem_total_kilobytes * .50) / 1024)))
        print("{0}m".format(calculated_value))
    # if total mem over 59.5 GB, set max value at 30.5 GB
    # https://goo.gl/ilHkld
    else:
        print("30500m")
...

csirtgadgets / massive-octo-spice

ES_HEAP_SIZE set to default on shell script all-in-one #387