memory needed > 16GB normal?

[FriedrichKayak]

Thank you for putting this image together; it's a great help in my experimentation with the ELK stack.

I am still very new to Docker- and somewhat new to Linux sysadmin tasks, overall. I am experiencing a problem of high memory consumption running the container and I wonder if this is normal, something I am doing wrong, or something else?

My Docker host is a Debian VM on a Hyper-V host. The VM is assigned 16GB of RAM. After adjusting the vm.max_map_count=262144 parameter, I was able to start the container without incident. I can successfully load the Elastic dashboard on :5601.

When I try to add the dummy entry per the walkthrough- the container can't start the OpenJDK Java VM due to memory constraints:

Using bundled JDK: /opt/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d4cc0000, 724828160, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory.
# An error report file with more information is saved as:
# //hs_err_pid497.log

Does this seem normal or do you think I need to do a better job configuring the host's memory use? Or, perhaps should I reduce the memory consumption expected by the Java VM?

If this is normal and expected, what amount of memory would be reasonable? This makes me feel like I am doing something wrong because I run Microsoft SQL Server with only 16GB or RAM, and I would normally assume that Microsoft's product would be much less efficient and more needy than this stack. :)

[FriedrichKayak]

I should have mentioned that the above mmap failure occurs when executing:

# /opt/logstash/bin/logstash --path.data /tmp/logstash/data \
    -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

[spujadas]

Thanks for the kind words – appreciated!

Your host set-up looks good to me, and you’re right 16GB is more than enough to run the ELK stack ;)

You might want to tweak the resources allocated to Docker itself in the Settings > Resources > Advanced section of Docker Desktop for Windows, as described here: https://docs.docker.com/docker-for-windows/#:~:text=Memory%3A%20By%20default%2C%20Docker%20Desktop,swap%20file%20size%20as%20needed

Not using Hyper-V myself, but if the screenshot is anything to go by, then the low default of 2GB (vs a more comfortable 4GB) may explain the issue.

[FriedrichKayak]

My Docker host is Debian (on top of Hyper-V), so I'm not actually using Docker Desktop for Windows. However, your suggestion was helpful because I looked at the memory allocated to the container in docker stats and it was only a few hundred megabytes (despite more being available on the host).

I used the --memory=10g argument to start the container with more memory allocated. When it all settled down, it reported using 7.4g of the 10g specified. Thereafter, the Java VM started with no further problems.

Thanks for pointing me in the right direction!

[spujadas]

Oh apologies, misread your set-up, but glad the suggestion helped you solve the issue anyway! :smile: