AlmightyOatmeal
commented
7 years ago I missed this:
2017-07-13T20:57:04.829726351Z # Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007fd36b24c700My bad.
Closed AlmightyOatmeal closed 5 years ago
AlmightyOatmeal
commented
7 years ago I missed this:
2017-07-13T20:57:04.829726351Z # Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007fd36b24c700My bad.
AlmightyOatmeal
commented
7 years ago After upgrading the host to 2G of ram, I'm still getting:
2017-07-13T21:05:28.702447206Z OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)
2017-07-13T21:05:28.702597083Z #
2017-07-13T21:05:28.702608733Z # There is insufficient memory for the Java Runtime Environment to continue.
2017-07-13T21:05:28.702613883Z # Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
2017-07-13T21:05:28.702618903Z # Can not save log file, dump to screen..
2017-07-13T21:05:28.702623436Z #
2017-07-13T21:05:28.702674430Z # There is insufficient memory for the Java Runtime Environment to continue.
2017-07-13T21:05:28.702679850Z # Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
2017-07-13T21:05:28.702684595Z # Possible reasons:
2017-07-13T21:05:28.702689081Z # The system is out of physical RAM or swap space
2017-07-13T21:05:28.702693410Z # In 32 bit mode, the process size limit was hit
2017-07-13T21:05:28.702697710Z # Possible solutions:
2017-07-13T21:05:28.702701813Z # Reduce memory load on the system
2017-07-13T21:05:28.702706255Z # Increase physical memory or swap space
2017-07-13T21:05:28.702710623Z # Check if swap backing store is full
2017-07-13T21:05:28.702715011Z # Use 64 bit Java on a 64 bit OS
2017-07-13T21:05:28.702719305Z # Decrease Java heap size (-Xmx/-Xms)
2017-07-13T21:05:28.702723623Z # Decrease number of Java threads
2017-07-13T21:05:28.702727793Z # Decrease Java thread stack sizes (-Xss)
2017-07-13T21:05:28.702732043Z # Set larger code cache with -XX:ReservedCodeCacheSize=
2017-07-13T21:05:28.702736248Z # This output file may be truncated or incomplete.
2017-07-13T21:05:28.702740550Z #
2017-07-13T21:05:28.702744583Z # Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007fdbc7c7e700
2017-07-13T21:05:28.702748983Z #
2017-07-13T21:05:28.702752973Z # JRE version: (8.0_121-b13) (build )
2017-07-13T21:05:28.702757186Z # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
2017-07-13T21:05:28.702761608Z # Core dump written. Default location: /usr/share/elasticsearch/core or core.1
2017-07-13T21:05:28.702795093Z #Do you have advice on configuring the JVM memory utilization arguments?
josegonzalez
commented
7 years ago Do you have swap enabled, and is anything else using the memory on that box?
Seems its trying to use all your ram...
AlmightyOatmeal
commented
7 years ago @josegonzalez,
I do not have swap enabled.
This is a brand new Dokku droplet running on Digital Ocean and I've resized twice now to give it more resources. I'm not sure how I can use this plugin to pass in the ES_JAVA_OPTS environmental variable because this didn't work for me:
# export ELASTICSEARCH_IMAGE="elasticsearch"
# export ELASTICSEARCH_IMAGE_VERSION="5.3.0"
# export ELASTICSEARCH_CUSTOM_ENV='ES_JAVA_OPTS="-Xms512m -Xmx512m"'
# dokku elasticsearch:create vonavi-dokku-01Because I still get:
2017-07-13T21:20:05.709147745Z OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)So the JVM is trying to allocate more than is available and not the amount specified in ES_JAVA_OPTS.
aaronshim
commented
7 years ago @AlmightyOatmeal @josegonzalez
This is also happening for me, but on Elasticsearch 5.4.3, also running on a clean 2GB Digital Ocean droplet. Were you guys ever able to find a way to limit the ES_JAVA_OPTS? Thank you for your help!
haggen
commented
7 years ago Mine is doing the same thing (restarting over and over) but here the logs keep repeating this:
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ElasticsearchParseException[duplicate settings key [network.host] found at line number [96], column number [15], previous value [0.0.0.0], current value [0.0.0.0]];
Likely root cause: ElasticsearchParseException[duplicate settings key [network.host] found at line number [96], column number [15], previous value [0.0.0.0], current value [0.0.0.0]]
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeValue(XContentSettingsLoader.java:151)
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:109)
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)
at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)
at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:226)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
rabbagliettiandrea
commented
6 years ago Same here (restarting and counting..) but with a different error:
=====> Copying config files into place
Container command retried 100 time(s): cp -arfp --no-clobber /etc/elasticsearch/. /usr/share/elasticsearch/config/
Failed to run command: cp -arfp --no-clobber /etc/elasticsearch/. /usr/share/elasticsearch/config/We don't yet support 5.x. If someone would like to sponsor my work to implement that, please reach out to me.
pnikrat
commented
6 years ago The problem with container stuck at restarting can be solved by changing the vm.max_map_count variable as hinted here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode
Of course you should remove the container that is stuck and recreate it after the vm.max_map_count change:
# export ELASTICSEARCH_IMAGE_VERSION="5.6.5"
# dokku elasticsearch:create somename
EvgenOleynikov
commented
6 years ago @pnikrat tnx! Work for me
archy-bold
commented
6 years ago I managed to solve the memory issue by updating the options in the jvm.options file in the configuration folder for the service. You can find that by running the following command:
dokku elasticsearch:info SERVICE_NAME --config-dirThen just update the -Xmx and -Xms settings from:
-Xms2g
-Xmx2gto
-Xms512m
-Xmx512mNote: while you can set the environment variable for ES_JAVA_OPTS as @AlmightyOatmeal suggested, the quotes are added to the command, which causes a different error. Updating the options file will mean you don't need to set the environment variable every time.
As for @haggen's error about a duplicate settings key [network.host]. I think on starting the service, the plugin must write the network.host setting to the elasticsearch.yml config file in the same directory as jvm.options. You just need to remove the lines network.host: 0.0.0.0 from the bottom of the file and restart, that should allow it to start up. I think that must be a bug.
josegonzalez
commented
6 years ago Ah they fixed it so that it listens on 0.0.0.0 now?
archy-bold
commented
6 years ago @josegonzalez I'm not sure, I just did the above to get version 5.3.2 working on my instance.
imWildCat
commented
6 years ago @archy-bold In my occasion, it caused Error: Could not find or load main class "-Xms256m after editing the jvm.options file. Any suggestions?
archy-bold
commented
6 years ago @imWildCat Did you also run the following command?
export ELASTICSEARCH_CUSTOM_ENV='ES_JAVA_OPTS="-Xms256m -Xmx256m"'As that will also add the options to the start service command. Just restart your ssh session on the machine or run the following to unset the env variable.
unset ELASTICSEARCH_CUSTOM_ENVIf you haven't done that, post up the top section of your jvm.options file and I'll take a look.
imWildCat
commented
6 years ago @archy-bold Thank you for your prompt reply!
I am sure that I have unset ELASTICSEARCH_CUSTOM_ENV, but it still does not work:
$ docker logs e036facc5608
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256mecho $ELASTICSEARCH_CUSTOM_ENV
[empty]As for my jvm.options:
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms128m
-Xmx512m
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# force the server VM (remove on 32-bit client JVMs)
-server
# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}
## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime
# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}
# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file
# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M
# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true@imWildCat Have you restarted the container? Set $ELASTICSEARCH_NAME to the name of your dokku-elasticsearch service.
ELASTICSEARCH_NAME=elastic-serivce-name-here
docker stop $(docker ps -aqf "name=$ELASTICSEARCH_NAME")
docker rm $(docker ps -aqf "name=$ELASTICSEARCH_NAME")
dokku elasticsearch:restart $ELASTICSEARCH_NAME@archy-bold
Same issue. It's wired. Not matter its version is 5.6.7 or 5.3.2:
~ ⌚ 16:23:16
$ ELASTICSEARCH_NAME=es-5-01
docker stop $(docker ps -aqf "name=$ELASTICSEARCH_NAME")
docker rm $(docker ps -aqf "name=$ELASTICSEARCH_NAME")
dokku elasticsearch:restart $ELASTICSEARCH_NAME
cac435693ef3
cac435693ef3
! Service is already stopped
-----> Starting container
=====> Copying config files into place
Waiting for container to be ready
ERROR: unable to connect~ ⌚ 16:24:30
$ docker logs dokku.elasticsearch.es-5-01
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256m
Error: Could not find or load main class "-Xms256mENVS:
~ ⌚ 16:24:55
$ env | grep ELASTICSEARCH
ELASTICSEARCH_IMAGE_VERSION=5.6.7And jvm config:
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms128m
-Xmx512m
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# force the server VM (remove on 32-bit client JVMs)
-server
# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}
## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime
# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}
# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file
# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M
# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true@imWildCat The only way you'll get that error is if ES_JAVA_OPTS is being set from somewhere. I'd check your container's environment variables, the configuration, etc. But I'm afraid I'm stumped.
The error appears to be coming from the script that starts elasticsearch and that script sets ES_JAVA_OPTS from jvm.options and the variable only. If you get the same error every time you create a new instance, it suggests it's getting the variable from somewhere in your machine. If you can't find it to clear it, I'd suggest uninstalling the plugin and trying again. Then, if you can, on another machine.
imWildCat
commented
6 years ago @archy-bold Many thanks for your kindness!
I tried on a VPS with even larger memory (8G). It started successfully at the first time with the default config. However, it failed to restart after changing the config file. I decided not to waste time on this beta quality plugin. Instead, I plan to just install ElasticSearch on the host and find some way to export the local port to a container. Or just call the public IP from the containers.
ariejan
commented
5 years ago All right, just a quick recap on this issue for people looking to setup Elasticsearch 5.x. There's a workable solution in the above comments, but was a bit of trial an error. Hope this speeds things up for others:
Edit /etc/sysctl.conf, add vm.max_map_count = 262144 at the bottom. Then run sysctl -p to load the changes.
Create a new Elasticsearch instance with dokku, per manual. I've only tested this with the latest 5.x release.
export ELASTICSEARCH_IMAGE="elasticsearch"
export ELASTICSEARCH_IMAGE_VERSION="5.6.12"
dokku elasticsearch:create lollipopThis will result in the "unable to connect" error. But now worries.
/var/lib/dokku/services/elasticsearch/lollipop/config/jvm.optionsChange the 2g values for -Xms and -Xmx to 512m:
-Xms2g -> -Xms512m
-Xmx2g -> -Xmx512m/var/lib/dokku/services/elasticsearch/lollipop/config/elasticsearch.ymlDouble check there is only one network.host: 0.0.0.0 at the bottom of the file. (For reasons unknown to me, it was there twice, making Elasticsearch barf on boot up.)
docker logs <container_id>. You can get the container_id by looking at docker ps. josegonzalez
commented
5 years ago Is the sysctl change necessary?
haggen
commented
5 years ago @josegonzalez I'm no expert but it seems so https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
josegonzalez
commented
5 years ago Doesn't sound like something we can automate with the plugin, but maybe alert on? If you miss that, what happens @ariejan?
ariejan
commented
5 years ago @josegonzalez @haggen yes, it's necessary. Elasticsearch fails to start with a message indicating the default vm.max_map_count needs to be increased.
It's also covered in Elasticsearch's documentation
That same documentation referes to something called maximum size virtual memory check. I've not made any changes for this and (with 17 million records indexed) does not seem to be a problem.
josegonzalez
commented
5 years ago What do you think we should do to upgrade the plugin? Seems like requiring us to set a sysctl setting on the host is pretty awful. Is that something we can do as a docker argument?
ariejan
commented
5 years ago It's not docker related, as the change needs to be made on the host system.
People are going to run into this one way or another when running Elasticsearch. I think it's fair to ask users to make this change to their system. I don't think it's a change this plugin should make on its own, though.
There are two places where I'd expect a message or alert (if I were unaware of this issue and installed the elasticsearch plugin).
First I would expect instructions in the README / installation instructions. That way 80% of the users of this plugin will make the change and not run into any trouble.
Second, when running elasticsearch:create it would be possible (and quite easy) to check the value of vm.max_map_count (use sysctl -n vm.max_map_count) to be at least 262144 and show a warning. Better yet, prevent a new container from being created because it will fail with Elasticsearch 5+.
I haven't checked where the jvm.options come from, but we should be able to fix memory allocation variables easily enough to not cause trouble by default. If more memory needs to be made available, let's include those instructions in the README.
The network.host = 0.0.0.0 does come from the plugin, but it's reasonable to think it was a user (e.g. my) error that it ended up in the elasticsearch.yml twice.
So, to summarize:
vm.max_map_count instructions-Xms and -Xmx jvm variablesvm.max_map_count and prevent creation of containers if it's not set correctly.-Xms and -Xmx from 2g to 512m.I can easily take care of the README. I'm new to dokku plugins, but I'll take a stab at making the two other changes as well.
ariejan
commented
5 years ago @josegonzalez I'd appreciate your feedback on #63, especially on how to add a proper test for creating a 5.x container.
josegonzalez
commented
5 years ago I'm going to close this as we now install 6.7.1 by default, which appears to work in unit testing on travis.
Creating an instance via:
It's stuck restarting:
It looks like it does right away:
Thoughts?