the elasticsearch program crashes when executing the reindex command

[mandrills]

Description OS Ver. : MacOS Majave Docker Ver. : 2.5.0 PHP Ver. : 7.2 Elasticsearch Ver. : 7.6.2-2 Magento Ver. : ee-2.3.5

Elasticsearch crashes after running bin/magento indexer:reindex command

Steps To Reproduce

1. elasticsearch container crashes after running bin/magento indexer:reindex command

2. after docker is started, the access is normal, but the elasticsearch container is shutdown after the reindex command is executed, and it needs to be manually started to work normally

Expected Result

run bin/magento indexer:reindex after elasticsearch container no shutdown

Actual Result

elasticsearch.log

thank you

[christophermallory]

I had a similar issue caused by not enough system resources being allocated to the container.

The container was crashing with status code 137 in my case.

[mandrills]

I had a similar issue caused by not enough system resources being allocated to the container.

The container was crashing with status code 137 in my case.

I will try it again, thanks for your reply

[mandrills]

It should not be a problem of insufficient disk space Failed installing analysis-phonetic in logs elasticsearch-error.log

[mandrills]

this issue has been solved, just add a configuration item under elasticsearch image into docker-compose.yml

- "ES_JAVA_OPTS=-Xms512m -Xmx512m -XX:-UseConcMarkSweepGC"

[markshust]

@mandrills can you elaborate on this, and do we need to update the standard docker-compose.yml file in order to prevent this? Does this apply to all o/s's?

[piotrkwiecinski]

@markshust for ES 7.7 the value is set to 1GB.

Maybe we could set the value to - "ES_JAVA_OPTS=-Xms1g -Xmx1g in the docker-compose.yml to make it easier to tweak.

curl elasticsearch:9200/_nodes/vwcoo1-zSJGJcWdcxowatw/jvm/?pretty=true

Output:

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "docker-cluster",
  "nodes": {
    "vwcoo1-zSJGJcWdcxowatw": {
      "name": "cd88278997f9",
      "transport_address": "172.18.0.3:9300",
      "host": "172.18.0.3",
      "ip": "172.18.0.3",
      "version": "7.7.1",
      "build_flavor": "default",
      "build_type": "docker",
      "build_hash": "ad56dce891c901a492bb1ee393f12dfff473a423",
      "roles": [
        "ingest",
        "master",
        "transform",
        "data",
        "remote_cluster_client",
        "ml"
      ],
      "attributes": {
        "ml.machine_memory": "16649859072",
        "xpack.installed": "true",
        "transform.node": "true",
        "ml.max_open_jobs": "20"
      },
      "jvm": {
        "pid": 6,
        "version": "14.0.1",
        "vm_name": "OpenJDK 64-Bit Server VM",
        "vm_version": "14.0.1+7",
        "vm_vendor": "AdoptOpenJDK",
        "bundled_jdk": true,
        "using_bundled_jdk": true,
        "start_time_in_millis": 1612717877493,
        "mem": {
          "heap_init_in_bytes": 1073741824,
          "heap_max_in_bytes": 1073741824,
          "non_heap_init_in_bytes": 7667712,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 0
        },
        "gc_collectors": [
          "G1 Young Generation",
          "G1 Old Generation"
        ],
        "memory_pools": [
          "CodeHeap 'non-nmethods'",
          "Metaspace",
          "CodeHeap 'profiled nmethods'",
          "Compressed Class Space",
          "G1 Eden Space",
          "G1 Old Gen",
          "G1 Survivor Space",
          "CodeHeap 'non-profiled nmethods'"
        ],
        "using_compressed_ordinary_object_pointers": "true",
        "input_arguments": [
          "-Xshare:auto",
          "-Des.networkaddress.cache.ttl=60",
          "-Des.networkaddress.cache.negative.ttl=10",
          "-XX:+AlwaysPreTouch",
          "-Xss1m",
          "-Djava.awt.headless=true",
          "-Dfile.encoding=UTF-8",
          "-Djna.nosys=true",
          "-XX:-OmitStackTraceInFastThrow",
          "-XX:+ShowCodeDetailsInExceptionMessages",
          "-Dio.netty.noUnsafe=true",
          "-Dio.netty.noKeySetOptimization=true",
          "-Dio.netty.recycler.maxCapacityPerThread=0",
          "-Dio.netty.allocator.numDirectArenas=0",
          "-Dlog4j.shutdownHookEnabled=false",
          "-Dlog4j2.disable.jmx=true",
          "-Djava.locale.providers=SPI,COMPAT",
          "-Xms1g",
          "-Xmx1g",
          "-XX:+UseG1GC",
          "-XX:G1ReservePercent=25",
          "-XX:InitiatingHeapOccupancyPercent=30",
          "-Djava.io.tmpdir=/tmp/elasticsearch-2095563895300564387",
          "-XX:+HeapDumpOnOutOfMemoryError",
          "-XX:HeapDumpPath=data",
          "-XX:ErrorFile=logs/hs_err_pid%p.log",
          "-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m",
          "-Des.cgroups.hierarchy.override=/",
          "-XX:MaxDirectMemorySize=536870912",
          "-Des.path.home=/usr/share/elasticsearch",
          "-Des.path.conf=/usr/share/elasticsearch/config",
          "-Des.distribution.flavor=default",
          "-Des.distribution.type=docker",
          "-Des.bundled_jdk=true"
        ]
      }
    }
  }
}

vwcoo1-zSJGJcWdcxowatw is a node name so it may be different for your container.

[markshust]

@piotrkwiecinski I like that approach as it's very easy to change

[markshust]

Added to docker-compose.yml file, closing this out. Please reopen if there are new issues or if the problem still persists.

[gaiterjones]

It's worth mentioning that if you set ES_JAVA_OPTS to 1G it will use 1GB RAM, it's not a limit. This caused problems for me on some smaller vmware dev environments with limited RAM. 512 is certainly too small for production where 4 to 16G fits better.