search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.94k stars 24.74k forks source link

Elastic Server Crash #14543

Closed dobariya closed 8 years ago

dobariya commented 8 years ago

I have three elastic servers all masters having total 32 GB ram each,16 GB allocated to elastic servers. From last two days all elastic servers crashes

Am using virtual environment with network file system to store data and logs java version :8.60 Elastic search version is 1.7.0 Indices count:around 300 Total documents in whole elastic search : around 60 million

Below is my config

cluster.name: Cluster1

cluster.routing.allocation.disk.threshold_enabled: false script.disable_dynamic: false node.name: "Master1"

node.master: true node.data: true

index.query.bool.max_clause_count: 50100 indices.fielddata.cache.size: 25% indices.fielddata.cache.expire: 5m action.disable_delete_all_indices: true indices.cluster.send_refresh_mapping: false index.cache.field.type: soft

path.data: \nas5\Elasticsearch\Data

path.logs: \nas5\Elasticsearch\Logs\Master1

bootstrap.mlockall: true

http.max_content_length: 999mb

indices.recovery.max_bytes_per_sec: 100mb

indices.recovery.concurrent_streams: 5

above config goes same for all three servers Master1,Master2,Master3.

here is my error log

 #
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x000000000404153b, pid=1820, tid=4464
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# J 10733 C2 org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Lorg/apache/lucene/store/Directory;Ljava/lang/String;Lorg/elasticsearch/common/collect/ImmutableMap$Builder;Lorg/elasticsearch/common/logging/ESLogger;Lorg/apache/lucene/util/Version;Z)V (290 bytes) @ 0x000000000404153b [0x00000000040405a0+0xf9b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007001e: The system cannot read from the specified device.

)
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x000000002e6d0000):  JavaThread "elasticsearch[Master101][fetch_shard_store][T#42]" daemon [_thread_in_Java, id=4464, stack(0x00000008fa2c0000,0x00000008fa3c0000)]

siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000000 0x0000000031a08000 0x00000000c000020c 

Registers:
RAX=0x0000000000000000, RBX=0x0000000000007ffe, RCX=0x0000000000007ffe, RDX=0x00000003cfb6dc98
RSP=0x00000008fa3be9a0, RBP=0x00000007c045ee70, RSI=0x00000003cfb6dc10, RDI=0x000000000000002e
R8 =0x00000003cfb6dba8, R9 =0x0000000031a00000, R10=0x00000003cfb6dc98, R11=0x0000000000000000
R12=0x0000000000000000, R13=0x00000003cfb6d918, R14=0x00000008fa3be800, R15=0x000000002e6d0000
RIP=0x000000000404153b, EFLAGS=0x0000000000010246

Top of Stack: (sp=0x00000008fa3be9a0)
0x00000008fa3be9a0:   ffffffffffffffff 00000003cfb69b30
0x00000008fa3be9b0:   0000000000000010 000000000273ca1c
0x00000008fa3be9c0:   79f6b17bf808bdce 00000003cfb4acd0
0x00000008fa3be9d0:   0000000454c19650 00000003cfb57c50
0x00000008fa3be9e0:   0000000300000000 00000003cfb58bd8
0x00000008fa3be9f0:   00000003cfb6dc98 0000000000008006
0x00000008fa3bea00:   00000007c00016d0 00000003cfb6dc98
0x00000008fa3bea10:   00000003cfb6d180 0000000002847c08
0x00000008fa3bea20:   00000007c000da48 0000000000000000
0x00000008fa3bea30:   00000003cfb594d8 0000000002ff9138
0x00000008fa3bea40:   00000003cfb59478 00000003cfb6a2b8
0x00000008fa3bea50:   00000003cfb6a588 0000000300000001
0x00000008fa3bea60:   00000003cfb6a308 00000000028c2170
0x00000008fa3bea70:   00000003cfb4fb40 0000000379f68d28
0x00000008fa3bea80:   0000000000000023 000000220211a5c0
0x00000008fa3bea90:   0000000079f6da2d 000000000470b460 

Instructions: (pc=0x000000000404153b)
0x000000000404151b:   8b 58 1c 41 8b 48 18 44 2b d9 41 83 fb 08 0f 8c
0x000000000404152b:   7e 17 00 00 4d 8b 48 10 45 0f b6 58 2a 48 63 d9
0x000000000404153b:   4d 8b 0c 19 83 c1 08 41 89 48 18 45 85 db 0f 85
0x000000000404154b:   b6 17 00 00 49 0f c9 4d 8b 5f 60 4d 8b d3 49 81 

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x0000000000007ffe is an unknown value
RCX=0x0000000000007ffe is an unknown value
RDX=
[error occurred during error reporting (printing register info), id 0xc0000005]

Stack: [0x00000008fa2c0000,0x00000008fa3c0000],  sp=0x00000008fa3be9a0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x000000000404153b

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x000000002ebc2000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#53]" daemon [_thread_blocked, id=4700, stack(0x00000009c5da0000,0x00000009c5ea0000)]
  0x000000002ebbe000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#52]" daemon [_thread_blocked, id=4176, stack(0x00000009042d0000,0x00000009043d0000)]
  0x000000002ebc2800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#51]" daemon [_thread_blocked, id=2692, stack(0x0000000039830000,0x0000000039930000)]
  0x000000002ebbd800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#50]" daemon [_thread_blocked, id=2180, stack(0x00000008f9560000,0x00000008f9660000)]
  0x000000002c881800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#49]" daemon [_thread_blocked, id=1308, stack(0x00000009c5a20000,0x00000009c5b20000)]
  0x000000002c881000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#48]" daemon [_thread_blocked, id=2796, stack(0x00000008fb750000,0x00000008fb850000)]
  0x000000002c882800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#47]" daemon [_thread_blocked, id=2004, stack(0x00000008fb880000,0x00000008fb980000)]
  0x000000002c880000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#46]" daemon [_thread_blocked, id=3176, stack(0x0000000041540000,0x0000000041640000)]
  0x000000002e6d3800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#45]" daemon [_thread_blocked, id=3236, stack(0x00000008fa3c0000,0x00000008fa4c0000)]
  0x000000002e6d1800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#44]" daemon [_thread_blocked, id=1068, stack(0x0000000040960000,0x0000000040a60000)]
  0x000000002e6d3000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#43]" daemon [_thread_blocked, id=4156, stack(0x00000008f7db0000,0x00000008f7eb0000)]
=>0x000000002e6d0000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#42]" daemon [_thread_in_Java, id=4464, stack(0x00000008fa2c0000,0x00000008fa3c0000)]
  0x000000002e324800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#41]" daemon [_thread_blocked, id=4904, stack(0x000000003e6d0000,0x000000003e7d0000)]
  0x000000002e328000 JavaThread "elasticsearch[Master101][fetch_shard_store][T#40]" daemon [_thread_blocked, id=4092, stack(0x00000000403e0000,0x00000000404e0000)]
  0x000000002e326800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#39]" daemon [_thread_blocked, id=4244, stack(0x00000000410d0000,0x00000000411d0000)]
  0x000000002e323000 JavaThread "elasticsearch[Master101][generic][T#133]" daemon [_thread_blocked, id=588, stack(0x0000000040810000,0x0000000040910000)]
  0x000000002c763800 JavaThread "elasticsearch[Master101][generic][T#130]" daemon [_thread_blocked, id=3480, stack(0x000000003e7d0000,0x000000003e8d0000)]
  0x000000002c764800 JavaThread "elasticsearch[Master101][generic][T#134]" daemon [_thread_blocked, id=4208, stack(0x000000003e5a0000,0x000000003e6a0000)]
  0x000000002c75e800 JavaThread "elasticsearch[Master101][generic][T#135]" daemon [_thread_blocked, id=4696, stack(0x000000003d520000,0x000000003d620000)]
  0x000000002c75d800 JavaThread "elasticsearch[Master101][generic][T#136]" daemon [_thread_blocked, id=4348, stack(0x0000000036cb0000,0x0000000036db0000)]
  0x000000002e4ff000 JavaThread "elasticsearch[Master101][generic][T#137]" daemon [_thread_blocked, id=3708, stack(0x000000003e3c0000,0x000000003e4c0000)]
  0x000000002e4fd000 JavaThread "elasticsearch[Master101][generic][T#129]" daemon [_thread_blocked, id=3216, stack(0x000000003e2c0000,0x000000003e3c0000)]
  0x000000002e4fd800 JavaThread "elasticsearch[Master101][generic][T#131]" daemon [_thread_blocked, id=3132, stack(0x000000003e1c0000,0x000000003e2c0000)]
  0x000000002e500000 JavaThread "elasticsearch[Master101][generic][T#132]" daemon [_thread_blocked, id=3292, stack(0x000000003ce70000,0x000000003cf70000)]
  0x000000002e506000 JavaThread "elasticsearch[Master101][generic][T#128]" daemon [_thread_blocked, id=3588, stack(0x000000003a8b0000,0x000000003a9b0000)]
  0x000000002e502000 JavaThread "elasticsearch[Master101][generic][T#127]" daemon [_thread_blocked, id=2160, stack(0x0000000039df0000,0x0000000039ef0000)]
  0x000000002e504800 JavaThread "elasticsearch[Master101][flush][T#8]" daemon [_thread_blocked, id=1484, stack(0x000000003a240000,0x000000003a340000)]
  0x000000002e6d2000 JavaThread "elasticsearch[Master101][warmer][T#3]" daemon [_thread_blocked, id=4280, stack(0x000000003dbd0000,0x000000003dcd0000)]
  0x000000002e32a000 JavaThread "elasticsearch[Master101][merge][T#1]" daemon [_thread_blocked, id=4320, stack(0x000000003cc60000,0x000000003cd60000)]
  0x000000002e325800 JavaThread "elasticsearch[Master101][index][T#8]" daemon [_thread_blocked, id=4396, stack(0x000000003d300000,0x000000003d400000)]
  0x000000002e4f8800 JavaThread "elasticsearch[Master101][index][T#7]" daemon [_thread_blocked, id=3592, stack(0x000000003d120000,0x000000003d220000)]
  0x000000002e506800 JavaThread "elasticsearch[Master101][index][T#6]" daemon [_thread_blocked, id=2244, stack(0x000000003cd70000,0x000000003ce70000)]
  0x000000002e4fb800 JavaThread "elasticsearch[Master101][index][T#5]" daemon [_thread_blocked, id=4384, stack(0x000000003ca10000,0x000000003cb10000)]
  0x000000002e501800 JavaThread "elasticsearch[Master101][index][T#4]" daemon [_thread_blocked, id=2368, stack(0x000000003cb30000,0x000000003cc30000)]
  0x000000002e503800 JavaThread "elasticsearch[Master101][refresh][T#2]" daemon [_thread_blocked, id=4248, stack(0x000000003a7b0000,0x000000003a8b0000)]
  0x000000002e4fe800 JavaThread "elasticsearch[Master101][index][T#3]" daemon [_thread_blocked, id=4804, stack(0x0000000039f50000,0x000000003a050000)]
  0x000000002e505000 JavaThread "elasticsearch[Master101][index][T#2]" daemon [_thread_blocked, id=2272, stack(0x0000000039cc0000,0x0000000039dc0000)]
  0x000000002e4f9000 JavaThread "elasticsearch[Master101][index][T#1]" daemon [_thread_blocked, id=2144, stack(0x0000000039990000,0x0000000039a90000)]
  0x000000002e328800 JavaThread "elasticsearch[Master101][fetch_shard_store][T#34]" daemon [_thread_blocked, id=4796, stack(0x0000007273b60000,0x0000007273c60000)]
  0x000000002c87f800 JavaThread "elasticsearch[Master101][search][T#13]" daemon [_thread_blocked, id=2224, stack(0x0000005723570000,0x0000005723670000)]
  0x000000002c87e800 JavaThread "elasticsearch[Master101][search][T#12]" daemon [_thread_blocked, id=3460, stack(0x00000056bb740000,0x00000056bb840000)]
  0x000000002c87e000 JavaThread "elasticsearch[Master101][search][T#11]" daemon [_thread_blocked, id=3524, stack(0x00000056bb2a0000,0x00000056bb3a0000)]
  0x000000002c87d000 JavaThread "elasticsearch[Master101][search][T#10]" daemon [_thread_blocked, id=3436, stack(0x00000056bb5f0000,0x00000056bb6f0000)]
  0x000000002c87c800 JavaThread "elasticsearch[Master101][search][T#9]" daemon [_thread_blocked, id=4108, stack(0x00000056bb4b0000,0x00000056bb5b0000)]
  0x000000002c763000 JavaThread "elasticsearch[Master101][search][T#8]" daemon [_thread_blocked, id=4552, stack(0x0000005616480000,0x0000005616580000)]
  0x000000002c762000 JavaThread "elasticsearch[Master101][search][T#7]" daemon [_thread_blocked, id=168, stack(0x0000005615ef0000,0x0000005615ff0000)]
  0x000000002c761800 JavaThread "elasticsearch[Master101][search][T#6]" daemon [_thread_blocked, id=2356, stack(0x0000005616050000,0x0000005616150000)]
  0x000000002c760800 JavaThread "elasticsearch[Master101][search][T#5]" daemon [_thread_blocked, id=3168, stack(0x0000005615de0000,0x0000005615ee0000)]
  0x000000002c760000 JavaThread "elasticsearch[Master101][search][T#4]" daemon [_thread_blocked, id=2088, stack(0x0000005615a20000,0x0000005615b20000)]
  0x000000002c75f000 JavaThread "elasticsearch[Master101][search][T#3]" daemon [_thread_blocked, id=4352, stack(0x00000056157f0000,0x00000056158f0000)]
  0x000000002e4f7800 JavaThread "elasticsearch[Master101][search][T#1]" daemon [_thread_blocked, id=2216, stack(0x000000560ce10000,0x000000560cf10000)]
  0x000000002e4fc000 JavaThread "elasticsearch[Master101][search][T#2]" daemon [_thread_blocked, id=3456, stack(0x000000560cc10000,0x000000560cd10000)]
  0x000000002e324000 JavaThread "elasticsearch[Master101][listener][T#4]" daemon [_thread_blocked, id=4220, stack(0x0000000c5de00000,0x0000000c5df00000)]
  0x000000002ebbf800 JavaThread "elasticsearch[Master101][listener][T#3]" daemon [_thread_blocked, id=4532, stack(0x0000000c43d70000,0x0000000c43e70000)]
  0x000000002ebbf000 JavaThread "elasticsearch[Master101][listener][T#2]" daemon [_thread_blocked, id=2320, stack(0x0000000c3ceb0000,0x0000000c3cfb0000)]
  0x000000002ebc0800 JavaThread "elasticsearch[Master101][listener][T#1]" daemon [_thread_blocked, id=4876, stack(0x0000000c14ed0000,0x0000000c14fd0000)]
  0x000000002e500800 JavaThread "elasticsearch[Master101][management][T#2]" daemon [_thread_blocked, id=2972, stack(0x0000000060250000,0x0000000060350000)]
  0x000000002e4fa000 JavaThread "elasticsearch[Master101][fetch_shard_started][T#10]" daemon [_thread_blocked, id=2292, stack(0x0000000038b60000,0x0000000038c60000)]
  0x000000002e6cf000 JavaThread "DestroyJavaVM" [_thread_blocked, id=1188, stack(0x0000000002300000,0x0000000002400000)]
  0x000000002e6ce800 JavaThread "elasticsearch[keepAlive/1.7.0]" [_thread_blocked, id=2136, stack(0x00000000356a0000,0x00000000357a0000)]
  0x000000002e6cd800 JavaThread "elasticsearch[Master101][http_server_boss][T#1]{New I/O server boss #51}" daemon [_thread_in_native, id=4556, stack(0x0000000037a50000,0x0000000037b50000)]
  0x000000002e6cd000 JavaThread "elasticsearch[Master101][http_server_worker][T#16]{New I/O worker #50}" daemon [_thread_blocked, id=3264, stack(0x0000000037920000,0x0000000037a20000)]
  0x000000002e6cc000 JavaThread "elasticsearch[Master101][http_server_worker][T#15]{New I/O worker #49}" daemon [_thread_blocked, id=4520, stack(0x00000000376c0000,0x00000000377c0000)]
  0x000000002e6cb800 JavaThread "elasticsearch[Master101][http_server_worker][T#14]{New I/O worker #48}" daemon [_thread_blocked, id=4012, stack(0x0000000037150000,0x0000000037250000)]
  0x000000002e6ca800 JavaThread "elasticsearch[Master101][http_server_worker][T#13]{New I/O worker #47}" daemon [_thread_blocked, id=3772, stack(0x00000000374f0000,0x00000000375f0000)]
  0x000000002e6ca000 JavaThread "elasticsearch[Master101][http_server_worker][T#12]{New I/O worker #46}" daemon [_thread_blocked, id=2316, stack(0x0000000037300000,0x0000000037400000)]
  0x000000002e6c9000 JavaThread "elasticsearch[Master101][http_server_worker][T#11]{New I/O worker #45}" daemon [_thread_blocked, id=944, stack(0x0000000037040000,0x0000000037140000)]
  0x000000002e6c8800 JavaThread "elasticsearch[Master101][http_server_worker][T#10]{New I/O worker #44}" daemon [_thread_blocked, id=4828, stack(0x0000000034a50000,0x0000000034b50000)]
  0x000000002e6c7800 JavaThread "elasticsearch[Master101][http_server_worker][T#9]{New I/O worker #43}" daemon [_thread_blocked, id=4684, stack(0x0000000036f20000,0x0000000037020000)]
  0x000000002e6c7000 JavaThread "elasticsearch[Master101][http_server_worker][T#8]{New I/O worker #42}" daemon [_thread_blocked, id=4812, stack(0x0000000036db0000,0x0000000036eb0000)]
  0x000000002e6c6000 JavaThread "elasticsearch[Master101][http_server_worker][T#7]{New I/O worker #41}" daemon [_thread_blocked, id=3256, stack(0x00000000320b0000,0x00000000321b0000)]
  0x000000002e6c5800 JavaThread "elasticsearch[Master101][http_server_worker][T#6]{New I/O worker #40}" daemon [_thread_blocked, id=2516, stack(0x0000000035af0000,0x0000000035bf0000)]
  0x000000002e6c4800 JavaThread "elasticsearch[Master101][http_server_worker][T#5]{New I/O worker #39}" daemon [_thread_blocked, id=1252, stack(0x00000000359d0000,0x0000000035ad0000)]
  0x000000002ebc7000 JavaThread "elasticsearch[Master101][http_server_worker][T#4]{New I/O worker #38}" daemon [_thread_blocked, id=4252, stack(0x00000000358d0000,0x00000000359d0000)]
  0x000000002ebc6800 JavaThread "elasticsearch[Master101][http_server_worker][T#3]{New I/O worker #37}" daemon [_thread_blocked, id=4264, stack(0x0000000034230000,0x0000000034330000)]
  0x000000002ebc5800 JavaThread "elasticsearch[Master101][http_server_worker][T#2]{New I/O worker #36}" daemon [_thread_blocked, id=2664, stack(0x00000000357a0000,0x00000000358a0000)]
  0x000000002ebc5000 JavaThread "elasticsearch[Master101][http_server_worker][T#1]{New I/O worker #35}" daemon [_thread_blocked, id=3184, stack(0x00000000355a0000,0x00000000356a0000)]
  0x000000002ebc4000 JavaThread "elasticsearch[Master101][management][T#1]" daemon [_thread_blocked, id=4272, stack(0x00000000353c0000,0x00000000354c0000)]
  0x000000002ebc3800 JavaThread "elasticsearch[Master101][riverClusterService#updateTask][T#1]" daemon [_thread_blocked, id=4132, stack(0x0000000035250000,0x0000000035350000)]
  0x000000002ebc1000 JavaThread "elasticsearch[Master101][transport_client_timer][T#1]{Hashed wheel timer #1}" daemon [_thread_blocked, id=4916, stack(0x00000000350e0000,0x00000000351e0000)]
  0x000000002ebbc800 JavaThread "elasticsearch[Master101][clusterService#updateTask][T#1]" daemon [_thread_blocked, id=4344, stack(0x0000000034020000,0x0000000034120000)]
  0x000000002ebbc000 JavaThread "elasticsearch[Master101][[http_server_boss.default]][T#1]{New I/O server boss #34}" daemon [_thread_in_native, id=5116, stack(0x00000000346d0000,0x00000000347d0000)]
  0x000000002ebbb000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#16]{New I/O worker #33}" daemon [_thread_blocked, id=4844, stack(0x0000000034550000,0x0000000034650000)]
  0x000000002ebba800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#15]{New I/O worker #32}" daemon [_thread_blocked, id=1040, stack(0x0000000034380000,0x0000000034480000)]
  0x000000002ebb9800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#14]{New I/O worker #31}" daemon [_thread_blocked, id=4608, stack(0x0000000034120000,0x0000000034220000)]
  0x000000002ebb9000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#13]{New I/O worker #30}" daemon [_thread_blocked, id=4768, stack(0x0000000033da0000,0x0000000033ea0000)]
  0x000000002ebb8000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#12]{New I/O worker #29}" daemon [_thread_blocked, id=5044, stack(0x0000000033ef0000,0x0000000033ff0000)]
  0x000000002cc82000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#11]{New I/O worker #28}" daemon [_thread_blocked, id=1696, stack(0x00000000329f0000,0x0000000032af0000)]
  0x000000002cc81000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#10]{New I/O worker #27}" daemon [_thread_blocked, id=3936, stack(0x0000000033c90000,0x0000000033d90000)]
  0x000000002cc80800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#9]{New I/O worker #26}" daemon [_thread_blocked, id=1000, stack(0x0000000033430000,0x0000000033530000)]
  0x000000002cc7f800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#8]{New I/O worker #25}" daemon [_thread_blocked, id=4920, stack(0x0000000033b90000,0x0000000033c90000)]
  0x000000002cc7f000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#7]{New I/O worker #24}" daemon [_thread_blocked, id=2620, stack(0x0000000033930000,0x0000000033a30000)]
  0x000000002cc7e000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#6]{New I/O worker #23}" daemon [_thread_blocked, id=4164, stack(0x0000000033a40000,0x0000000033b40000)]
  0x000000002cc7d800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#5]{New I/O worker #22}" daemon [_thread_blocked, id=1028, stack(0x0000000033830000,0x0000000033930000)]
  0x000000002cc7c800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#4]{New I/O worker #21}" daemon [_thread_blocked, id=3032, stack(0x0000000032df0000,0x0000000032ef0000)]
  0x000000002cc7c000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#3]{New I/O worker #20}" daemon [_thread_blocked, id=972, stack(0x00000000336b0000,0x00000000337b0000)]
  0x000000002cc7b000 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#2]{New I/O worker #19}" daemon [_thread_blocked, id=3484, stack(0x0000000033540000,0x0000000033640000)]
  0x000000002cc7a800 JavaThread "elasticsearch[Master101][[http_server_worker.default]][T#1]{New I/O worker #18}" daemon [_thread_blocked, id=4892, stack(0x00000000331e0000,0x00000000332e0000)]
  0x000000002cc79800 JavaThread "elasticsearch[Master101][transport_client_boss][T#1]{New I/O boss #17}" daemon [_thread_blocked, id=4172, stack(0x0000000033320000,0x0000000033420000)]
  0x000000002cc79000 JavaThread "elasticsearch[Master101][transport_client_worker][T#16]{New I/O worker #16}" daemon [_thread_blocked, id=4004, stack(0x0000000032680000,0x0000000032780000)]
  0x000000002cc78000 JavaThread "elasticsearch[Master101][transport_client_worker][T#15]{New I/O worker #15}" daemon [_thread_blocked, id=3052, stack(0x00000000330d0000,0x00000000331d0000)]
  0x000000002cc77800 JavaThread "elasticsearch[Master101][transport_client_worker][T#14]{New I/O worker #14}" daemon [_thread_blocked, id=3912, stack(0x0000000032f00000,0x0000000033000000)]
  0x000000002cc76800 JavaThread "elasticsearch[Master101][transport_client_worker][T#13]{New I/O worker #13}" daemon [_thread_blocked, id=3024, stack(0x0000000032cf0000,0x0000000032df0000)]
  0x000000002cc75800 JavaThread "elasticsearch[Master101][transport_client_worker][T#12]{New I/O worker #12}" daemon [_thread_blocked, id=2644, stack(0x0000000032b00000,0x0000000032c00000)]
  0x000000002cc75000 JavaThread "elasticsearch[Master101][transport_client_worker][T#11]{New I/O worker #11}" daemon [_thread_in_vm, id=2416, stack(0x00000000328e0000,0x00000000329e0000)]
  0x000000002cc74000 JavaThread "elasticsearch[Master101][transport_client_worker][T#10]{New I/O worker #10}" daemon [_thread_in_vm, id=4816, stack(0x00000000327a0000,0x00000000328a0000)]
  0x000000002cc73800 JavaThread "elasticsearch[Master101][transport_client_worker][T#9]{New I/O worker #9}" daemon [_thread_in_vm, id=4800, stack(0x0000000032580000,0x0000000032680000)]
  0x000000002cc72800 JavaThread "elasticsearch[Master101][transport_client_worker][T#8]{New I/O worker #8}" daemon [_thread_in_vm, id=184, stack(0x0000000032450000,0x0000000032550000)]
  0x000000002cc6a000 JavaThread "elasticsearch[Master101][transport_client_worker][T#7]{New I/O worker #7}" daemon [_thread_blocked, id=2124, stack(0x00000000322e0000,0x00000000323e0000)]
  0x000000002cc66000 JavaThread "elasticsearch[Master101][transport_client_worker][T#6]{New I/O worker #6}" daemon [_thread_in_vm, id=2612, stack(0x00000000321b0000,0x00000000322b0000)]
  0x000000002cfe8000 JavaThread "elasticsearch[Master101][transport_client_worker][T#5]{New I/O worker #5}" daemon [_thread_blocked, id=2880, stack(0x0000000031730000,0x0000000031830000)]
  0x000000002cfe6000 JavaThread "elasticsearch[Master101][transport_client_worker][T#4]{New I/O worker #4}" daemon [_thread_blocked, id=2788, stack(0x0000000031fb0000,0x00000000320b0000)]
  0x000000002cfe4800 JavaThread "elasticsearch[Master101][transport_client_worker][T#3]{New I/O worker #3}" daemon [_thread_blocked, id=2172, stack(0x0000000031db0000,0x0000000031eb0000)]
  0x000000002b791000 JavaThread "elasticsearch[Master101][transport_client_worker][T#2]{New I/O worker #2}" daemon [_thread_blocked, id=2112, stack(0x0000000031bb0000,0x0000000031cb0000)]
  0x000000002b790000 JavaThread "elasticsearch[Master101][transport_client_worker][T#1]{New I/O worker #1}" daemon [_thread_blocked, id=3376, stack(0x0000000031a20000,0x0000000031b20000)]
  0x000000002b3d6000 JavaThread "elasticsearch[Master101][[ttl_expire]]" daemon [_thread_blocked, id=4304, stack(0x0000000031400000,0x0000000031500000)]
  0x000000002b406000 JavaThread "elasticsearch[Master101][master_mapping_updater]" [_thread_blocked, id=5076, stack(0x000000002ecd0000,0x000000002edd0000)]
  0x000000002feeb800 JavaThread "elasticsearch[Master101][scheduler][T#1]" daemon [_thread_in_vm, id=4224, stack(0x0000000031840000,0x0000000031940000)]
  0x000000002b6ce800 JavaThread "elasticsearch[Master101][[timer]]" daemon [_thread_blocked, id=3696, stack(0x000000002c100000,0x000000002c200000)]
  0x00000000298e4000 JavaThread "Service Thread" daemon [_thread_blocked, id=4508, stack(0x000000002b290000,0x000000002b390000)]
  0x000000002988a000 JavaThread "C1 CompilerThread3" daemon [_thread_blocked, id=4620, stack(0x000000002b080000,0x000000002b180000)]
  0x0000000029861800 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=2260, stack(0x000000002af20000,0x000000002b020000)]
  0x0000000029860800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=4444, stack(0x000000002a920000,0x000000002aa20000)]
  0x0000000029855000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=2776, stack(0x000000002acc0000,0x000000002adc0000)]
  0x0000000029852000 JavaThread "Attach Listener" daemon [_thread_blocked, id=4736, stack(0x000000002ab90000,0x000000002ac90000)]
  0x000000002984d800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=4412, stack(0x000000002aa20000,0x000000002ab20000)]
  0x000000002985a000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=4716, stack(0x000000002a740000,0x000000002a840000)]
  0x00000000297ef000 JavaThread "Finalizer" daemon [_thread_blocked, id=3924, stack(0x000000002a560000,0x000000002a660000)]
  0x00000000297e6000 JavaThread "Reference Handler" daemon [_thread_blocked, id=4868, stack(0x000000002a380000,0x000000002a480000)]

Other Threads:
  0x00000000297e3000 VMThread [stack: 0x000000002a260000,0x000000002a360000] [id=4944]
  0x00000000298ea800 WatcherThread [stack: 0x000000002adc0000,0x000000002aec0000] [id=4636]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00000000021fd160] Safepoint_lock - owner thread: 0x00000000297e3000
[0x00000000021fd1e0] Threads_lock - owner thread: 0x00000000297e3000

Heap:
 par new generation   total 613440K, used 295871K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K,  48% used [0x00000003c0000000, 0x00000003d0473258, 0x00000003e1490000)
  from space 68096K,  42% used [0x00000003e5710000, 0x00000003e738cd30, 0x00000003e9990000)
  to   space 68096K,   0% used [0x00000003e1490000, 0x00000003e1490000, 0x00000003e5710000)
 concurrent mark-sweep generation total 16095680K, used 6672644K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43851K, capacity 44256K, committed 44452K, reserved 1089536K
  class space    used 5122K, capacity 5230K, committed 5344K, reserved 1048576K

Card table byte_map: [0x00000000126b0000,0x00000000146c0000] byte_map_base: 0x00000000108b0000

Marking Bits: (CMSBitMap*) 0x00000000021ea468
 Bits: [0x0000000016cf0000, 0x0000000026289c00)

Mod Union Table: (CMSBitMap*) 0x00000000021ea528
 Bits: [0x0000000026290000, 0x0000000026666670)

Polling page: 0x0000000000150000

CodeCache: size=245760Kb used=43602Kb max_used=43938Kb free=202157Kb
 bounds [0x00000000026d0000, 0x0000000005270000, 0x00000000116d0000]
 total_blobs=11005 nmethods=10423 adapters=493
 compilation: enabled

Compilation events (10 events):
Event: 11343.637 Thread 0x000000002988a000 15521       3       java.util.TreeMap$EntryIterator::next (5 bytes)
Event: 11343.638 Thread 0x000000002988a000 nmethod 15521 0x0000000003f76490 code [0x0000000003f76600, 0x0000000003f768e8]
Event: 11343.638 Thread 0x000000002988a000 15522       3       java.util.TreeMap$EntryIterator::next (5 bytes)
Event: 11343.638 Thread 0x000000002988a000 nmethod 15522 0x0000000004cf4690 code [0x0000000004cf4800, 0x0000000004cf4a08]
Event: 11343.639 Thread 0x000000002988a000 15524   !   3       org.elasticsearch.common.netty.util.internal.ConversionUtil::toBoolean (104 bytes)
Event: 11343.639 Thread 0x000000002988a000 nmethod 15524 0x0000000004352ad0 code [0x0000000004352d20, 0x0000000004353c18]
Event: 11343.639 Thread 0x000000002988a000 15525       3       sun.nio.ch.Net::setSocketOption (352 bytes)
Event: 11343.642 Thread 0x000000002988a000 nmethod 15525 0x0000000004d58bd0 code [0x0000000004d58f80, 0x0000000004d5b338]
Event: 11343.642 Thread 0x000000002988a000 15526       3       sun.nio.ch.SocketOptionRegistry::findOption (23 bytes)
Event: 11343.642 Thread 0x000000002988a000 nmethod 15526 0x0000000003d87d90 code [0x0000000003d87f60, 0x0000000003d88748]

GC Heap History (10 events):
Event: 6730.117 GC heap before
{Heap before GC invocations=476 (full 1):
 par new generation   total 613440K, used 559963K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K, 100% used [0x00000003c0000000, 0x00000003e1490000, 0x00000003e1490000)
  from space 68096K,  21% used [0x00000003e1490000, 0x00000003e22d6ce8, 0x00000003e5710000)
  to   space 68096K,   0% used [0x00000003e5710000, 0x00000003e5710000, 0x00000003e9990000)
 concurrent mark-sweep generation total 16095680K, used 6670433K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43772K, capacity 44128K, committed 44452K, reserved 1089536K
  class space    used 5111K, capacity 5230K, committed 5344K, reserved 1048576K
Event: 6730.189 GC heap after
Heap after GC invocations=477 (full 1):
 par new generation   total 613440K, used 36305K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K,   0% used [0x00000003c0000000, 0x00000003c0000000, 0x00000003e1490000)
  from space 68096K,  53% used [0x00000003e5710000, 0x00000003e7a846a0, 0x00000003e9990000)
  to   space 68096K,   0% used [0x00000003e1490000, 0x00000003e1490000, 0x00000003e5710000)
 concurrent mark-sweep generation total 16095680K, used 6670534K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43772K, capacity 44128K, committed 44452K, reserved 1089536K
  class space    used 5111K, capacity 5230K, committed 5344K, reserved 1048576K
}
Event: 7866.592 GC heap before
{Heap before GC invocations=477 (full 1):
 par new generation   total 613440K, used 581649K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K, 100% used [0x00000003c0000000, 0x00000003e1490000, 0x00000003e1490000)
  from space 68096K,  53% used [0x00000003e5710000, 0x00000003e7a846a0, 0x00000003e9990000)
  to   space 68096K,   0% used [0x00000003e1490000, 0x00000003e1490000, 0x00000003e5710000)
 concurrent mark-sweep generation total 16095680K, used 6670534K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43780K, capacity 44128K, committed 44452K, reserved 1089536K
  class space    used 5111K, capacity 5230K, committed 5344K, reserved 1048576K
Event: 7866.666 GC heap after
Heap after GC invocations=478 (full 1):
 par new generation   total 613440K, used 45069K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K,   0% used [0x00000003c0000000, 0x00000003c0000000, 0x00000003e1490000)
  from space 68096K,  66% used [0x00000003e1490000, 0x00000003e40935b0, 0x00000003e5710000)
  to   space 68096K,   0% used [0x00000003e5710000, 0x00000003e5710000, 0x00000003e9990000)
 concurrent mark-sweep generation total 16095680K, used 6671112K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43780K, capacity 44128K, committed 44452K, reserved 1089536K
  class space    used 5111K, capacity 5230K, committed 5344K, reserved 1048576K
}
Event: 8966.310 GC heap before
{Heap before GC invocations=478 (full 1):
 par new generation   total 613440K, used 590413K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K, 100% used [0x00000003c0000000, 0x00000003e1490000, 0x00000003e1490000)
  from space 68096K,  66% used [0x00000003e1490000, 0x00000003e40935b0, 0x00000003e5710000)
  to   space 68096K,   0% used [0x00000003e5710000, 0x00000003e5710000, 0x00000003e9990000)
 concurrent mark-sweep generation total 16095680K, used 6671112K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43804K, capacity 44192K, committed 44452K, reserved 1089536K
  class space    used 5115K, capacity 5230K, committed 5344K, reserved 1048576K
Event: 8966.353 GC heap after
Heap after GC invocations=479 (full 1):
 par new generation   total 613440K, used 39423K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K,   0% used [0x00000003c0000000, 0x00000003c0000000, 0x00000003e1490000)
  from space 68096K,  57% used [0x00000003e5710000, 0x00000003e7d8ff00, 0x00000003e9990000)
  to   space 68096K,   0% used [0x00000003e1490000, 0x00000003e1490000, 0x00000003e5710000)
 concurrent mark-sweep generation total 16095680K, used 6671222K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43804K, capacity 44192K, committed 44452K, reserved 1089536K
  class space    used 5115K, capacity 5230K, committed 5344K, reserved 1048576K
}
Event: 9983.773 GC heap before
{Heap before GC invocations=479 (full 1):
 par new generation   total 613440K, used 584767K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K, 100% used [0x00000003c0000000, 0x00000003e1490000, 0x00000003e1490000)
  from space 68096K,  57% used [0x00000003e5710000, 0x00000003e7d8ff00, 0x00000003e9990000)
  to   space 68096K,   0% used [0x00000003e1490000, 0x00000003e1490000, 0x00000003e5710000)
 concurrent mark-sweep generation total 16095680K, used 6671222K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43804K, capacity 44192K, committed 44452K, reserved 1089536K
  class space    used 5115K, capacity 5230K, committed 5344K, reserved 1048576K
Event: 9983.842 GC heap after
Heap after GC invocations=480 (full 1):
 par new generation   total 613440K, used 25941K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K,   0% used [0x00000003c0000000, 0x00000003c0000000, 0x00000003e1490000)
  from space 68096K,  38% used [0x00000003e1490000, 0x00000003e2de5640, 0x00000003e5710000)
  to   space 68096K,   0% used [0x00000003e5710000, 0x00000003e5710000, 0x00000003e9990000)
 concurrent mark-sweep generation total 16095680K, used 6671399K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43804K, capacity 44192K, committed 44452K, reserved 1089536K
  class space    used 5115K, capacity 5230K, committed 5344K, reserved 1048576K
}
Event: 11007.433 GC heap before
{Heap before GC invocations=480 (full 1):
 par new generation   total 613440K, used 571285K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K, 100% used [0x00000003c0000000, 0x00000003e1490000, 0x00000003e1490000)
  from space 68096K,  38% used [0x00000003e1490000, 0x00000003e2de5640, 0x00000003e5710000)
  to   space 68096K,   0% used [0x00000003e5710000, 0x00000003e5710000, 0x00000003e9990000)
 concurrent mark-sweep generation total 16095680K, used 6671399K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43804K, capacity 44192K, committed 44452K, reserved 1089536K
  class space    used 5115K, capacity 5230K, committed 5344K, reserved 1048576K
Event: 11007.495 GC heap after
Heap after GC invocations=481 (full 1):
 par new generation   total 613440K, used 29171K [0x00000003c0000000, 0x00000003e9990000, 0x00000003e9990000)
  eden space 545344K,   0% used [0x00000003c0000000, 0x00000003c0000000, 0x00000003e1490000)
  from space 68096K,  42% used [0x00000003e5710000, 0x00000003e738cd30, 0x00000003e9990000)
  to   space 68096K,   0% used [0x00000003e1490000, 0x00000003e1490000, 0x00000003e5710000)
 concurrent mark-sweep generation total 16095680K, used 6672644K [0x00000003e9990000, 0x00000007c0000000, 0x00000007c0000000)
 Metaspace       used 43804K, capacity 44192K, committed 44452K, reserved 1089536K
  class space    used 5115K, capacity 5230K, committed 5344K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 11340.806 Thread 0x000000002ebbc800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000309f80c method=org.elasticsearch.cluster.routing.RoutingNodes.assignedShards(Lorg/elasticsearch/index/shard/ShardId;)Ljava/util/List; @ 15
Event: 11340.806 Thread 0x000000002ebbc800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003604bb4 method=org.elasticsearch.cluster.routing.allocation.AllocationService.electPrimariesAndUnassignedDanglingReplicas(Lorg/elasticsearch/cluster/routing/allocation/RoutingAllocation”ÀæLg&Æ@
Event: 11340.807 Thread 0x000000002ebbc800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000309e890 method=org.elasticsearch.cluster.routing.RoutingNodes.assignedShards(Lorg/elasticsearch/index/shard/ShardId;)Ljava/util/List; @ 15
Event: 11340.808 Thread 0x000000002ebbc800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003039680 method=org.elasticsearch.cluster.routing.RoutingNodes.activeReplica(Lorg/elasticsearch/cluster/routing/ShardRouting;)Lorg/elasticsearch/cluster/routing/MutableShardRouting; @ 10M®‘g&Æ@
Event: 11340.809 Thread 0x000000002ebbc800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003039680 method=org.elasticsearch.cluster.routing.RoutingNodes.activeReplica(Lorg/elasticsearch/cluster/routing/ShardRouting;)Lorg/elasticsearch/cluster/routing/MutableShardRouting; @ 10ÖHכֿ&Æ@
Event: 11341.107 Thread 0x000000002ebbc800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x0000000005064854 method=org.apache.lucene.util.IntroSorter.quicksort(III)V @ 156
Event: 11341.111 Thread 0x000000002ebbc800 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x0000000005064854 method=org.apache.lucene.util.IntroSorter.quicksort(III)V @ 156
Event: 11341.120 Thread 0x000000002ebbc800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00000000042731c0 method=java.io.BufferedOutputStream.write([BII)V @ 20
Event: 11343.637 Thread 0x000000002cc79800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000004383e9c method=org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(Ljava/util/Set;J)V @ 13
Event: 11343.637 Thread 0x000000002cc79800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000305c814 method=sun.nio.ch.SocketChannelImpl.isConnected()Z @ 12

Internal exceptions (10 events):
Event: 11343.856 Thread 0x000000002ebbe000 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cf42ff88) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11343.860 Thread 0x000000002c880000 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003ceda9ad0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11343.877 Thread 0x000000002c882800 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cf4c7188) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11343.884 Thread 0x000000002e6d1800 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cf660fe0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11344.188 Thread 0x000000002e6d3800 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cf604f50) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11344.241 Thread 0x000000002ebc2000 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cf187008) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11344.325 Thread 0x000000002c881000 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cff0df50) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11344.325 Thread 0x000000002e6d0000 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cfb55030) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11344.328 Thread 0x000000002c880000 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cf978fd0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 11344.376 Thread 0x000000002e324800 Exception <a 'sun/nio/fs/WindowsException'> (0x00000003cfdb2178) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u60\4407\hotspot\src\share\vm\prims\jni.cpp, line 709]

Events (10 events):
Event: 11341.111 Thread 0x000000002ebbc800 DEOPT UNPACKING pc=0x000000000271582a sp=0x000000003411ec20 mode 2
Event: 11341.120 Thread 0x000000002ebbc800 Uncommon trap: trap_request=0xffffffde fr.pc=0x00000000042731c0
Event: 11341.120 Thread 0x000000002ebbc800 DEOPT PACKING pc=0x00000000042731c0 sp=0x000000003411e810
Event: 11341.120 Thread 0x000000002ebbc800 DEOPT UNPACKING pc=0x000000000271582a sp=0x000000003411e850 mode 2
Event: 11343.637 Thread 0x000000002cc79800 Uncommon trap: trap_request=0xffffff65 fr.pc=0x0000000004383e9c
Event: 11343.637 Thread 0x000000002cc79800 DEOPT PACKING pc=0x0000000004383e9c sp=0x000000003341f140
Event: 11343.637 Thread 0x000000002cc79800 DEOPT UNPACKING pc=0x000000000271582a sp=0x000000003341efc8 mode 2
Event: 11343.637 Thread 0x000000002cc79800 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000000000305c814
Event: 11343.637 Thread 0x000000002cc79800 DEOPT PACKING pc=0x000000000305c814 sp=0x000000003341f080
Event: 11343.637 Thread 0x000000002cc79800 DEOPT UNPACKING pc=0x000000000271582a sp=0x000000003341f030 mode 2

Dynamic libraries:
0x000000013f0d0000 - 0x000000013f107000     C:\Program Files\Java\jre1.8.0_60\bin\java.exe
0x0000000077790000 - 0x0000000077939000     C:\Windows\SYSTEM32\ntdll.dll
0x0000000077570000 - 0x000000007768f000     C:\Windows\system32\kernel32.dll
0x000007fefd5a0000 - 0x000007fefd60c000     C:\Windows\system32\KERNELBASE.dll
0x000007fefdbc0000 - 0x000007fefdc9b000     C:\Windows\system32\ADVAPI32.dll
0x000007feff480000 - 0x000007feff51f000     C:\Windows\system32\msvcrt.dll
0x000007feff370000 - 0x000007feff38f000     C:\Windows\SYSTEM32\sechost.dll
0x000007fefda90000 - 0x000007fefdbbd000     C:\Windows\system32\RPCRT4.dll
0x0000000077690000 - 0x000000007778a000     C:\Windows\system32\USER32.dll
0x000007fefe4d0000 - 0x000007fefe537000     C:\Windows\system32\GDI32.dll
0x000007feff470000 - 0x000007feff47e000     C:\Windows\system32\LPK.dll
0x000007fefdca0000 - 0x000007fefdd69000     C:\Windows\system32\USP10.dll
0x000007fefbe40000 - 0x000007fefc034000     C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.18837_none_fa3b1e3d17594757\COMCTL32.dll
0x000007feff5b0000 - 0x000007feff621000     C:\Windows\system32\SHLWAPI.dll
0x000007feff580000 - 0x000007feff5ae000     C:\Windows\system32\IMM32.DLL
0x000007feff710000 - 0x000007feff819000     C:\Windows\system32\MSCTF.dll
0x0000000074410000 - 0x00000000744e2000     C:\Program Files\Java\jre1.8.0_60\bin\msvcr100.dll
0x0000000066f30000 - 0x00000000677b9000     C:\Program Files\Java\jre1.8.0_60\bin\server\jvm.dll
0x000007fef8d20000 - 0x000007fef8d29000     C:\Windows\system32\WSOCK32.dll
0x000007feff410000 - 0x000007feff45d000     C:\Windows\system32\WS2_32.dll
0x000007feff460000 - 0x000007feff468000     C:\Windows\system32\NSI.dll
0x000007fef8a40000 - 0x000007fef8a7b000     C:\Windows\system32\WINMM.dll
0x000007fefc610000 - 0x000007fefc61c000     C:\Windows\system32\VERSION.dll
0x0000000077950000 - 0x0000000077957000     C:\Windows\system32\PSAPI.DLL
0x0000000074780000 - 0x000000007478f000     C:\Program Files\Java\jre1.8.0_60\bin\verify.dll
0x0000000074520000 - 0x0000000074549000     C:\Program Files\Java\jre1.8.0_60\bin\java.dll
0x00000000743f0000 - 0x0000000074406000     C:\Program Files\Java\jre1.8.0_60\bin\zip.dll
0x000007fefe540000 - 0x000007feff2c9000     C:\Windows\system32\SHELL32.dll
0x000007fefe060000 - 0x000007fefe263000     C:\Windows\system32\ole32.dll
0x000007fefd540000 - 0x000007fefd54f000     C:\Windows\system32\profapi.dll
0x00000000743d0000 - 0x00000000743ea000     C:\Program Files\Java\jre1.8.0_60\bin\net.dll
0x000007fefcbf0000 - 0x000007fefcc45000     C:\Windows\system32\mswsock.dll
0x000007fefd1f0000 - 0x000007fefd1f7000     C:\Windows\System32\wship6.dll
0x000007fef9ec0000 - 0x000007fef9ee7000     C:\Windows\system32\IPHLPAPI.DLL
0x000007fef9eb0000 - 0x000007fef9ebb000     C:\Windows\system32\WINNSI.DLL
0x000007fef9c30000 - 0x000007fef9c41000     C:\Windows\system32\dhcpcsvc6.DLL
0x000007fef9bd0000 - 0x000007fef9be8000     C:\Windows\system32\dhcpcsvc.DLL
0x000007fefd040000 - 0x000007fefd058000     C:\Windows\system32\CRYPTSP.dll
0x000007fefca20000 - 0x000007fefca67000     C:\Windows\system32\rsaenh.dll
0x000007fefd890000 - 0x000007fefd8ae000     C:\Windows\system32\USERENV.dll
0x000007fefd3a0000 - 0x000007fefd3af000     C:\Windows\system32\CRYPTBASE.dll
0x00000000743b0000 - 0x00000000743c1000     C:\Program Files\Java\jre1.8.0_60\bin\nio.dll
0x0000000074770000 - 0x000000007477d000     C:\Program Files\Java\jre1.8.0_60\bin\management.dll
0x000007fefb930000 - 0x000007fefb945000     C:\Windows\system32\NLAapi.dll
0x000007fef9820000 - 0x000007fef9835000     C:\Windows\system32\napinsp.dll
0x000007fefcc70000 - 0x000007fefcccb000     C:\Windows\system32\DNSAPI.dll
0x000007fef9850000 - 0x000007fef985b000     C:\Windows\System32\winrnr.dll
0x000007fefc6e0000 - 0x000007fefc6e7000     C:\Windows\System32\wshtcpip.dll
0x000007fef9860000 - 0x000007fef9868000     C:\Windows\system32\rasadhlp.dll
0x000007fef83b0000 - 0x000007fef8403000     C:\Windows\System32\fwpuclnt.dll
0x0000000180000000 - 0x0000000180038000     C:\Users\ldprsvc\AppData\Local\Temp\2\jna-36009638\jna3897599377805093171.dll
0x000007feec720000 - 0x000007feec845000     C:\Windows\system32\DBGHELP.DLL

VM Arguments:
jvm_args: -Xms16g -Xmx16g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des-foreground=yes -Des.path.home=E:\Share\elasticsearch-1.7.0 
java_command: org.elasticsearch.bootstrap.Elasticsearch
java_class_path (initial): ;E:\Share\elasticsearch-1.7.0/lib/elasticsearch-1.7.0.jar;E:\Share\elasticsearch-1.7.0/lib/antlr-runtime-3.5.jar;E:\Share\elasticsearch-1.7.0/lib/apache-log4j-extras-1.2.17.jar;E:\Share\elasticsearch-1.7.0/lib/asm-4.1.jar;E:\Share\elasticsearch-1.7.0/lib/asm-commons-4.1.jar;E:\Share\elasticsearch-1.7.0/lib/elasticsearch-1.7.0.jar;E:\Share\elasticsearch-1.7.0/lib/groovy-all-2.4.4.jar;E:\Share\elasticsearch-1.7.0/lib/jna-4.1.0.jar;E:\Share\elasticsearch-1.7.0/lib/jts-1.13.jar;E:\Share\elasticsearch-1.7.0/lib/log4j-1.2.17.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-analyzers-common-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-core-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-expressions-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-grouping-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-highlighter-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-join-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-memory-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-misc-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-queries-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-queryparser-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-sandbox-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-spatial-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/lucene-suggest-4.10.4.jar;E:\Share\elasticsearch-1.7.0/lib/spatial4j-0.4.1.jar;E:\Share\elasticsearch-1.7.0/lib/sigar/*
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\Java\jre1.8.0_60
PATH=C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;E:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;E:\Program Files\Microsoft SQL Server\100\Tools\Binn\;E:\Program Files\Microsoft SQL Server\100\DTS\Binn\;E:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;E:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;E:\Program Files\Microsoft SQL Server\90\DTS\Binn\;E:\Program Files (x86)\Microsoft SQL Server\90\DTS\Binn\
USERNAME=ldprsvc
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 45 Stepping 7, GenuineIntel

---------------  S Y S T E M  ---------------

OS: Windows Server 2008 R2 , 64 bit Build 7601 (6.1.7601.18939)

CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, tsc, tscinvbit, tscinv

Memory: 4k page, physical 33553912k(10603016k free), swap 36172740k(12197716k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.60-b23) for windows-amd64 JRE (1.8.0_60-b27), built on Aug  4 2015 11:06:27 by "java_re" with MS VC++ 10.0 (VS2010)

time: Wed Nov 04 13:30:18 2015
elapsed time: 11365 seconds (0d 3h 9m 25s)
rmuir commented 8 years ago

You are using ES on windows with a network filesystem. I don't recommend this: or you will run into this bug: https://bugs.openjdk.java.net/browse/JDK-6415680

Nothing we can do on our side here, it needs to be fixed in the JDK. Feel free to use the bug information to report it there (they will likely just close it as dup):

#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#