Closed avleen closed 10 years ago
I can't repro locally. Can you post /_cluster/state?filter_metadata
and /_nodes?all
somewhere?
or do you see a stacktrace in the logs somewhere?
Hi @drewr ew, I hope you don't mind, I emailed them to you as they contain data I'd prefer not to post publicly :)
@s1monw no stack traces from this in any logs on any ES node.
@avleen I don't mind at all. However, I haven't seen anything yet. Where did you send it? You can use first.last@elasticsearch.
Update: Nevermind, spam. :smiling_imp:
Hi Drew,
Just double checked.. Yup that's where I sent it, at 17:29 ET. Has two attachments, a zip file and a text file. On May 23, 2014 6:32 PM, "Drew Raines" notifications@github.com wrote:
@avleen https://github.com/avleen I don't mind at all. However, I haven't seen anything yet. You can use first.last@elasticsearch.
Reply to this email directly or view it on GitHubhttps://github.com/elasticsearch/elasticsearch/issues/6297#issuecomment-44067235 .
FWIW I'm also seeing this, on a freshly-updated 1.2.1 cluster (from 1.1.1) with 64-bit Java 1.7.0_55, CentOS 5.8.
This PR addresses should fix this NullPointerException: https://github.com/elasticsearch/elasticsearch/pull/6190
The fix will be included in 1.2.2
Hi, For info - just installed ES for the first time a week ago (1.2.1 debian wheezy 64bit), and hit this issue. Upgraded to 1.2.2 yesterday, and it still reports the NullPointerException when requesting _cat/nodes. Installed using the debian packages. Also get on a newly installed 1.2.2 node (though joined to the same cluster). Cheers, Matthew
Any stack trace in nodes logs?
No, nothing in node logs. Is there any way to increase debugging, maybe? I wondered if the package hadn't upgraded correctly, but it's all reporting 1.2.2 (logs say version[1.2.2], pid[10140], build[9902f08/2014-07-09T12:02:32Z]).
I can confirm this bug is still present in 1.3.0, even after upgrading all nodes:
$ curl 0:9200
{
"status" : 200,
"name" : "mycluster",
"version" : {
"number" : "1.3.0",
"build_hash" : "1265b1454eee7725a6918f57415c480028700fb4",
"build_timestamp" : "2014-07-23T13:46:36Z",
"build_snapshot" : false,
"lucene_version" : "4.9"
},
"tagline" : "You Know, for Search"
}
$ curl 0:9200/_cat/nodes
{"error":"NullPointerException[null]","status":500}
Nothing in the logfile.
@faxm0dem do you have multiple nodes running? if so did you look in the logs for all the nodes?
could you send the output of:
curl 0:9200/_nodes
curl 0:9200/_nodes/stats
thanks
Yes, two nodes, nothing on either side. Here's the output:
curl 0:9200/_nodes
{
"cluster_name" : "telecom",
"nodes" : {
"JUOZAr-mT6e8e9nmwWH9ww" : {
"name" : "node07-telecom",
"transport_address" : "inet[/10.0.104.214:9300]",
"host" : "node07",
"ip" : "10.0.104.214",
"version" : "1.3.0",
"build" : "1265b14",
"http_address" : "inet[/10.0.104.214:9200]",
"settings" : {
"node" : {
"name" : "node07-telecom"
},
"index" : {
"number_of_replicas" : "1",
"number_of_shards" : "8"
},
"bootstrap" : {
"mlockall" : "true"
},
"name" : "node07-telecom",
"pidfile" : "/var/run/elasticsearch/elasticsearch-telecom.pid",
"path" : {
"data" : "/var/lib/elasticsearch/telecom",
"work" : "/tmp/elasticsearch",
"home" : "/usr/share/elasticsearch",
"conf" : "/etc/elasticsearch/telecom",
"logs" : "/var/log/elasticsearch/telecom"
},
"cluster" : {
"name" : "telecom"
},
"indices" : {
"memory" : {
"index_buffer_size" : "30%"
}
},
"discovery" : {
"zen" : {
"minimum_master_nodes" : "1",
"ping" : {
"unicast" : {
"hosts" : [ "node07", "node38" ]
},
"multicast" : {
"enabled" : "false"
},
"timeout" : "30s"
}
}
}
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 16,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 2527,
"total_cores" : 16,
"total_sockets" : 1,
"cores_per_socket" : 16,
"cache_size_in_bytes" : 8192
},
"mem" : {
"total_in_bytes" : 25185079296
},
"swap" : {
"total_in_bytes" : 279650304
}
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 27129,
"max_file_descriptors" : 65535,
"mlockall" : false
},
"jvm" : {
"pid" : 27129,
"version" : "1.7.0_65",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.65-b04",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1406268341800,
"mem" : {
"heap_init_in_bytes" : 17179869184,
"heap_max_in_bytes" : 17066491904,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 17066491904
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 16,
"max" : 16,
"queue_size" : "200"
},
"snapshot_data" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"bench" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"get" : {
"type" : "fixed",
"min" : 16,
"max" : 16,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 16,
"max" : 16,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 16,
"max" : 16,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 48,
"max" : 48,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 16,
"max" : 16,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 8,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000,
"primary_interface" : {
"address" : "10.0.104.214",
"name" : "eth0",
"mac_address" : ""
}
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9300]",
"publish_address" : "inet[/10.0.104.214:9300]"
},
"http" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9200]",
"publish_address" : "inet[/10.0.104.214:9200]",
"max_content_length_in_bytes" : 104857600
},
"plugins" : [ ]
},
"JL38-jS9Sn67hQk2XxNZxw" : {
"name" : "logstash-netflow",
"transport_address" : "inet[/10.0.108.171:9303]",
"host" : "node38",
"ip" : "10.0.108.171",
"version" : "1.1.1",
"build" : "f1585f0",
"attributes" : {
"client" : "true",
"data" : "false"
},
"settings" : {
"path" : {
"logs" : "/home/sysunix/logs"
},
"cluster" : {
"name" : "telecom"
},
"node" : {
"client" : "true",
"name" : "logstash-netflow"
},
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : "node38:9300,node38:9301,node38:9302,node38:9303,node38:9304,node38:9305"
},
"multicast" : {
"enabled" : "false"
}
}
}
},
"http" : {
"enabled" : "false"
},
"name" : "logstash-netflow"
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 32
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 8365,
"max_file_descriptors" : 4096,
"mlockall" : false
},
"jvm" : {
"pid" : 8365,
"version" : "1.7.0_55",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.51-b03",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1402068863618,
"mem" : {
"heap_init_in_bytes" : 524288000,
"heap_max_in_bytes" : 506855424,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 506855424
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "200"
},
"get" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 96,
"max" : 96,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 10,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000,
"primary_interface" : {
"address" : "",
"name" : "",
"mac_address" : ""
}
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9303]",
"publish_address" : "inet[/10.0.108.171:9303]"
},
"plugins" : [ ]
},
"rPZ9EsahRl-9Cs_AOucOJQ" : {
"name" : "logstash-netflow",
"transport_address" : "inet[/10.0.108.171:9302]",
"host" : "node38",
"ip" : "10.0.108.171",
"version" : "1.1.1",
"build" : "f1585f0",
"attributes" : {
"client" : "true",
"data" : "false"
},
"settings" : {
"path" : {
"logs" : "/home/sysunix/logs"
},
"cluster" : {
"name" : "telecom"
},
"node" : {
"client" : "true",
"name" : "logstash-netflow"
},
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : "node38:9300,node38:9301,node38:9302,node38:9303,node38:9304,node38:9305"
},
"multicast" : {
"enabled" : "false"
}
}
}
},
"http" : {
"enabled" : "false"
},
"name" : "logstash-netflow"
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 32
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 8365,
"max_file_descriptors" : 4096,
"mlockall" : false
},
"jvm" : {
"pid" : 8365,
"version" : "1.7.0_55",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.51-b03",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1402068863618,
"mem" : {
"heap_init_in_bytes" : 524288000,
"heap_max_in_bytes" : 506855424,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 506855424
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "200"
},
"get" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 96,
"max" : 96,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 10,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000,
"primary_interface" : {
"address" : "",
"name" : "",
"mac_address" : ""
}
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9302]",
"publish_address" : "inet[/10.0.108.171:9302]"
},
"plugins" : [ ]
},
"myfhcxbYRCaqpJ_yAapmHw" : {
"name" : "node38-telecom",
"transport_address" : "inet[/10.0.108.171:9300]",
"host" : "node38",
"ip" : "10.0.108.171",
"version" : "1.3.0",
"build" : "1265b14",
"http_address" : "inet[/10.0.108.171:9200]",
"settings" : {
"node" : {
"name" : "node38-telecom"
},
"index" : {
"number_of_replicas" : "1",
"number_of_shards" : "8"
},
"bootstrap" : {
"mlockall" : "true"
},
"name" : "node38-telecom",
"pidfile" : "/var/run/elasticsearch/elasticsearch-telecom.pid",
"path" : {
"data" : "/var/lib/elasticsearch/telecom",
"work" : "/tmp/elasticsearch",
"home" : "/usr/share/elasticsearch",
"conf" : "/etc/elasticsearch/telecom",
"logs" : "/var/log/elasticsearch/telecom"
},
"cluster" : {
"name" : "telecom"
},
"indices" : {
"memory" : {
"index_buffer_size" : "30%"
}
},
"discovery" : {
"zen" : {
"minimum_master_nodes" : "1",
"ping" : {
"unicast" : {
"hosts" : [ "node07", "node38" ]
},
"multicast" : {
"enabled" : "false"
},
"timeout" : "30s"
}
}
}
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 32,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 2000,
"total_cores" : 32,
"total_sockets" : 1,
"cores_per_socket" : 32,
"cache_size_in_bytes" : 20480
},
"mem" : {
"total_in_bytes" : 33617100800
},
"swap" : {
"total_in_bytes" : 17459511296
}
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 12731,
"max_file_descriptors" : 65535,
"mlockall" : false
},
"jvm" : {
"pid" : 12731,
"version" : "1.7.0_65",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.65-b04",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1406268268972,
"mem" : {
"heap_init_in_bytes" : 17179869184,
"heap_max_in_bytes" : 16979263488,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 16979263488
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "200"
},
"snapshot_data" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"bench" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"get" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 96,
"max" : 96,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 10,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000,
"primary_interface" : {
"address" : "10.0.108.171",
"name" : "eth0",
"mac_address" : ""
}
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
"publish_address" : "inet[/10.0.108.171:9300]"
},
"http" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
"publish_address" : "inet[/10.0.108.171:9200]",
"max_content_length_in_bytes" : 104857600
},
"plugins" : [ ]
},
"9FwiLpriR12926UVB-YuVw" : {
"name" : "logstash-netflow",
"transport_address" : "inet[/10.0.108.171:9301]",
"host" : "node38",
"ip" : "10.0.108.171",
"version" : "1.1.1",
"build" : "f1585f0",
"attributes" : {
"client" : "true",
"data" : "false"
},
"settings" : {
"path" : {
"logs" : "/home/sysunix/logs"
},
"cluster" : {
"name" : "telecom"
},
"node" : {
"client" : "true",
"name" : "logstash-netflow"
},
"discovery" : {
"zen" : {
"ping" : {
"unicast" : {
"hosts" : "node38:9300,node38:9301,node38:9302,node38:9303,node38:9304,node38:9305"
},
"multicast" : {
"enabled" : "false"
}
}
}
},
"http" : {
"enabled" : "false"
},
"name" : "logstash-netflow"
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 32
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 8365,
"max_file_descriptors" : 4096,
"mlockall" : false
},
"jvm" : {
"pid" : 8365,
"version" : "1.7.0_55",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "24.51-b03",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1402068863618,
"mem" : {
"heap_init_in_bytes" : 524288000,
"heap_max_in_bytes" : 506855424,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 224395264,
"direct_max_in_bytes" : 506855424
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "200"
},
"get" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 96,
"max" : 96,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 32,
"max" : 32,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 10,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000,
"primary_interface" : {
"address" : "",
"name" : "",
"mac_address" : ""
}
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9301]",
"publish_address" : "inet[/10.0.108.171:9301]"
},
"plugins" : [ ]
}
}
}
curl 0:9200/_nodes/stats
{
"cluster_name" : "telecom",
"nodes" : {
"JUOZAr-mT6e8e9nmwWH9ww" : {
"timestamp" : 1406277844538,
"name" : "node07-telecom",
"transport_address" : "inet[/10.0.104.214:9300]",
"host" : "node07",
"ip" : [ "inet[/10.0.104.214:9300]", "NONE" ],
"indices" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 0,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
"index_time_in_millis" : 0,
"index_current" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time_in_millis" : 0,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 0,
"total_time_in_millis" : 0,
"total_docs" : 0,
"total_size_in_bytes" : 0
},
"refresh" : {
"total" : 0,
"total_time_in_millis" : 0
},
"flush" : {
"total" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 0,
"total_time_in_millis" : 0
},
"filter_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 0,
"memory_in_bytes" : 0,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0
},
"translog" : {
"operations" : 0,
"size_in_bytes" : 0
},
"suggest" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0
}
},
"os" : {
"timestamp" : 1406277844539,
"uptime_in_millis" : 2050688,
"load_average" : [ 0.0, 0.0, 0.0 ],
"cpu" : {
"sys" : 0,
"user" : 0,
"idle" : 99,
"usage" : 0,
"stolen" : 0
},
"mem" : {
"free_in_bytes" : 13301526528,
"used_in_bytes" : 11883552768,
"free_percent" : 94,
"used_percent" : 5,
"actual_free_in_bytes" : 23725084672,
"actual_used_in_bytes" : 1459994624
},
"swap" : {
"used_in_bytes" : 26796032,
"free_in_bytes" : 252854272
}
},
"process" : {
"timestamp" : 1406277844540,
"open_file_descriptors" : 464,
"cpu" : {
"percent" : 0,
"sys_in_millis" : 17990,
"user_in_millis" : 33500,
"total_in_millis" : 51490
},
"mem" : {
"resident_in_bytes" : 616067072,
"share_in_bytes" : 13942784,
"total_virtual_in_bytes" : 26445426688
}
},
"jvm" : {
"timestamp" : 1406277844540,
"uptime_in_millis" : 9502740,
"mem" : {
"heap_used_in_bytes" : 575853064,
"heap_used_percent" : 3,
"heap_committed_in_bytes" : 17066491904,
"heap_max_in_bytes" : 17066491904,
"non_heap_used_in_bytes" : 31055048,
"non_heap_committed_in_bytes" : 32374784,
"pools" : {
"young" : {
"used_in_bytes" : 546352176,
"max_in_bytes" : 907345920,
"peak_used_in_bytes" : 907345920,
"peak_max_in_bytes" : 907345920
},
"survivor" : {
"used_in_bytes" : 29500888,
"max_in_bytes" : 113377280,
"peak_used_in_bytes" : 29500888,
"peak_max_in_bytes" : 113377280
},
"old" : {
"used_in_bytes" : 0,
"max_in_bytes" : 16045768704,
"peak_used_in_bytes" : 0,
"peak_max_in_bytes" : 16045768704
}
}
},
"threads" : {
"count" : 112,
"peak_count" : 115
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 1,
"collection_time_in_millis" : 58
},
"old" : {
"collection_count" : 0,
"collection_time_in_millis" : 0
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 179,
"used_in_bytes" : 49807360,
"total_capacity_in_bytes" : 49807360
},
"mapped" : {
"count" : 0,
"used_in_bytes" : 0,
"total_capacity_in_bytes" : 0
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 4,
"completed" : 972
},
"index" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot_data" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bench" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"flush" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"search" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 1,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 1,
"completed" : 653
},
"refresh" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
}
},
"network" : {
"tcp" : {
"active_opens" : 421209,
"passive_opens" : 138326,
"curr_estab" : 132,
"in_segs" : 110881241,
"out_segs" : 114912697,
"retrans_segs" : 105,
"estab_resets" : 253951,
"attempt_fails" : 4321,
"in_errs" : 0,
"out_rsts" : 68724493
}
},
"fs" : {
"timestamp" : 1406277844541,
"total" : {
"total_in_bytes" : 1648462135296,
"free_in_bytes" : 1632006266880,
"available_in_bytes" : 1632006266880,
"disk_reads" : 11399,
"disk_writes" : 3156978,
"disk_io_op" : 3168377,
"disk_read_size_in_bytes" : 515517952,
"disk_write_size_in_bytes" : 302869040640,
"disk_io_size_in_bytes" : 303384558592,
"disk_queue" : "1.5E-4",
"disk_service_time" : "0.1"
},
"data" : [ {
"path" : "/var/lib/elasticsearch/telecom/telecom/nodes/0",
"mount" : "/var/lib/elasticsearch",
"dev" : "/dev/mapper/rootvg-elasticsearch",
"total_in_bytes" : 1648462135296,
"free_in_bytes" : 1632006266880,
"available_in_bytes" : 1632006266880,
"disk_reads" : 11399,
"disk_writes" : 3156978,
"disk_io_op" : 3168377,
"disk_read_size_in_bytes" : 515517952,
"disk_write_size_in_bytes" : 302869040640,
"disk_io_size_in_bytes" : 303384558592,
"disk_queue" : "1.5E-4",
"disk_service_time" : "0.1"
} ]
},
"transport" : {
"server_open" : 65,
"rx_count" : 19014,
"rx_size_in_bytes" : 934110,
"tx_count" : 19013,
"tx_size_in_bytes" : 1151504
},
"http" : {
"current_open" : 0,
"total_opened" : 643
},
"fielddata_breaker" : {
"maximum_size_in_bytes" : 10239895142,
"maximum_size" : "9.5gb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.03,
"tripped" : 0
}
},
"JL38-jS9Sn67hQk2XxNZxw" : {
"timestamp" : 1406277844542,
"name" : "logstash-netflow",
"transport_address" : "inet[/10.0.108.171:9303]",
"host" : "node38",
"ip" : [ "inet[/10.0.108.171:9303]", "NONE" ],
"attributes" : {
"client" : "true",
"data" : "false"
},
"indices" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 0,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
"index_time_in_millis" : 0,
"index_current" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time_in_millis" : 0,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 0,
"total_time_in_millis" : 0,
"total_docs" : 0,
"total_size_in_bytes" : 0
},
"refresh" : {
"total" : 0,
"total_time_in_millis" : 0
},
"flush" : {
"total" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 0,
"total_time_in_millis" : 0
},
"filter_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 0,
"memory_in_bytes" : 0,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0
},
"translog" : {
"operations" : 0,
"size_in_bytes" : 0
}
},
"os" : {
"timestamp" : 1406277844542
},
"process" : {
"timestamp" : 1406277844542,
"open_file_descriptors" : 1377
},
"jvm" : {
"timestamp" : 1406277844544,
"uptime_in_millis" : 4208980926,
"mem" : {
"heap_used_in_bytes" : 226119440,
"heap_used_percent" : 44,
"heap_committed_in_bytes" : 506855424,
"heap_max_in_bytes" : 506855424,
"non_heap_used_in_bytes" : 68490160,
"non_heap_committed_in_bytes" : 107167744,
"pools" : {
"young" : {
"used_in_bytes" : 46953832,
"max_in_bytes" : 139853824,
"peak_used_in_bytes" : 139853824,
"peak_max_in_bytes" : 139853824
},
"survivor" : {
"used_in_bytes" : 110032,
"max_in_bytes" : 17432576,
"peak_used_in_bytes" : 17432576,
"peak_max_in_bytes" : 17432576
},
"old" : {
"used_in_bytes" : 179055576,
"max_in_bytes" : 349569024,
"peak_used_in_bytes" : 339622256,
"peak_max_in_bytes" : 349569024
}
}
},
"threads" : {
"count" : 428,
"peak_count" : 442
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 11840,
"collection_time_in_millis" : 67781
},
"old" : {
"collection_count" : 14,
"collection_time_in_millis" : 203
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 723,
"used_in_bytes" : 199419308,
"total_capacity_in_bytes" : 199419308
},
"mapped" : {
"count" : 0,
"used_in_bytes" : 0,
"total_capacity_in_bytes" : 0
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 4,
"completed" : 420988
},
"index" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"flush" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"search" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 1,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 2,
"completed" : 766
},
"refresh" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
}
},
"network" : { },
"fs" : {
"timestamp" : 1406277844544,
"total" : { },
"data" : [ ]
},
"transport" : {
"server_open" : 26,
"rx_count" : 9160629,
"rx_size_in_bytes" : 2570548643,
"tx_count" : 9160591,
"tx_size_in_bytes" : 1643358255
},
"fielddata_breaker" : {
"maximum_size_in_bytes" : 405484339,
"maximum_size" : "386.6mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.03,
"tripped" : -1
}
},
"rPZ9EsahRl-9Cs_AOucOJQ" : {
"timestamp" : 1406277844542,
"name" : "logstash-netflow",
"transport_address" : "inet[/10.0.108.171:9302]",
"host" : "node38",
"ip" : [ "inet[/10.0.108.171:9302]", "NONE" ],
"attributes" : {
"client" : "true",
"data" : "false"
},
"indices" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 0,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
"index_time_in_millis" : 0,
"index_current" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time_in_millis" : 0,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 0,
"total_time_in_millis" : 0,
"total_docs" : 0,
"total_size_in_bytes" : 0
},
"refresh" : {
"total" : 0,
"total_time_in_millis" : 0
},
"flush" : {
"total" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 0,
"total_time_in_millis" : 0
},
"filter_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 0,
"memory_in_bytes" : 0,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0
},
"translog" : {
"operations" : 0,
"size_in_bytes" : 0
}
},
"os" : {
"timestamp" : 1406277844543
},
"process" : {
"timestamp" : 1406277844543,
"open_file_descriptors" : 1375
},
"jvm" : {
"timestamp" : 1406277844544,
"uptime_in_millis" : 4208980926,
"mem" : {
"heap_used_in_bytes" : 226123632,
"heap_used_percent" : 44,
"heap_committed_in_bytes" : 506855424,
"heap_max_in_bytes" : 506855424,
"non_heap_used_in_bytes" : 68490160,
"non_heap_committed_in_bytes" : 107167744,
"pools" : {
"young" : {
"used_in_bytes" : 46960112,
"max_in_bytes" : 139853824,
"peak_used_in_bytes" : 139853824,
"peak_max_in_bytes" : 139853824
},
"survivor" : {
"used_in_bytes" : 110032,
"max_in_bytes" : 17432576,
"peak_used_in_bytes" : 17432576,
"peak_max_in_bytes" : 17432576
},
"old" : {
"used_in_bytes" : 179055576,
"max_in_bytes" : 349569024,
"peak_used_in_bytes" : 339622256,
"peak_max_in_bytes" : 349569024
}
}
},
"threads" : {
"count" : 428,
"peak_count" : 442
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 11840,
"collection_time_in_millis" : 67781
},
"old" : {
"collection_count" : 14,
"collection_time_in_millis" : 203
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 723,
"used_in_bytes" : 199419308,
"total_capacity_in_bytes" : 199419308
},
"mapped" : {
"count" : 0,
"used_in_bytes" : 0,
"total_capacity_in_bytes" : 0
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 3,
"completed" : 420986
},
"index" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"flush" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"search" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 1,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 2,
"completed" : 766
},
"refresh" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
}
},
"network" : { },
"fs" : {
"timestamp" : 1406277844544,
"total" : { },
"data" : [ ]
},
"transport" : {
"server_open" : 26,
"rx_count" : 9160652,
"rx_size_in_bytes" : 2567605017,
"tx_count" : 9160605,
"tx_size_in_bytes" : 1635812229
},
"fielddata_breaker" : {
"maximum_size_in_bytes" : 405484339,
"maximum_size" : "386.6mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.03,
"tripped" : -1
}
},
"myfhcxbYRCaqpJ_yAapmHw" : {
"timestamp" : 1406277844542,
"name" : "node38-telecom",
"transport_address" : "inet[/10.0.108.171:9300]",
"host" : "node38",
"ip" : [ "inet[/10.0.108.171:9300]", "NONE" ],
"indices" : {
"docs" : {
"count" : 33434392,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 16439595838,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
"index_time_in_millis" : 0,
"index_current" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time_in_millis" : 0,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 0,
"total_time_in_millis" : 0,
"total_docs" : 0,
"total_size_in_bytes" : 0
},
"refresh" : {
"total" : 26,
"total_time_in_millis" : 0
},
"flush" : {
"total" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 52,
"total_time_in_millis" : 2
},
"filter_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 331,
"memory_in_bytes" : 380822712,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0
},
"translog" : {
"operations" : 0,
"size_in_bytes" : 0
},
"suggest" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0
}
},
"os" : {
"timestamp" : 1406277844605,
"uptime_in_millis" : 11125608,
"load_average" : [ 0.02, 0.02, 0.0 ],
"cpu" : {
"sys" : 0,
"user" : 0,
"idle" : 99,
"usage" : 0,
"stolen" : 0
},
"mem" : {
"free_in_bytes" : 9554894848,
"used_in_bytes" : 24062205952,
"free_percent" : 85,
"used_percent" : 14,
"actual_free_in_bytes" : 28848246784,
"actual_used_in_bytes" : 4768854016
},
"swap" : {
"used_in_bytes" : 306176000,
"free_in_bytes" : 17153335296
}
},
"process" : {
"timestamp" : 1406277844605,
"open_file_descriptors" : 1796,
"cpu" : {
"percent" : 1,
"sys_in_millis" : 64390,
"user_in_millis" : 113440,
"total_in_millis" : 177830
},
"mem" : {
"resident_in_bytes" : 2292084736,
"share_in_bytes" : 16846848,
"total_virtual_in_bytes" : 38485655552
}
},
"jvm" : {
"timestamp" : 1406277844607,
"uptime_in_millis" : 9575635,
"mem" : {
"heap_used_in_bytes" : 1142720912,
"heap_used_percent" : 6,
"heap_committed_in_bytes" : 16979263488,
"heap_max_in_bytes" : 16979263488,
"non_heap_used_in_bytes" : 39112624,
"non_heap_committed_in_bytes" : 39387136,
"pools" : {
"young" : {
"used_in_bytes" : 978794664,
"max_in_bytes" : 1605304320,
"peak_used_in_bytes" : 1605304320,
"peak_max_in_bytes" : 1605304320
},
"survivor" : {
"used_in_bytes" : 3106256,
"max_in_bytes" : 200605696,
"peak_used_in_bytes" : 166923624,
"peak_max_in_bytes" : 200605696
},
"old" : {
"used_in_bytes" : 160819992,
"max_in_bytes" : 15173353472,
"peak_used_in_bytes" : 160819992,
"peak_max_in_bytes" : 15173353472
}
}
},
"threads" : {
"count" : 214,
"peak_count" : 229
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 7,
"collection_time_in_millis" : 403
},
"old" : {
"collection_count" : 0,
"collection_time_in_millis" : 0
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 336,
"used_in_bytes" : 87049216,
"total_capacity_in_bytes" : 87049216
},
"mapped" : {
"count" : 440,
"used_in_bytes" : 3388615670,
"total_capacity_in_bytes" : 3388615670
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 19,
"completed" : 1054
},
"index" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot_data" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bench" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 1,
"completed" : 26
},
"flush" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"search" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 5,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 5,
"completed" : 1354
},
"refresh" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
}
},
"network" : {
"tcp" : {
"active_opens" : 140097716,
"passive_opens" : 1880178,
"curr_estab" : 290,
"in_segs" : 5172144515,
"out_segs" : 5814393262,
"retrans_segs" : 212947,
"estab_resets" : 1056639,
"attempt_fails" : 27282,
"in_errs" : 50905,
"out_rsts" : 1112514
}
},
"fs" : {
"timestamp" : 1406277844608,
"total" : {
"total_in_bytes" : 2197949513728,
"free_in_bytes" : 2181463736320,
"available_in_bytes" : 2181463736320,
"disk_reads" : 38753157,
"disk_writes" : 243000636,
"disk_io_op" : 281753793,
"disk_read_size_in_bytes" : 4230850646528,
"disk_write_size_in_bytes" : 13743848959488,
"disk_io_size_in_bytes" : 17974699606016
},
"data" : [ {
"path" : "/var/lib/elasticsearch/telecom/telecom/nodes/0",
"mount" : "/var/lib/elasticsearch",
"dev" : "/dev/mapper/raidvg-elasticsearch",
"total_in_bytes" : 2197949513728,
"free_in_bytes" : 2181463736320,
"available_in_bytes" : 2181463736320,
"disk_reads" : 38753157,
"disk_writes" : 243000636,
"disk_io_op" : 281753793,
"disk_read_size_in_bytes" : 4230850646528,
"disk_write_size_in_bytes" : 13743848959488,
"disk_io_size_in_bytes" : 17974699606016
} ]
},
"transport" : {
"server_open" : 65,
"rx_count" : 76220,
"rx_size_in_bytes" : 4624394,
"tx_count" : 76219,
"tx_size_in_bytes" : 3602292
},
"http" : {
"current_open" : 1,
"total_opened" : 655
},
"fielddata_breaker" : {
"maximum_size_in_bytes" : 10187558092,
"maximum_size" : "9.4gb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.03,
"tripped" : 0
}
},
"9FwiLpriR12926UVB-YuVw" : {
"timestamp" : 1406277844542,
"name" : "logstash-netflow",
"transport_address" : "inet[/10.0.108.171:9301]",
"host" : "node38",
"ip" : [ "inet[/10.0.108.171:9301]", "NONE" ],
"attributes" : {
"client" : "true",
"data" : "false"
},
"indices" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 0,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
"index_time_in_millis" : 0,
"index_current" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 0,
"query_time_in_millis" : 0,
"query_current" : 0,
"fetch_total" : 0,
"fetch_time_in_millis" : 0,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 0,
"total_time_in_millis" : 0,
"total_docs" : 0,
"total_size_in_bytes" : 0
},
"refresh" : {
"total" : 0,
"total_time_in_millis" : 0
},
"flush" : {
"total" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 0,
"total_time_in_millis" : 0
},
"filter_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 0,
"memory_in_bytes" : 0,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0
},
"translog" : {
"operations" : 0,
"size_in_bytes" : 0
}
},
"os" : {
"timestamp" : 1406277844543
},
"process" : {
"timestamp" : 1406277844543,
"open_file_descriptors" : 1376
},
"jvm" : {
"timestamp" : 1406277844544,
"uptime_in_millis" : 4208980926,
"mem" : {
"heap_used_in_bytes" : 226119440,
"heap_used_percent" : 44,
"heap_committed_in_bytes" : 506855424,
"heap_max_in_bytes" : 506855424,
"non_heap_used_in_bytes" : 68490160,
"non_heap_committed_in_bytes" : 107167744,
"pools" : {
"young" : {
"used_in_bytes" : 46955928,
"max_in_bytes" : 139853824,
"peak_used_in_bytes" : 139853824,
"peak_max_in_bytes" : 139853824
},
"survivor" : {
"used_in_bytes" : 110032,
"max_in_bytes" : 17432576,
"peak_used_in_bytes" : 17432576,
"peak_max_in_bytes" : 17432576
},
"old" : {
"used_in_bytes" : 179055576,
"max_in_bytes" : 349569024,
"peak_used_in_bytes" : 339622256,
"peak_max_in_bytes" : 349569024
}
}
},
"threads" : {
"count" : 428,
"peak_count" : 442
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 11840,
"collection_time_in_millis" : 67781
},
"old" : {
"collection_count" : 14,
"collection_time_in_millis" : 203
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 723,
"used_in_bytes" : 199419308,
"total_capacity_in_bytes" : 199419308
},
"mapped" : {
"count" : 0,
"used_in_bytes" : 0,
"total_capacity_in_bytes" : 0
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 4,
"completed" : 420979
},
"index" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"flush" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"search" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 1,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 2,
"completed" : 766
},
"refresh" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
}
},
"network" : { },
"fs" : {
"timestamp" : 1406277844544,
"total" : { },
"data" : [ ]
},
"transport" : {
"server_open" : 26,
"rx_count" : 8688419,
"rx_size_in_bytes" : 2179836154,
"tx_count" : 8688418,
"tx_size_in_bytes" : 486036255
},
"fielddata_breaker" : {
"maximum_size_in_bytes" : 405484339,
"maximum_size" : "386.6mb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.03,
"tripped" : -1
}
}
}
}
thanks @faxm0dem
@drewr could you take another look at this with the new info that has been provided please?
Sorry, just now seeing this... @faxm0dem did anything else anomalous happen in the cluster before this? Like, nodes not communicating properly, or running out of mem, anything like that?
Not really. I just upgraded my prod cluster to 1.3.2, and have the same issue.
Getting same error after connecting with logstash. Its working fine as single node and in cluster mode without logstash. Tried with ES 1.3.2 and 1.24 with logstash 1.4.2. is there any workaround for same? But data indexing and query is working fine.
I don't have logstash nodes in my devel cluster, and I still get the error
Fixed by #7815.
Just to confirm - was still seeing this in 1.3.4, but upgraded to 1.4.0 today and it is now working correctly. Thanks! Matthew
I confirm this is finally working!
I upgraded to 1.4.0 today and now I see this error.
http://localhost:9200/_aliases?pretty=1 { "error" : "RemoteTransportException[[Raza][inet[/123.456.17.4:9300]][indices:admin/get]]; nested: ActionNotFoundTransportException[No handler for action [indices:admin/get]]; ", "status" : 500 }
http://0:9200/_cat/indices {"error":"NullPointerException[null]","status":500}
However I am able to search indices directly
http://localhost:9200/testindex/_search? {"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
@Grauen are all your nodes 1.4.0?
@Grauen This sounds like a bug that was fixed in https://github.com/elasticsearch/elasticsearch/pull/8387. if you are still seeing this problems when all of your nodes are running 1.4.0, then please open a new issue.
curl -s 'http://localhost:9200/_cat/nodes' {"error":"NullPointerException[null]","status":500}
Nothing else is logged on the masters or data nodes. We have Logstash 1.4.1 connected to the cluster, but this happens regardless of whether the Logstash nodes are connected or not.