search

lmenezes / cerebro

MIT License
5.53k stars 717 forks source link

Cerebro shows only 3 out of 5 ES nodes #91

Closed trondhindenes closed 7 years ago

trondhindenes commented 7 years ago

We have a setup with 5 ES nodes in the same cluster, 3 of which are load-balanced behind nginx. The 2 last ones are not part of the load-balanced setup. Using cerebro and pointing it to the nginx shows only 3 out of 5 nodes. Not sure if that's because those are the only ones accessuble thru nginx or if there's something else going on.

Since Cerebro performs ES queries using its backend and not simple js json calls against ES, its hard for me to say what's going on. So I guess:

  1. Is there a requirement for cerebro to be able to directly connect to every single node in the cluster?
  2. If not, what should be my first stop in troubleshooting this?
lmenezes commented 7 years ago

@trondhindenes I imagine you mean you can only see 3 out of the 5 nodes in the overview, right? If thats the case, could it be that 2 of your 5 nodes are not data nodes? In this case this would be correct, as the overview shows shard allocation and non master nodes cannot host shards. On the latest version there is also a nodes page, there you should be able to see all nodes. If this is not the case, please re-open the issue :)

trondhindenes commented 7 years ago

Thanks for replying. Turns out I wasn't on the recent version so I updated but I'm still seeing the same. Here's my "nodes" view, which shows all nodes: 2017-03-10 16_06_36-cerebro rikstv-prod-es

And here's the overview where nodes are missing. All my nodes are data nodes. 2017-03-10 16_06_49-cerebro rikstv-prod-es

lmenezes commented 7 years ago

Can you paste the output of running "/_cat/nodes?v" on your cluster?

marke72 commented 7 years ago

The nodes page shows your nodes "es11" and "es12" are just master nodes. Check your configuration for those two nodes and make sure that node.data isn't set to false if you do mean for them to be data nodes. Their CPU usage is quite low compared to the other three which does make me believe that node.data is set to false for those two.

trondhindenes commented 7 years ago

They definetely have data on them. cat output: host ip heap.percent ram.percent load node.role master name
10.244.13.164 10.244.13.164 46 99 -1.00 d m es05 10.244.13.165 10.244.13.165 46 94 -1.00 d m es06 10.244.13.167 10.244.13.167 60 94 -1.00 d m es07 10.244.13.169 10.244.13.169 31 48 -1.00 d m es11 10.244.13.170 10.244.13.170 30 38 -1.00 d * es12

The 11 and 12 nodes are "cold data" nodes using spinnig disks rather than ssds for long-term storage of old logstash indexes, but they definetely have data.

lmenezes commented 7 years ago

@trondhindenes could I have the output of /_nodes ? Feel free to obscure any private date from it :)

lmenezes commented 7 years ago

@trondhindenes is this still an issue?

trondhindenes commented 7 years ago

it is. Here's the node info from my cluster:

{
  "cluster_name": "stuff-prod-es",
  "nodes": {
    "OTfpV0vVT_GqaPRkAwg7ww": {
      "name": "zzz-prod-es07",
      "transport_address": "10.244.13.167:9300",
      "host": "10.244.13.167",
      "ip": "10.244.13.167",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.244.13.167:9200",
      "attributes": {
        "node.type": "hot"
      },
      "settings": {
        "cluster": {
          "name": "stuff-prod-es"
        },
        "node": {
          "name": "zzz-prod-es07",
          "node": {
            "type": "hot"
          }
        },
        "path": {
          "conf": "F:\\ElasticSearch\\elasticsearch-2.4.1\\config",
          "data": "F:/Elasticsearch/data",
          "logs": "F:/ElasticSearch/elasticsearch-2.4.1/logs",
          "home": "F:\\ElasticSearch\\elasticsearch-2.4.1"
        },
        "indices": {
          "memory": {
            "index_buffer_size": "50%"
          }
        },
        "discovery": {
          "zen": {
            "minimum_master_nodes": "2",
            "ping": {
              "unicast": {
                "hosts": [
                  "zzz-prod-es05",
                  "zzz-prod-es06",
                  "zzz-prod-es07"
                ]
              }
            }
          }
        },
        "name": "zzz-prod-es07",
        "client": {
          "type": "node"
        },
        "config": {
          "ignore_system_properties": "true"
        },
        "network": {
          "host": "_site_",
          "bind_host": "0.0.0.0"
        }
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Windows Server 2012 R2",
        "arch": "amd64",
        "version": "6.3",
        "available_processors": 2,
        "allocated_processors": 2
      },
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 1240,
        "mlockall": false
      },
      "jvm": {
        "pid": 1240,
        "version": "1.8.0_73",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.73-b02",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1489624004077,
        "mem": {
          "heap_init_in_bytes": 9663676416,
          "heap_max_in_bytes": 9646243840,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 9646243840
        },
        "gc_collectors": [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "Compressed Class Space",
          "Par Eden Space",
          "Par Survivor Space",
          "CMS Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "true"
      },
      "thread_pool": {
        "force_merge": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "percolate": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "fetch_shard_started": {
          "type": "scaling",
          "min": 1,
          "max": 4,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "index": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 200
        },
        "refresh": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "suggest": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "generic": {
          "type": "cached",
          "keep_alive": "30s",
          "queue_size": -1
        },
        "warmer": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "flush": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
          "type": "scaling",
          "min": 1,
          "max": 4,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "get": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "bulk": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 50
        },
        "snapshot": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        }
      },
      "transport": {
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "10.244.13.167:9300",
        "profiles": {}
      },
      "http": {
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "10.244.13.167:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
          "name": "cloud-aws",
          "version": "2.4.1",
          "description": "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
          "jvm": true,
          "classname": "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "head",
          "version": "master",
          "description": "head - A web front end for an elastic search cluster",
          "url": "/_plugin/head/",
          "jvm": false,
          "site": true
        },
        {
          "name": "hq",
          "version": "1.0.0",
          "description": "Elastic-HQ",
          "url": "/_plugin/hq/",
          "jvm": false,
          "site": true
        },
        {
          "name": "mapper-attachments",
          "version": "2.4.1",
          "description": "The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.",
          "jvm": true,
          "classname": "org.elasticsearch.mapper.attachments.MapperAttachmentsPlugin",
          "isolated": true,
          "site": false
        }
      ],
      "modules": [
        {
          "name": "lang-expression",
          "version": "2.4.1",
          "description": "Lucene expressions integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.expression.ExpressionPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "lang-groovy",
          "version": "2.4.1",
          "description": "Groovy scripting integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.groovy.GroovyPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "reindex",
          "version": "2.4.1",
          "description": "_reindex and _update_by_query APIs",
          "jvm": true,
          "classname": "org.elasticsearch.index.reindex.ReindexPlugin",
          "isolated": true,
          "site": false
        }
      ]
    },
    "t6FJD_WOQc6RJUFgmWIGeg": {
      "name": "zzz-prod-es12",
      "transport_address": "10.244.13.170:9300",
      "host": "10.244.13.170",
      "ip": "10.244.13.170",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.244.13.170:9200",
      "attributes": {
        "data": "yes",
        "node.type": "cold"
      },
      "settings": {
        "cloud": {
          "aws": {
            "region": "eu-central"
          }
        },
        "cluster": {
          "name": "stuff-prod-es"
        },
        "node": {
          "data": "yes",
          "name": "zzz-prod-es12",
          "node": {
            "type": "cold"
          }
        },
        "path": {
          "data": "F:/apps/elasticsearch/data",
          "logs": "F:/apps/elasticsearch/elasticsearch-2.4.1/logs",
          "home": "F:\\apps\\elasticsearch\\elasticsearch-2.4.1"
        },
        "indices": {
          "memory": {
            "index_buffer_size": "50%"
          }
        },
        "discovery": {
          "zen": {
            "minimum_master_nodes": "2",
            "ping": {
              "unicast": {
                "hosts": [
                  "zzz-prod-es05",
                  "zzz-prod-es06",
                  "zzz-prod-es07"
                ]
              }
            }
          }
        },
        "name": "zzz-prod-es12",
        "client": {
          "type": "node"
        },
        "config": {
          "ignore_system_properties": "true"
        },
        "network": {
          "host": "_site_",
          "bind_host": "0.0.0.0"
        }
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Windows NT (unknown)",
        "arch": "amd64",
        "version": "10.0",
        "available_processors": 4,
        "allocated_processors": 4
      },
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 2716,
        "mlockall": false
      },
      "jvm": {
        "pid": 2716,
        "version": "1.8.0_73",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.73-b02",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1489686484805,
        "mem": {
          "heap_init_in_bytes": 17179869184,
          "heap_max_in_bytes": 17145004032,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 17145004032
        },
        "gc_collectors": [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "Compressed Class Space",
          "Par Eden Space",
          "Par Survivor Space",
          "CMS Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "true"
      },
      "thread_pool": {
        "force_merge": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "percolate": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "fetch_shard_started": {
          "type": "scaling",
          "min": 1,
          "max": 8,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": -1
        },
        "index": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 200
        },
        "refresh": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "suggest": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "generic": {
          "type": "cached",
          "keep_alive": "30s",
          "queue_size": -1
        },
        "warmer": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
          "type": "fixed",
          "min": 7,
          "max": 7,
          "queue_size": 1000
        },
        "flush": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
          "type": "scaling",
          "min": 1,
          "max": 8,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "get": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "bulk": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 50
        },
        "snapshot": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        }
      },
      "transport": {
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "10.244.13.170:9300",
        "profiles": {}
      },
      "http": {
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "10.244.13.170:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
          "name": "cloud-aws",
          "version": "2.4.1",
          "description": "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
          "jvm": true,
          "classname": "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "head",
          "version": "master",
          "description": "head - A web front end for an elastic search cluster",
          "url": "/_plugin/head/",
          "jvm": false,
          "site": true
        },
        {
          "name": "hq",
          "version": "1.0.0",
          "description": "Elastic-HQ",
          "url": "/_plugin/hq/",
          "jvm": false,
          "site": true
        },
        {
          "name": "mapper-attachments",
          "version": "2.4.1",
          "description": "The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.",
          "jvm": true,
          "classname": "org.elasticsearch.mapper.attachments.MapperAttachmentsPlugin",
          "isolated": true,
          "site": false
        }
      ],
      "modules": [
        {
          "name": "lang-expression",
          "version": "2.4.1",
          "description": "Lucene expressions integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.expression.ExpressionPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "lang-groovy",
          "version": "2.4.1",
          "description": "Groovy scripting integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.groovy.GroovyPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "reindex",
          "version": "2.4.1",
          "description": "_reindex and _update_by_query APIs",
          "jvm": true,
          "classname": "org.elasticsearch.index.reindex.ReindexPlugin",
          "isolated": true,
          "site": false
        }
      ]
    },
    "LtPYSHGJQt6zxEqozp-ofg": {
      "name": "zzz-prod-es05",
      "transport_address": "10.244.13.164:9300",
      "host": "10.244.13.164",
      "ip": "10.244.13.164",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.244.13.164:9200",
      "attributes": {
        "node.type": "hot"
      },
      "settings": {
        "cluster": {
          "name": "stuff-prod-es"
        },
        "node": {
          "name": "zzz-prod-es05",
          "node": {
            "type": "hot"
          }
        },
        "path": {
          "conf": "F:\\ElasticSearch\\elasticsearch-2.4.1\\config",
          "data": "F:/Elasticsearch/data",
          "logs": "F:/ElasticSearch/elasticsearch-2.4.1/logs",
          "home": "F:\\ElasticSearch\\elasticsearch-2.4.1"
        },
        "discovery": {
          "zen": {
            "minimum_master_nodes": "2",
            "ping": {
              "unicast": {
                "hosts": [
                  "zzz-prod-es05",
                  "zzz-prod-es06",
                  "zzz-prod-es07"
                ]
              }
            }
          }
        },
        "name": "zzz-prod-es05",
        "client": {
          "type": "node"
        },
        "config": {
          "ignore_system_properties": "true"
        },
        "network": {
          "host": "_site_",
          "bind_host": "0.0.0.0"
        }
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Windows Server 2012 R2",
        "arch": "amd64",
        "version": "6.3",
        "available_processors": 2,
        "allocated_processors": 2
      },
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 1232,
        "mlockall": false
      },
      "jvm": {
        "pid": 1232,
        "version": "1.8.0_73",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.73-b02",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1489557533368,
        "mem": {
          "heap_init_in_bytes": 9663676416,
          "heap_max_in_bytes": 9646243840,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 9646243840
        },
        "gc_collectors": [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "Compressed Class Space",
          "Par Eden Space",
          "Par Survivor Space",
          "CMS Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "true"
      },
      "thread_pool": {
        "force_merge": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "percolate": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "fetch_shard_started": {
          "type": "scaling",
          "min": 1,
          "max": 4,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "index": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 200
        },
        "refresh": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "suggest": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "generic": {
          "type": "cached",
          "keep_alive": "30s",
          "queue_size": -1
        },
        "warmer": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "flush": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
          "type": "scaling",
          "min": 1,
          "max": 4,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "get": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "bulk": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 50
        },
        "snapshot": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        }
      },
      "transport": {
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "10.244.13.164:9300",
        "profiles": {}
      },
      "http": {
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "10.244.13.164:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
          "name": "cloud-aws",
          "version": "2.4.1",
          "description": "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
          "jvm": true,
          "classname": "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "elasticsearch-migration",
          "version": "2.0.4",
          "description": "Elasticsearch Migration Helper",
          "url": "/_plugin/elasticsearch-migration/",
          "jvm": false,
          "site": true
        },
        {
          "name": "head",
          "version": "master",
          "description": "head - A web front end for an elastic search cluster",
          "url": "/_plugin/head/",
          "jvm": false,
          "site": true
        },
        {
          "name": "hq",
          "version": "1.0.0",
          "description": "Elastic-HQ",
          "url": "/_plugin/hq/",
          "jvm": false,
          "site": true
        },
        {
          "name": "mapper-attachments",
          "version": "2.4.1",
          "description": "The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.",
          "jvm": true,
          "classname": "org.elasticsearch.mapper.attachments.MapperAttachmentsPlugin",
          "isolated": true,
          "site": false
        }
      ],
      "modules": [
        {
          "name": "lang-expression",
          "version": "2.4.1",
          "description": "Lucene expressions integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.expression.ExpressionPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "lang-groovy",
          "version": "2.4.1",
          "description": "Groovy scripting integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.groovy.GroovyPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "reindex",
          "version": "2.4.1",
          "description": "_reindex and _update_by_query APIs",
          "jvm": true,
          "classname": "org.elasticsearch.index.reindex.ReindexPlugin",
          "isolated": true,
          "site": false
        }
      ]
    },
    "nxtyqYFGQESeMqHvoMRs1Q": {
      "name": "zzz-prod-es06",
      "transport_address": "10.244.13.165:9300",
      "host": "10.244.13.165",
      "ip": "10.244.13.165",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.244.13.165:9200",
      "attributes": {
        "node.type": "hot"
      },
      "settings": {
        "cluster": {
          "name": "stuff-prod-es"
        },
        "node": {
          "name": "zzz-prod-es06",
          "node": {
            "type": "hot"
          }
        },
        "path": {
          "data": "F:/Elasticsearch/data",
          "logs": "F:/ElasticSearch/elasticsearch-2.4.1/logs",
          "home": "F:\\ElasticSearch\\elasticsearch-2.4.1"
        },
        "indices": {
          "memory": {
            "index_buffer_size": "50%"
          }
        },
        "discovery": {
          "zen": {
            "minimum_master_nodes": "2",
            "ping": {
              "unicast": {
                "hosts": [
                  "zzz-prod-es05",
                  "zzz-prod-es06",
                  "zzz-prod-es07"
                ]
              }
            }
          }
        },
        "name": "zzz-prod-es06",
        "client": {
          "type": "node"
        },
        "config": {
          "ignore_system_properties": "true"
        },
        "network": {
          "host": "_site_",
          "bind_host": "0.0.0.0"
        }
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Windows Server 2012 R2",
        "arch": "amd64",
        "version": "6.3",
        "available_processors": 2,
        "allocated_processors": 2
      },
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 1320,
        "mlockall": false
      },
      "jvm": {
        "pid": 1320,
        "version": "1.8.0_73",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.73-b02",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1489632821868,
        "mem": {
          "heap_init_in_bytes": 7516192768,
          "heap_max_in_bytes": 7498760192,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 7498760192
        },
        "gc_collectors": [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "Compressed Class Space",
          "Par Eden Space",
          "Par Survivor Space",
          "CMS Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "true"
      },
      "thread_pool": {
        "force_merge": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "percolate": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "fetch_shard_started": {
          "type": "scaling",
          "min": 1,
          "max": 4,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "index": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 200
        },
        "refresh": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "suggest": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "generic": {
          "type": "cached",
          "keep_alive": "30s",
          "queue_size": -1
        },
        "warmer": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "flush": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
          "type": "scaling",
          "min": 1,
          "max": 4,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "get": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 1000
        },
        "bulk": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": 50
        },
        "snapshot": {
          "type": "scaling",
          "min": 1,
          "max": 1,
          "keep_alive": "5m",
          "queue_size": -1
        }
      },
      "transport": {
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "10.244.13.165:9300",
        "profiles": {}
      },
      "http": {
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "10.244.13.165:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
          "name": "cloud-aws",
          "version": "2.4.1",
          "description": "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
          "jvm": true,
          "classname": "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "head",
          "version": "master",
          "description": "head - A web front end for an elastic search cluster",
          "url": "/_plugin/head/",
          "jvm": false,
          "site": true
        },
        {
          "name": "hq",
          "version": "1.0.0",
          "description": "Elastic-HQ",
          "url": "/_plugin/hq/",
          "jvm": false,
          "site": true
        },
        {
          "name": "mapper-attachments",
          "version": "2.4.1",
          "description": "The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.",
          "jvm": true,
          "classname": "org.elasticsearch.mapper.attachments.MapperAttachmentsPlugin",
          "isolated": true,
          "site": false
        }
      ],
      "modules": [
        {
          "name": "lang-expression",
          "version": "2.4.1",
          "description": "Lucene expressions integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.expression.ExpressionPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "lang-groovy",
          "version": "2.4.1",
          "description": "Groovy scripting integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.groovy.GroovyPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "reindex",
          "version": "2.4.1",
          "description": "_reindex and _update_by_query APIs",
          "jvm": true,
          "classname": "org.elasticsearch.index.reindex.ReindexPlugin",
          "isolated": true,
          "site": false
        }
      ]
    },
    "rtI_0d7sSd2k6-E-Bz6Rhg": {
      "name": "zzz-prod-es11",
      "transport_address": "10.244.13.169:9300",
      "host": "10.244.13.169",
      "ip": "10.244.13.169",
      "version": "2.4.1",
      "build": "c67dc32",
      "http_address": "10.244.13.169:9200",
      "attributes": {
        "data": "yes",
        "node.type": "cold"
      },
      "settings": {
        "cloud": {
          "aws": {
            "region": "eu-central"
          }
        },
        "cluster": {
          "name": "stuff-prod-es"
        },
        "node": {
          "data": "yes",
          "name": "zzz-prod-es11",
          "node": {
            "type": "cold"
          }
        },
        "path": {
          "data": "F:/apps/elasticsearch/data",
          "logs": "F:/apps/elasticsearch/elasticsearch-2.4.1/logs",
          "home": "F:\\apps\\elasticsearch\\elasticsearch-2.4.1"
        },
        "indices": {
          "memory": {
            "index_buffer_size": "50%"
          }
        },
        "discovery": {
          "zen": {
            "minimum_master_nodes": "2",
            "ping": {
              "unicast": {
                "hosts": [
                  "zzz-prod-es05",
                  "zzz-prod-es06",
                  "zzz-prod-es07"
                ]
              }
            }
          }
        },
        "name": "zzz-prod-es11",
        "client": {
          "type": "node"
        },
        "config": {
          "ignore_system_properties": "true"
        },
        "network": {
          "host": "_site_",
          "bind_host": "0.0.0.0"
        }
      },
      "os": {
        "refresh_interval_in_millis": 1000,
        "name": "Windows NT (unknown)",
        "arch": "amd64",
        "version": "10.0",
        "available_processors": 4,
        "allocated_processors": 4
      },
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 3188,
        "mlockall": false
      },
      "jvm": {
        "pid": 3188,
        "version": "1.8.0_73",
        "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
        "vm_version": "25.73-b02",
        "vm_vendor": "Oracle Corporation",
        "start_time_in_millis": 1488917587728,
        "mem": {
          "heap_init_in_bytes": 17179869184,
          "heap_max_in_bytes": 17145004032,
          "non_heap_init_in_bytes": 2555904,
          "non_heap_max_in_bytes": 0,
          "direct_max_in_bytes": 17145004032
        },
        "gc_collectors": [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools": [
          "Code Cache",
          "Metaspace",
          "Compressed Class Space",
          "Par Eden Space",
          "Par Survivor Space",
          "CMS Old Gen"
        ],
        "using_compressed_ordinary_object_pointers": "true"
      },
      "thread_pool": {
        "force_merge": {
          "type": "fixed",
          "min": 1,
          "max": 1,
          "queue_size": -1
        },
        "percolate": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "fetch_shard_started": {
          "type": "scaling",
          "min": 1,
          "max": 8,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "listener": {
          "type": "fixed",
          "min": 2,
          "max": 2,
          "queue_size": -1
        },
        "index": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 200
        },
        "refresh": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "suggest": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "generic": {
          "type": "cached",
          "keep_alive": "30s",
          "queue_size": -1
        },
        "warmer": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "search": {
          "type": "fixed",
          "min": 7,
          "max": 7,
          "queue_size": 1000
        },
        "flush": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "fetch_shard_store": {
          "type": "scaling",
          "min": 1,
          "max": 8,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "management": {
          "type": "scaling",
          "min": 1,
          "max": 5,
          "keep_alive": "5m",
          "queue_size": -1
        },
        "get": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 1000
        },
        "bulk": {
          "type": "fixed",
          "min": 4,
          "max": 4,
          "queue_size": 50
        },
        "snapshot": {
          "type": "scaling",
          "min": 1,
          "max": 2,
          "keep_alive": "5m",
          "queue_size": -1
        }
      },
      "transport": {
        "bound_address": [
          "[::]:9300"
        ],
        "publish_address": "10.244.13.169:9300",
        "profiles": {}
      },
      "http": {
        "bound_address": [
          "[::]:9200"
        ],
        "publish_address": "10.244.13.169:9200",
        "max_content_length_in_bytes": 104857600
      },
      "plugins": [
        {
          "name": "cloud-aws",
          "version": "2.4.1",
          "description": "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.",
          "jvm": true,
          "classname": "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "head",
          "version": "master",
          "description": "head - A web front end for an elastic search cluster",
          "url": "/_plugin/head/",
          "jvm": false,
          "site": true
        },
        {
          "name": "hq",
          "version": "1.0.0",
          "description": "Elastic-HQ",
          "url": "/_plugin/hq/",
          "jvm": false,
          "site": true
        },
        {
          "name": "mapper-attachments",
          "version": "2.4.1",
          "description": "The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.",
          "jvm": true,
          "classname": "org.elasticsearch.mapper.attachments.MapperAttachmentsPlugin",
          "isolated": true,
          "site": false
        }
      ],
      "modules": [
        {
          "name": "lang-expression",
          "version": "2.4.1",
          "description": "Lucene expressions integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.expression.ExpressionPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "lang-groovy",
          "version": "2.4.1",
          "description": "Groovy scripting integration for Elasticsearch",
          "jvm": true,
          "classname": "org.elasticsearch.script.groovy.GroovyPlugin",
          "isolated": true,
          "site": false
        },
        {
          "name": "reindex",
          "version": "2.4.1",
          "description": "_reindex and _update_by_query APIs",
          "jvm": true,
          "classname": "org.elasticsearch.index.reindex.ReindexPlugin",
          "isolated": true,
          "site": false
        }
      ]
    }
  }
}
lmenezes commented 7 years ago

Ok, I see the problem. The nodes you do not see listed have the data property set to "yes". I have actually just found out this was possible, and hence, only accepted a boolean as a valid value for this. I will fix this to also support yes/no as value.

thanks for the feedback :)

trondhindenes commented 7 years ago

ah, sweet. Yep, those are some huge json's - and even Elastic themselves don't always follow their own spec documentation. Anyways, glad you figured it out, it's (cerebro that is) a fantastic app!