Connecting elasticsearch cluster which has auth enabled

[chitenderkumar]

• OS Platform and Distribution: Centos 7
• ElasticHQ Version: 3.3.0
• Elasticsearch Version: 2.3.5
• Python version: 3.6

we have multiple ElasticSearch cluster Running in private subnet. our goal is to integrate all ElasticSearch cluster to single ElasticHQ server. to access the ElasticSearch from public subnet so that Elastic-HQ can connect to it we have defined the proxy rule in Nginx and routing it to ElasticSearch client which is in private subnet. for security reason and compliance issue we have enabled the basic authentication in Nginx for ElasticSearch endpoint. when we tried to connect to ElasticSearch endpoint we received:

cluster connection failed server responded with UNAUTHORISED 2018-04-16 06:44:14,819 ERROR elastichq ConnectionService.create_connection:88 ('Unable to create connection! Server returned 401 - UNAUTHORIZED: http://10.10.3.222:19200', 401)

is it possible to add ElasticSearch endpoint which has auth enabled?

[royrusso]

Hi,

Did you pass along the auth credentials in the connection URL? http://docs.elastichq.org/installation.html#diagnosing-connection-errors

Also, can you paste what's in the log here?

[chitenderkumar]

i did not. i again tried to connect to the ES node by providing the credentials in URL it got connected.

post login i am not able to see any performance data in the Metrics graph.

`--- Logging error --- Traceback (most recent call last): File "/opt/packages/elastic_hq/elasticsearch-HQ/elastichq/model/Task.py", line 94, in run "fs_used_in_bytes": total_in_bytes - available_in_bytes, TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib64/python3.6/logging/init.py", line 992, in emit msg = self.format(record) File "/usr/lib64/python3.6/logging/init.py", line 838, in format return fmt.format(record) File "/usr/lib64/python3.6/logging/init.py", line 575, in format record.message = record.getMessage() File "/usr/lib64/python3.6/logging/init.py", line 338, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/opt/packages/elastic_hq/elasticsearch-HQ/ELASTIC-HQ/lib64/python3.6/site-packages/eventlet/green/thread.py", line 41, in __thread_body func(*args, *kwargs) File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/opt/packages/elastic_hq/elasticsearch-HQ/ELASTIC-HQ/lib64/python3.6/site-packages/eventlet/green/thread.py", line 62, in wrap_bootstrap_inner bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(self._args, **self._kwargs) File "/opt/packages/elastic_hq/elasticsearch-HQ/elastichq/model/Task.py", line 104, in run LOG.error("Error fetching node metrics!", ex) Message: 'Error fetching node metrics!' Arguments: (TypeError("unsupported operand type(s) for -: 'NoneType' and 'NoneType'",),) --- Logging error --- Traceback (most recent call last): File "/opt/packages/elastic_hq/elasticsearch-HQ/elastichq/model/Task.py", line 94, in run "fs_used_in_bytes": total_in_bytes - available_in_bytes, TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib64/python3.6/logging/handlers.py", line 71, in emit if self.shouldRollover(record): File "/usr/lib64/python3.6/logging/handlers.py", line 187, in shouldRollover msg = "%s\n" % self.format(record) File "/usr/lib64/python3.6/logging/init.py", line 838, in format return fmt.format(record) File "/usr/lib64/python3.6/logging/init.py", line 575, in format record.message = record.getMessage() File "/usr/lib64/python3.6/logging/init.py", line 338, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/opt/packages/elastic_hq/elasticsearch-HQ/ELASTIC-HQ/lib64/python3.6/site-packages/eventlet/green/thread.py", line 41, in __thread_body func(*args, *kwargs) File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/opt/packages/elastic_hq/elasticsearch-HQ/ELASTIC-HQ/lib64/python3.6/site-packages/eventlet/green/thread.py", line 62, in wrap_bootstrap_inner bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(self._args, **self._kwargs) File "/opt/packages/elastic_hq/elasticsearch-HQ/elastichq/model/Task.py", line 104, in run LOG.error("Error fetching node metrics!", ex) Message: 'Error fetching node metrics!' Arguments: (TypeError("unsupported operand type(s) for -: 'NoneType' and 'NoneType'",),)`

[royrusso]

Ok. May be a bug/weirdness with Elasticsearch 2.3.5 for that api call. Looks like it’s returning empty values for filesystem stats, which isn’t normal. I’ll look in to it.

[royrusso]

I've added a patch for the next release to address the None-type values being returned from ES. However, there is a bigger issue here in that I don't believe your ES is returning correct data. Can you look at curl -XGET 'http://localhost:9200/_nodes/stats' and paste what it returns?

[chitenderkumar]

{"cluster_name":"jupiter","nodes":{"uoCPfRMFQamBgeFnyyab_Q":{"timestamp":1523896933714,"name":"DN1","transport_address":"172.16.1.10:9300","host":"172.16.1.10","ip":["172.16.1.10:9300","NONE"],"attributes":{"master":"true"},"indices":{"docs":{"count":38414354,"deleted":597319},"store":{"size_in_bytes":27320737282,"throttle_time_in_millis":0},"indexing":{"index_total":45893615,"index_time_in_millis":22821665,"index_current":1245,"index_failed":4542,"delete_total":0,"delete_time_in_millis":0,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":10981094,"query_time_in_millis":6321523,"query_current":0,"fetch_total":5170158,"fetch_time_in_millis":6403190,"fetch_current":0,"scroll_total":5146514,"scroll_time_in_millis":1696898415509,"scroll_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":16059,"total_time_in_millis":17988734,"total_docs":282132240,"total_size_in_bytes":129858031196,"total_stopped_time_in_millis":0,"total_throttled_time_in_millis":8265614,"total_auto_throttle_in_bytes":1045365665},"refresh":{"total":99392,"total_time_in_millis":3849456},"flush":{"total":344,"total_time_in_millis":263132},"warmer":{"current":0,"total":203742,"total_time_in_millis":65345},"query_cache":{"memory_size_in_bytes":6071560,"total_count":9855738,"hit_count":535495,"miss_count":9320243,"cache_size":1791,"cache_count":1791,"evictions":0},"fielddata":{"memory_size_in_bytes":493981208,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":414,"memory_in_bytes":78998881,"terms_memory_in_bytes":70298921,"stored_fields_memory_in_bytes":6385688,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":1644672,"doc_values_memory_in_bytes":669600,"index_writer_memory_in_bytes":0,"index_writer_max_memory_in_bytes":22528000,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":520},"translog":{"operations":0,"size_in_bytes":1892},"suggest":{"total":0,"time_in_millis":0,"current":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0},"recovery":{"current_as_source":0,"current_as_target":0,"throttle_time_in_millis":0}},"os":{"timestamp":1523896933862,"cpu_percent":0,"load_average":0.14,"mem":{"total_in_bytes":33567973376,"free_in_bytes":2016792576,"used_in_bytes":31551180800,"free_percent":6,"used_percent":94},"swap":{"total_in_bytes":0,"free_in_bytes":0,"used_in_bytes":0}},"process":{"timestamp":1523896933863,"open_file_descriptors":1045,"max_file_descriptors":65535,"cpu":{"percent":0,"total_in_millis":61398170},"mem":{"total_virtual_in_bytes":25854144512}},"jvm":{"timestamp":1523896933866,"uptime_in_millis":2189780536,"mem":{"heap_used_in_bytes":5429736080,"heap_used_percent":42,"heap_committed_in_bytes":12815171584,"heap_max_in_bytes":12815171584,"non_heap_used_in_bytes":108841248,"non_heap_committed_in_bytes":110874624,"pools":{"young":{"used_in_bytes":141584376,"max_in_bytes":558432256,"peak_used_in_bytes":558432256,"peak_max_in_bytes":558432256},"survivor":{"used_in_bytes":238272,"max_in_bytes":69730304,"peak_used_in_bytes":69730304,"peak_max_in_bytes":69730304},"old":{"used_in_bytes":5287913432,"max_in_bytes":12187009024,"peak_used_in_bytes":9226754840,"peak_max_in_bytes":12187009024}}},"threads":{"count":97,"peak_count":116},"gc":{"collectors":{"young":{"collection_count":15597,"collection_time_in_millis":668950},"old":{"collection_count":21,"collection_time_in_millis":3514}}},"buffer_pools":{"direct":{"count":167,"used_in_bytes":68618923,"total_capacity_in_bytes":68618923},"mapped":{"count":321,"used_in_bytes":6721070814,"total_capacity_in_bytes":6721070814}}},"thread_pool":{"bulk":{"threads":8,"queue":0,"active":0,"rejected":9,"largest":8,"completed":927683},"fetch_shard_started":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":10,"completed":10},"fetch_shard_store":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":8,"completed":10},"flush":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":3572},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":6,"completed":2407119},"get":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"index":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"listener":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":1559},"management":{"threads":5,"queue":0,"active":1,"rejected":0,"largest":5,"completed":196380},"percolate":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"refresh":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":99401},"search":{"threads":13,"queue":0,"active":0,"rejected":0,"largest":13,"completed":16246127},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":112279}},"fs":{"timestamp":1523896933866,"total":{"total_in_bytes":644233539584,"free_in_bytes":562406752256,"available_in_bytes":562406752256},"data":[{"path":"/elastic_data/elastic/jupiter/nodes/0","mount":"/ (rootfs)","type":"rootfs","total_in_bytes":644233539584,"free_in_bytes":562406752256,"available_in_bytes":562406752256}]},"transport":{"server_open":52,"rx_count":22417475,"rx_size_in_bytes":11564326361,"tx_count":21950864,"tx_size_in_bytes":46398591327},"http":{"current_open":0,"total_opened":2120},"breakers":{"request":{"limit_size_in_bytes":5126068633,"limit_size":"4.7gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":7689102950,"limit_size":"7.1gb","estimated_size_in_bytes":493981208,"estimated_size":"471mb","overhead":1.03,"tripped":0},"parent":{"limit_size_in_bytes":8970620108,"limit_size":"8.3gb","estimated_size_in_bytes":493981208,"estimated_size":"471mb","overhead":1.0,"tripped":0}},"script":{"compilations":3,"cache_evictions":2}},"8h4CnI1pS3-cLoRL4UEXNQ":{"timestamp":1523896933717,"name":"DN2","transport_address":"172.16.1.3:9300","host":"172.16.1.3","ip":["172.16.1.3:9300","NONE"],"indices":{"docs":{"count":37233690,"deleted":483494},"store":{"size_in_bytes":24818979227,"throttle_time_in_millis":0},"indexing":{"index_total":44537994,"index_time_in_millis":18625253,"index_current":215,"index_failed":2192,"delete_total":0,"delete_time_in_millis":0,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":10895320,"query_time_in_millis":5792104,"query_current":0,"fetch_total":5088169,"fetch_time_in_millis":6239722,"fetch_current":0,"scroll_total":5143972,"scroll_time_in_millis":1695656540539,"scroll_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":15313,"total_time_in_millis":15295598,"total_docs":272441587,"total_size_in_bytes":118391995430,"total_stopped_time_in_millis":0,"total_throttled_time_in_millis":7619069,"total_auto_throttle_in_bytes":1100784207},"refresh":{"total":99490,"total_time_in_millis":3227556},"flush":{"total":338,"total_time_in_millis":472108},"warmer":{"current":0,"total":201766,"total_time_in_millis":37323},"query_cache":{"memory_size_in_bytes":3964568,"total_count":7237593,"hit_count":436729,"miss_count":6800864,"cache_size":1702,"cache_count":1702,"evictions":0},"fielddata":{"memory_size_in_bytes":405001380,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":416,"memory_in_bytes":75302496,"terms_memory_in_bytes":67374104,"stored_fields_memory_in_bytes":5943872,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":1427392,"doc_values_memory_in_bytes":557128,"index_writer_memory_in_bytes":0,"index_writer_max_memory_in_bytes":22016000,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":432},"translog":{"operations":0,"size_in_bytes":1849},"suggest":{"total":0,"time_in_millis":0,"current":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0},"recovery":{"current_as_source":0,"current_as_target":0,"throttle_time_in_millis":0}},"os":{"timestamp":1523896933910,"cpu_percent":0,"load_average":0.09,"mem":{"total_in_bytes":33567973376,"free_in_bytes":279203840,"used_in_bytes":33288769536,"free_percent":1,"used_percent":99},"swap":{"total_in_bytes":0,"free_in_bytes":0,"used_in_bytes":0}},"process":{"timestamp":1523896933911,"open_file_descriptors":1043,"max_file_descriptors":65535,"cpu":{"percent":0,"total_in_millis":63634160},"mem":{"total_virtual_in_bytes":25061613568}},"jvm":{"timestamp":1523896933914,"uptime_in_millis":2189780181,"mem":{"heap_used_in_bytes":2870499816,"heap_used_percent":22,"heap_committed_in_bytes":12815171584,"heap_max_in_bytes":12815171584,"non_heap_used_in_bytes":115288784,"non_heap_committed_in_bytes":117919744,"pools":{"young":{"used_in_bytes":146016224,"max_in_bytes":558432256,"peak_used_in_bytes":558432256,"peak_max_in_bytes":558432256},"survivor":{"used_in_bytes":389200,"max_in_bytes":69730304,"peak_used_in_bytes":69730304,"peak_max_in_bytes":69730304},"old":{"used_in_bytes":2724094392,"max_in_bytes":12187009024,"peak_used_in_bytes":9175548944,"peak_max_in_bytes":12187009024}}},"threads":{"count":97,"peak_count":112},"gc":{"collectors":{"young":{"collection_count":14887,"collection_time_in_millis":940254},"old":{"collection_count":15,"collection_time_in_millis":2956}}},"buffer_pools":{"direct":{"count":166,"used_in_bytes":72326232,"total_capacity_in_bytes":72326232},"mapped":{"count":321,"used_in_bytes":5933530202,"total_capacity_in_bytes":5933530202}}},"thread_pool":{"bulk":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":920515},"fetch_shard_started":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":5,"completed":10},"fetch_shard_store":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":9,"completed":10},"flush":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":3180},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":13,"completed":219325},"get":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"index":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"listener":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":15173737},"management":{"threads":5,"queue":0,"active":1,"rejected":0,"largest":5,"completed":269121},"percolate":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"refresh":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":99504},"search":{"threads":13,"queue":0,"active":0,"rejected":0,"largest":13,"completed":16065043},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":111447}},"fs":{"timestamp":1523896933914,"total":{"total_in_bytes":644233539584,"free_in_bytes":554335948800,"available_in_bytes":554335948800},"data":[{"path":"/elastic_data/elastic/jupiter/nodes/0","mount":"/ (rootfs)","type":"rootfs","total_in_bytes":644233539584,"free_in_bytes":554335948800,"available_in_bytes":554335948800}]},"transport":{"server_open":52,"rx_count":51483339,"rx_size_in_bytes":14681148105,"tx_count":49903635,"tx_size_in_bytes":46458594376},"http":{"current_open":0,"total_opened":2054},"breakers":{"request":{"limit_size_in_bytes":5126068633,"limit_size":"4.7gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":7689102950,"limit_size":"7.1gb","estimated_size_in_bytes":405001380,"estimated_size":"386.2mb","overhead":1.03,"tripped":0},"parent":{"limit_size_in_bytes":8970620108,"limit_size":"8.3gb","estimated_size_in_bytes":405001380,"estimated_size":"386.2mb","overhead":1.0,"tripped":0}},"script":{"compilations":9,"cache_evictions":8}},"eo9lwWO3RqC9KpN9uRJZ-w":{"timestamp":1523896933711,"name":"DB","transport_address":"172.16.1.7:9300","host":"172.16.1.7","ip":["172.16.1.7:9300","NONE"],"attributes":{"data":"false","master":"false"},"indices":{"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":0,"throttle_time_in_millis":0},"indexing":{"index_total":0,"index_time_in_millis":0,"index_current":0,"index_failed":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0,"scroll_total":0,"scroll_time_in_millis":0,"scroll_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0,"total_stopped_time_in_millis":0,"total_throttled_time_in_millis":0,"total_auto_throttle_in_bytes":0},"refresh":{"total":0,"total_time_in_millis":0},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":0,"total_time_in_millis":0},"query_cache":{"memory_size_in_bytes":0,"total_count":0,"hit_count":0,"miss_count":0,"cache_size":0,"cache_count":0,"evictions":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":0,"memory_in_bytes":0,"terms_memory_in_bytes":0,"stored_fields_memory_in_bytes":0,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":0,"doc_values_memory_in_bytes":0,"index_writer_memory_in_bytes":0,"index_writer_max_memory_in_bytes":0,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":0},"translog":{"operations":0,"size_in_bytes":0},"suggest":{"total":0,"time_in_millis":0,"current":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0},"recovery":{"current_as_source":0,"current_as_target":0,"throttle_time_in_millis":0}},"os":{"timestamp":1523896933711,"cpu_percent":1,"load_average":0.04,"mem":{"total_in_bytes":8202694656,"free_in_bytes":321605632,"used_in_bytes":7881089024,"free_percent":4,"used_percent":96},"swap":{"total_in_bytes":0,"free_in_bytes":0,"used_in_bytes":0}},"process":{"timestamp":1523896933712,"open_file_descriptors":223,"max_file_descriptors":65535,"cpu":{"percent":0,"total_in_millis":10510370},"mem":{"total_virtual_in_bytes":5915729920}},"jvm":{"timestamp":1523896933714,"uptime_in_millis":979984123,"mem":{"heap_used_in_bytes":67260816,"heap_used_percent":2,"heap_committed_in_bytes":3203792896,"heap_max_in_bytes":3203792896,"non_heap_used_in_bytes":69367648,"non_heap_committed_in_bytes":70496256,"pools":{"young":{"used_in_bytes":47290032,"max_in_bytes":139591680,"peak_used_in_bytes":139591680,"peak_max_in_bytes":139591680},"survivor":{"used_in_bytes":29344,"max_in_bytes":17432576,"peak_used_in_bytes":17432576,"peak_max_in_bytes":17432576},"old":{"used_in_bytes":19941440,"max_in_bytes":3046768640,"peak_used_in_bytes":19941440,"peak_max_in_bytes":3046768640}}},"threads":{"count":32,"peak_count":39},"gc":{"collectors":{"young":{"collection_count":7883,"collection_time_in_millis":87178},"old":{"collection_count":1,"collection_time_in_millis":53}}},"buffer_pools":{"direct":{"count":41,"used_in_bytes":14187746,"total_capacity_in_bytes":14187746},"mapped":{"count":0,"used_in_bytes":0,"total_capacity_in_bytes":0}},"classes":{"current_loaded_count":7160,"total_loaded_count":7160,"total_unloaded_count":0}},"thread_pool":{"bulk":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch_shard_started":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch_shard_store":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"flush":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":6,"completed":1077104},"get":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"index":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"listener":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":1,"completed":14974383},"management":{"threads":1,"queue":0,"active":1,"rejected":0,"largest":3,"completed":17639},"percolate":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"refresh":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"search":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":3384193},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0}},"fs":{"timestamp":1523896933715,"total":{},"data":[]},"transport":{"server_open":52,"rx_count":70092422,"rx_size_in_bytes":95296073571,"tx_count":67050971,"tx_size_in_bytes":11133540067},"http":{"current_open":6,"total_opened":1552069},"breakers":{"request":{"limit_size_in_bytes":1281517158,"limit_size":"1.1gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":1922275737,"limit_size":"1.7gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.03,"tripped":0},"parent":{"limit_size_in_bytes":2242655027,"limit_size":"2gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0}},"script":{"compilations":0,"cache_evictions":0}},"9nVfSjfFTJe57joT7NF6XQ":{"timestamp":1523896933716,"name":"DEVELOP-DB","transport_address":"172.16.1.14:9300","host":"172.16.1.14","ip":["172.16.1.14:9300","NONE"],"attributes":{"data":"false","master":"false"},"indices":{"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":0,"throttle_time_in_millis":0},"indexing":{"index_total":0,"index_time_in_millis":0,"index_current":0,"index_failed":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0,"scroll_total":0,"scroll_time_in_millis":0,"scroll_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0,"total_stopped_time_in_millis":0,"total_throttled_time_in_millis":0,"total_auto_throttle_in_bytes":0},"refresh":{"total":0,"total_time_in_millis":0},"flush":{"total":0,"total_time_in_millis":0},"warmer":{"current":0,"total":0,"total_time_in_millis":0},"query_cache":{"memory_size_in_bytes":0,"total_count":0,"hit_count":0,"miss_count":0,"cache_size":0,"cache_count":0,"evictions":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":0,"memory_in_bytes":0,"terms_memory_in_bytes":0,"stored_fields_memory_in_bytes":0,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":0,"doc_values_memory_in_bytes":0,"index_writer_memory_in_bytes":0,"index_writer_max_memory_in_bytes":0,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":0},"translog":{"operations":0,"size_in_bytes":0},"suggest":{"total":0,"time_in_millis":0,"current":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0},"recovery":{"current_as_source":0,"current_as_target":0,"throttle_time_in_millis":0}},"os":{"timestamp":1523896933716,"cpu_percent":1,"load_average":0.03,"mem":{"total_in_bytes":8202686464,"free_in_bytes":1511804928,"used_in_bytes":6690881536,"free_percent":18,"used_percent":82},"swap":{"total_in_bytes":0,"free_in_bytes":0,"used_in_bytes":0}},"process":{"timestamp":1523896933718,"open_file_descriptors":231,"max_file_descriptors":65535,"cpu":{"percent":0,"total_in_millis":11994030},"mem":{"total_virtual_in_bytes":3802107904}},"jvm":{"timestamp":1523896933721,"uptime_in_millis":2189778574,"mem":{"heap_used_in_bytes":152691840,"heap_used_percent":14,"heap_committed_in_bytes":251002880,"heap_max_in_bytes":1056309248,"non_heap_used_in_bytes":76764968,"non_heap_committed_in_bytes":79626240,"pools":{"young":{"used_in_bytes":95936344,"max_in_bytes":139591680,"peak_used_in_bytes":139591680,"peak_max_in_bytes":139591680},"survivor":{"used_in_bytes":422608,"max_in_bytes":17432576,"peak_used_in_bytes":17432576,"peak_max_in_bytes":17432576},"old":{"used_in_bytes":56332888,"max_in_bytes":899284992,"peak_used_in_bytes":56332888,"peak_max_in_bytes":899284992}}},"threads":{"count":32,"peak_count":37},"gc":{"collectors":{"young":{"collection_count":2387,"collection_time_in_millis":15854},"old":{"collection_count":1,"collection_time_in_millis":58}}},"buffer_pools":{"direct":{"count":29,"used_in_bytes":35284732,"total_capacity_in_bytes":35284732},"mapped":{"count":0,"used_in_bytes":0,"total_capacity_in_bytes":0}}},"thread_pool":{"bulk":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch_shard_started":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch_shard_store":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"flush":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":6,"completed":2406162},"get":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"index":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"listener":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":1,"completed":37827},"management":{"threads":1,"queue":0,"active":1,"rejected":0,"largest":4,"completed":34328},"percolate":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"refresh":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"search":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":6264},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0}},"fs":{"timestamp":1523896933721,"total":{},"data":[]},"transport":{"server_open":66,"rx_count":5963568,"rx_size_in_bytes":22587327911,"tx_count":5963559,"tx_size_in_bytes":1128915360},"http":{"current_open":0,"total_opened":5624628},"breakers":{"request":{"limit_size_in_bytes":422523699,"limit_size":"402.9mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":633785548,"limit_size":"604.4mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.03,"tripped":0},"parent":{"limit_size_in_bytes":739416473,"limit_size":"705.1mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0}},"script":{"compilations":1,"cache_evictions":0}},"TK78rIdlRTyjnj8VWsCBlg":{"timestamp":1523896933718,"name":"DN3","transport_address":"172.16.1.11:9300","host":"172.16.1.11","ip":["172.16.1.11:9300","NONE"],"indices":{"docs":{"count":43741682,"deleted":358416},"store":{"size_in_bytes":28660039198,"throttle_time_in_millis":0},"indexing":{"index_total":50082253,"index_time_in_millis":27577734,"index_current":1052,"index_failed":4232,"delete_total":0,"delete_time_in_millis":0,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":14473581,"query_time_in_millis":11313339,"query_current":0,"fetch_total":6734114,"fetch_time_in_millis":12467492,"fetch_current":0,"scroll_total":6856054,"scroll_time_in_millis":2261842713961,"scroll_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":19414,"total_time_in_millis":19914095,"total_docs":334226567,"total_size_in_bytes":139208351771,"total_stopped_time_in_millis":0,"total_throttled_time_in_millis":9075330,"total_auto_throttle_in_bytes":994035861},"refresh":{"total":121887,"total_time_in_millis":3992410},"flush":{"total":420,"total_time_in_millis":114339},"warmer":{"current":0,"total":252426,"total_time_in_millis":103159},"query_cache":{"memory_size_in_bytes":3736488,"total_count":8187482,"hit_count":388697,"miss_count":7798785,"cache_size":1251,"cache_count":1251,"evictions":0},"fielddata":{"memory_size_in_bytes":515627560,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":459,"memory_in_bytes":82547745,"terms_memory_in_bytes":73362237,"stored_fields_memory_in_bytes":6975856,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":1560640,"doc_values_memory_in_bytes":649012,"index_writer_memory_in_bytes":0,"index_writer_max_memory_in_bytes":22016000,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":376},"translog":{"operations":0,"size_in_bytes":1849},"suggest":{"total":0,"time_in_millis":0,"current":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0},"recovery":{"current_as_source":0,"current_as_target":0,"throttle_time_in_millis":0}},"os":{"timestamp":1523896933928,"cpu_percent":2,"load_average":0.38,"mem":{"total_in_bytes":33567973376,"free_in_bytes":627195904,"used_in_bytes":32940777472,"free_percent":2,"used_percent":98},"swap":{"total_in_bytes":0,"free_in_bytes":0,"used_in_bytes":0}},"process":{"timestamp":1523896933930,"open_file_descriptors":1084,"max_file_descriptors":65535,"cpu":{"percent":0,"total_in_millis":91639720},"mem":{"total_virtual_in_bytes":25611481088}},"jvm":{"timestamp":1523896933934,"uptime_in_millis":2189779676,"mem":{"heap_used_in_bytes":5121156736,"heap_used_percent":39,"heap_committed_in_bytes":12815171584,"heap_max_in_bytes":12815171584,"non_heap_used_in_bytes":108789760,"non_heap_committed_in_bytes":111362048,"pools":{"young":{"used_in_bytes":558370920,"max_in_bytes":558432256,"peak_used_in_bytes":558432256,"peak_max_in_bytes":558432256},"survivor":{"used_in_bytes":269568,"max_in_bytes":69730304,"peak_used_in_bytes":69730304,"peak_max_in_bytes":69730304},"old":{"used_in_bytes":4562516248,"max_in_bytes":12187009024,"peak_used_in_bytes":9186743816,"peak_max_in_bytes":12187009024}}},"threads":{"count":98,"peak_count":113},"gc":{"collectors":{"young":{"collection_count":17393,"collection_time_in_millis":1132359},"old":{"collection_count":21,"collection_time_in_millis":6273}}},"buffer_pools":{"direct":{"count":168,"used_in_bytes":66752362,"total_capacity_in_bytes":66752362},"mapped":{"count":324,"used_in_bytes":6481560589,"total_capacity_in_bytes":6481560589}}},"thread_pool":{"bulk":{"threads":8,"queue":0,"active":0,"rejected":23,"largest":8,"completed":1160849},"fetch_shard_started":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch_shard_store":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":6,"completed":6},"flush":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":4245},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":14,"completed":2406239},"get":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"index":{"threads":2,"queue":0,"active":0,"rejected":0,"largest":2,"completed":2},"listener":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":1526},"management":{"threads":5,"queue":0,"active":1,"rejected":0,"largest":5,"completed":196378},"percolate":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"refresh":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":121902},"search":{"threads":13,"queue":0,"active":0,"rejected":0,"largest":13,"completed":21269039},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":137746}},"fs":{"timestamp":1523896933935,"total":{"total_in_bytes":536859357184,"free_in_bytes":453174370304,"available_in_bytes":453174370304},"data":[{"path":"/elastic_data/elastic/jupiter/nodes/0","mount":"/ (rootfs)","type":"rootfs","total_in_bytes":536859357184,"free_in_bytes":453174370304,"available_in_bytes":453174370304}]},"transport":{"server_open":52,"rx_count":27714037,"rx_size_in_bytes":12959994710,"tx_count":25898748,"tx_size_in_bytes":51967641832},"http":{"current_open":0,"total_opened":1828},"breakers":{"request":{"limit_size_in_bytes":5126068633,"limit_size":"4.7gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":7689102950,"limit_size":"7.1gb","estimated_size_in_bytes":515627560,"estimated_size":"491.7mb","overhead":1.03,"tripped":0},"parent":{"limit_size_in_bytes":8970620108,"limit_size":"8.3gb","estimated_size_in_bytes":515627560,"estimated_size":"491.7mb","overhead":1.0,"tripped":0}},"script":{"compilations":2,"cache_evictions":1}}}}

[chitenderkumar]

would it possible to use the patch in 3.3.0?

[royrusso]

I'll update here when so. I see the problem now... One of the nodes is not configured for data and the code expects them all to be. So this is a bug worth fixing soon as a hot fix on 3.3.0

[chitenderkumar]

@royrusso hey if i download the repo for 3.3.0 ElastiHq this hotfix would be available in it, if not then in which release it will be available to use?

[royrusso]

Yes. Download the master branch from the main page. Don’t try and download any specific version, just the master.