search

braedon / prometheus-es-exporter

Prometheus Elasticsearch Exporter
MIT License
247 stars 90 forks source link

Nodes stats parser fails on AWS managed Elasticsearch #45

Closed Mitrofanov closed 4 years ago

Mitrofanov commented 5 years ago

Hi,

I faced with errors in some queries. Errors are as follow (pls check that response code was 200):

[2019-08-19 10:06:45,275] elasticsearch.INFO Thread-2 GET https://my-es-instance.ap-southeast-1.es.amazonaws.com:443/_nodes/stats [status:200 request:0.800s]
[2019-08-19 10:06:45,475] root.ERROR Thread-2 Error while fetching Nodes Stats.
Traceback (most recent call last):
  File "/usr/src/app/prometheus_es_exporter/__init__.py", line 168, in collect
    metrics = nodes_stats_parser.parse_response(response, self.metric_name_list)
  File "/usr/src/app/prometheus_es_exporter/nodes_stats_parser.py", line 77, in parse_response
    result.extend(parse_node(value, metric=metric, labels=OrderedDict({'node_id': [key]})))
  File "/usr/src/app/prometheus_es_exporter/nodes_stats_parser.py", line 66, in parse_node
    return parse_block(node, metric=metric, labels=labels)
  File "/usr/src/app/prometheus_es_exporter/nodes_stats_parser.py", line 47, in parse_block
    result.extend(parse_block(value, metric=metric + [key], labels=labels))
  File "/usr/src/app/prometheus_es_exporter/nodes_stats_parser.py", line 52, in parse_block
    bucket_name = n_value[bucket_name_key]
KeyError: 'path'

I use AWS managed ES version 6.3. I've just upgraded es exporter to latest version (0.5.2), but issue still persists.

Please suggest.

braedon commented 5 years ago

Hi @Mitrofanov,

I haven't been able to reproduce the issue locally with the official ES docker images (though there doesn't seem to be a tag for 6.3.x - I tried the earliest 6.x versions available: 6.4.0 and 6.8.2). It's probably something specific with your setup and/or AWS's modifications.

Are you able to attach the output of a manual _nodes/stats call (redacted as needed)? Feel free to send me an email if that's easier.

Thanks, Braedon

Mitrofanov commented 5 years ago

Hi, i think problem is with AWS managed ES. So looks like it blocks requests to system endpoints (like _nodes/stats), that's why i get such error.

Just wonder if we can add switch to disable system metrics collection.

braedon commented 5 years ago

There are already arguments to disable the different system metrics collectors. e.g. --nodes-stats-disable for _nodes/stats.

If you can, could you still send me that output of the manual _nodes/stats call, or the return value of the es_client.nodes.stats() call in NodesStatsCollector.collect() if the manual call doesn't get a response at all. It'd be good to work out why its getting past the failure checks.

jonny-wg2 commented 4 years ago

AWS Managed Elastic Search 7.4 output

/_nodes/stats output ``` { "_nodes" : { "total" : 2, "successful" : 2, "failed" : 0 }, "cluster_name" : "123456789:dev-elasticsearch", "nodes" : { "zemdNAFGTyqs2-US_c07Hg" : { "timestamp" : 1594711331498, "name" : "3179bdc0602788fdc819fc29c5897b7f", "roles" : [ "ingest", "master", "data" ], "indices" : { "docs" : { "count" : 189356792, "deleted" : 11 }, "store" : { "size_in_bytes" : 55376252679 }, "indexing" : { "index_total" : 1139129503, "index_time_in_millis" : 196072773, "index_current" : 0, "index_failed" : 0, "delete_total" : 0, "delete_time_in_millis" : 0, "delete_current" : 0, "noop_update_total" : 0, "is_throttled" : false, "throttle_time_in_millis" : 0 }, "get" : { "total" : 216111, "time_in_millis" : 18897, "exists_total" : 216111, "exists_time_in_millis" : 18897, "missing_total" : 0, "missing_time_in_millis" : 0, "current" : 0 }, "search" : { "open_contexts" : 0, "query_total" : 5285223, "query_time_in_millis" : 1179886, "query_current" : 0, "fetch_total" : 3205, "fetch_time_in_millis" : 1807099, "fetch_current" : 0, "scroll_total" : 0, "scroll_time_in_millis" : 0, "scroll_current" : 0, "suggest_total" : 0, "suggest_time_in_millis" : 0, "suggest_current" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 221892, "total_time_in_millis" : 301678756, "total_docs" : 4594425315, "total_size_in_bytes" : 1923684833322, "total_stopped_time_in_millis" : 15292723, "total_throttled_time_in_millis" : 171783762, "total_auto_throttle_in_bytes" : 7748842991 }, "refresh" : { "total" : 1782258, "total_time_in_millis" : 63458489, "external_total" : 1775570, "external_total_time_in_millis" : 64424613, "listeners" : 0 }, "flush" : { "total" : 2449, "periodic" : 1695, "total_time_in_millis" : 549681 }, "warmer" : { "current" : 0, "total" : 124978, "total_time_in_millis" : 21764 }, "query_cache" : { "memory_size_in_bytes" : 3168528, "total_count" : 381799, "hit_count" : 816, "miss_count" : 380983, "cache_size" : 405, "cache_count" : 519, "evictions" : 114 }, "fielddata" : { "memory_size_in_bytes" : 400, "evictions" : 0 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 184, "memory_in_bytes" : 99030983, "terms_memory_in_bytes" : 56821807, "stored_fields_memory_in_bytes" : 41864856, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 3584, "points_memory_in_bytes" : 0, "doc_values_memory_in_bytes" : 340736, "index_writer_memory_in_bytes" : 9300240, "version_map_memory_in_bytes" : 0, "fixed_bit_set_memory_in_bytes" : 656, "max_unsafe_auto_id_timestamp" : 1589459764343, "file_sizes" : { } }, "translog" : { "operations" : 404424, "size_in_bytes" : 512553507, "uncommitted_operations" : 404424, "uncommitted_size_in_bytes" : 512553507, "earliest_last_modified_age" : 0 }, "request_cache" : { "memory_size_in_bytes" : 24680, "evictions" : 0, "hit_count" : 36, "miss_count" : 39 }, "recovery" : { "current_as_source" : 0, "current_as_target" : 0, "throttle_time_in_millis" : 507656 } }, "os" : { "timestamp" : 1594711331524, "cpu" : { "percent" : 19, "load_average" : { "1m" : 0.45, "5m" : 0.54, "15m" : 0.72 } }, "mem" : { "total_in_bytes" : 8147742720, "free_in_bytes" : 353103872, "used_in_bytes" : 7794638848, "free_percent" : 4, "used_percent" : 96 }, "swap" : { "total_in_bytes" : 2147479552, "free_in_bytes" : 1623052288, "used_in_bytes" : 524427264 } }, "process" : { "timestamp" : 1594711331524, "open_file_descriptors" : 1281, "max_file_descriptors" : 1024000, "cpu" : { "percent" : 15, "total_in_millis" : 520342770 }, "mem" : { "total_virtual_in_bytes" : 24140759040 } }, "jvm" : { "timestamp" : 1594711331525, "uptime_in_millis" : 4314500458, "mem" : { "heap_used_in_bytes" : 2525028232, "heap_used_percent" : 59, "heap_committed_in_bytes" : 4277534720, "heap_max_in_bytes" : 4277534720, "non_heap_used_in_bytes" : 301990040, "non_heap_committed_in_bytes" : 364273664, "pools" : { "young" : { "used_in_bytes" : 57668512, "max_in_bytes" : 139591680, "peak_used_in_bytes" : 139591680, "peak_max_in_bytes" : 139591680 }, "survivor" : { "used_in_bytes" : 17171712, "max_in_bytes" : 17432576, "peak_used_in_bytes" : 17432576, "peak_max_in_bytes" : 17432576 }, "old" : { "used_in_bytes" : 2450201344, "max_in_bytes" : 4120510464, "peak_used_in_bytes" : 3111568240, "peak_max_in_bytes" : 4120510464 } } }, "threads" : { "count" : 138, "peak_count" : 139 }, "gc" : { "collectors" : { "young" : { "collection_count" : 1813429, "collection_time_in_millis" : 36671505 }, "old" : { "collection_count" : 887, "collection_time_in_millis" : 80911 } } }, "buffer_pools" : { "mapped" : { "count" : 254, "used_in_bytes" : 16426966449, "total_capacity_in_bytes" : 16426966449 }, "direct" : { "count" : 80, "used_in_bytes" : 72555719, "total_capacity_in_bytes" : 72555717 } }, "classes" : { "current_loaded_count" : 35425, "total_loaded_count" : 35668, "total_unloaded_count" : 243 } }, "thread_pool" : { "ad-threadpool" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "analyze" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "fetch_shard_started" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "fetch_shard_store" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "flush" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 20237 }, "force_merge" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "generic" : { "threads" : 7, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 17, "completed" : 3020584 }, "get" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 2, "completed" : 216017 }, "listener" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 60 }, "management" : { "threads" : 5, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 5, "completed" : 51716026 }, "open_distro_job_scheduler" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "refresh" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 23965890 }, "search" : { "threads" : 4, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 4, "completed" : 6972983 }, "search_throttled" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "snapshot" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "snapshot_segments" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 135255 }, "snapshot_shards" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 24707 }, "sql-worker" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "ultrawarm_migration" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "warmer" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 749 }, "write" : { "threads" : 2, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 2, "completed" : 45229832 } }, "fs" : { "timestamp" : 1594711331526, "total" : { "total_in_bytes" : 105152176128, "free_in_bytes" : 49196556288, "available_in_bytes" : 43811069952 }, "data" : [ { "type" : "ext4", "total_in_bytes" : 105152176128, "free_in_bytes" : 49196556288, "available_in_bytes" : 43811069952 } ], "io_stats" : { "devices" : [ { "device_name" : "nvme2n1", "operations" : 252994663, "read_operations" : 13804597, "write_operations" : 239190066, "read_kilobytes" : 1630186876, "write_kilobytes" : 5477868928 } ], "total" : { "operations" : 252994663, "read_operations" : 13804597, "write_operations" : 239190066, "read_kilobytes" : 1630186876, "write_kilobytes" : 5477868928 } } }, "http" : { "current_open" : 0, "total_opened" : 0 }, "breakers" : { "request" : { "limit_size_in_bytes" : 2566520832, "limit_size" : "2.3gb", "estimated_size_in_bytes" : 0, "estimated_size" : "0b", "overhead" : 1.0, "tripped" : 0 }, "fielddata" : { "limit_size_in_bytes" : 1711013888, "limit_size" : "1.5gb", "estimated_size_in_bytes" : 400, "estimated_size" : "400b", "overhead" : 1.03, "tripped" : 0 }, "in_flight_requests" : { "limit_size_in_bytes" : 4277534720, "limit_size" : "3.9gb", "estimated_size_in_bytes" : 158419, "estimated_size" : "154.7kb", "overhead" : 2.0, "tripped" : 0 }, "accounting" : { "limit_size_in_bytes" : 4277534720, "limit_size" : "3.9gb", "estimated_size_in_bytes" : 99030599, "estimated_size" : "94.4mb", "overhead" : 1.0, "tripped" : 0 }, "parent" : { "limit_size_in_bytes" : 4063657984, "limit_size" : "3.7gb", "estimated_size_in_bytes" : 2525101448, "estimated_size" : "2.3gb", "overhead" : 1.0, "tripped" : 0 } }, "script" : { "compilations" : 1, "cache_evictions" : 0, "compilation_limit_triggered" : 0 }, "discovery" : { "cluster_state_queue" : { "total" : 0, "pending" : 0, "committed" : 0 }, "published_cluster_states" : { "full_states" : 1, "incompatible_diffs" : 0, "compatible_diffs" : 53422 } }, "ingest" : { "total" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0 }, "pipelines" : { } }, "adaptive_selection" : { "zemdNAFGTyqs2-US_c07Hg" : { "outgoing_searches" : 0, "avg_queue_size" : 1, "avg_service_time_ns" : 42553, "avg_response_time_ns" : 946399, "rank" : "165.4" } } }, "J8n1VQySQvmmc4x-J7IbPA" : { "timestamp" : 1594711331498, "name" : "71a31238e4a82031f084303aa1168bf1", "roles" : [ "ingest", "master", "data" ], "indices" : { "docs" : { "count" : 189357384, "deleted" : 11 }, "store" : { "size_in_bytes" : 55560505891 }, "indexing" : { "index_total" : 1139116759, "index_time_in_millis" : 190995891, "index_current" : 0, "index_failed" : 0, "delete_total" : 0, "delete_time_in_millis" : 0, "delete_current" : 0, "noop_update_total" : 0, "is_throttled" : false, "throttle_time_in_millis" : 0 }, "get" : { "total" : 216176, "time_in_millis" : 19149, "exists_total" : 216176, "exists_time_in_millis" : 19149, "missing_total" : 0, "missing_time_in_millis" : 0, "current" : 0 }, "search" : { "open_contexts" : 0, "query_total" : 5299820, "query_time_in_millis" : 2055862, "query_current" : 0, "fetch_total" : 3211, "fetch_time_in_millis" : 1902399, "fetch_current" : 0, "scroll_total" : 0, "scroll_time_in_millis" : 0, "scroll_current" : 0, "suggest_total" : 0, "suggest_time_in_millis" : 0, "suggest_current" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 216070, "total_time_in_millis" : 290962500, "total_docs" : 4589654704, "total_size_in_bytes" : 1918328906664, "total_stopped_time_in_millis" : 14946077, "total_throttled_time_in_millis" : 174204095, "total_auto_throttle_in_bytes" : 7733986777 }, "refresh" : { "total" : 1781079, "total_time_in_millis" : 61377395, "external_total" : 1776247, "external_total_time_in_millis" : 62356281, "listeners" : 0 }, "flush" : { "total" : 3412, "periodic" : 2550, "total_time_in_millis" : 669643 }, "warmer" : { "current" : 0, "total" : 125192, "total_time_in_millis" : 21224 }, "query_cache" : { "memory_size_in_bytes" : 2764600, "total_count" : 336065, "hit_count" : 211, "miss_count" : 335854, "cache_size" : 171, "cache_count" : 197, "evictions" : 26 }, "fielddata" : { "memory_size_in_bytes" : 400, "evictions" : 0 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 207, "memory_in_bytes" : 99792296, "terms_memory_in_bytes" : 57571380, "stored_fields_memory_in_bytes" : 41846168, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 3584, "points_memory_in_bytes" : 0, "doc_values_memory_in_bytes" : 371164, "index_writer_memory_in_bytes" : 11676000, "version_map_memory_in_bytes" : 0, "fixed_bit_set_memory_in_bytes" : 656, "max_unsafe_auto_id_timestamp" : 1594711104744, "file_sizes" : { } }, "translog" : { "operations" : 316184, "size_in_bytes" : 389257548, "uncommitted_operations" : 316184, "uncommitted_size_in_bytes" : 389257548, "earliest_last_modified_age" : 0 }, "request_cache" : { "memory_size_in_bytes" : 24680, "evictions" : 0, "hit_count" : 50, "miss_count" : 69 }, "recovery" : { "current_as_source" : 0, "current_as_target" : 0, "throttle_time_in_millis" : 461579 } }, "os" : { "timestamp" : 1594711331520, "cpu" : { "percent" : 18, "load_average" : { "1m" : 0.43, "5m" : 0.63, "15m" : 0.84 } }, "mem" : { "total_in_bytes" : 8059662336, "free_in_bytes" : 175951872, "used_in_bytes" : 7883710464, "free_percent" : 2, "used_percent" : 98 }, "swap" : { "total_in_bytes" : 2147479552, "free_in_bytes" : 1910140928, "used_in_bytes" : 237338624 } }, "process" : { "timestamp" : 1594711331520, "open_file_descriptors" : 1281, "max_file_descriptors" : 1024000, "cpu" : { "percent" : 14, "total_in_millis" : 507603860 }, "mem" : { "total_virtual_in_bytes" : 25600729088 } }, "jvm" : { "timestamp" : 1594711331521, "uptime_in_millis" : 4314496470, "mem" : { "heap_used_in_bytes" : 2878264640, "heap_used_percent" : 67, "heap_committed_in_bytes" : 4277534720, "heap_max_in_bytes" : 4277534720, "non_heap_used_in_bytes" : 309037096, "non_heap_committed_in_bytes" : 371548160, "pools" : { "young" : { "used_in_bytes" : 118754808, "max_in_bytes" : 139591680, "peak_used_in_bytes" : 139591680, "peak_max_in_bytes" : 139591680 }, "survivor" : { "used_in_bytes" : 16264104, "max_in_bytes" : 17432576, "peak_used_in_bytes" : 17432576, "peak_max_in_bytes" : 17432576 }, "old" : { "used_in_bytes" : 2743259216, "max_in_bytes" : 4120510464, "peak_used_in_bytes" : 3117059664, "peak_max_in_bytes" : 4120510464 } } }, "threads" : { "count" : 144, "peak_count" : 153 }, "gc" : { "collectors" : { "young" : { "collection_count" : 1791235, "collection_time_in_millis" : 38510484 }, "old" : { "collection_count" : 868, "collection_time_in_millis" : 86925 } } }, "buffer_pools" : { "mapped" : { "count" : 277, "used_in_bytes" : 17867024991, "total_capacity_in_bytes" : 17867024991 }, "direct" : { "count" : 82, "used_in_bytes" : 72553994, "total_capacity_in_bytes" : 72553992 } }, "classes" : { "current_loaded_count" : 35588, "total_loaded_count" : 35819, "total_unloaded_count" : 231 } }, "thread_pool" : { "ad-threadpool" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "analyze" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "fetch_shard_started" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "fetch_shard_store" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "flush" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 21316 }, "force_merge" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "generic" : { "threads" : 8, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 16, "completed" : 3242023 }, "get" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 2, "completed" : 216148 }, "listener" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 59 }, "management" : { "threads" : 5, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 5, "completed" : 52082749 }, "open_distro_job_scheduler" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "refresh" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 23975050 }, "search" : { "threads" : 4, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 4, "completed" : 7096076 }, "search_throttled" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "snapshot" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 4792 }, "snapshot_segments" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 6306 }, "snapshot_shards" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 15127 }, "sql-worker" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "ultrawarm_migration" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "warmer" : { "threads" : 1, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 1, "completed" : 686 }, "write" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 2, "completed" : 45224629 } }, "fs" : { "timestamp" : 1594711331521, "total" : { "total_in_bytes" : 105152176128, "free_in_bytes" : 49135415296, "available_in_bytes" : 43749928960 }, "data" : [ { "type" : "ext4", "total_in_bytes" : 105152176128, "free_in_bytes" : 49135415296, "available_in_bytes" : 43749928960 } ], "io_stats" : { "devices" : [ { "device_name" : "nvme2n1", "operations" : 248173027, "read_operations" : 8796005, "write_operations" : 239377022, "read_kilobytes" : 1033724160, "write_kilobytes" : 5498005224 } ], "total" : { "operations" : 248173027, "read_operations" : 8796005, "write_operations" : 239377022, "read_kilobytes" : 1033724160, "write_kilobytes" : 5498005224 } } }, "http" : { "current_open" : 0, "total_opened" : 0 }, "breakers" : { "request" : { "limit_size_in_bytes" : 2566520832, "limit_size" : "2.3gb", "estimated_size_in_bytes" : 0, "estimated_size" : "0b", "overhead" : 1.0, "tripped" : 0 }, "fielddata" : { "limit_size_in_bytes" : 1711013888, "limit_size" : "1.5gb", "estimated_size_in_bytes" : 400, "estimated_size" : "400b", "overhead" : 1.03, "tripped" : 0 }, "in_flight_requests" : { "limit_size_in_bytes" : 4277534720, "limit_size" : "3.9gb", "estimated_size_in_bytes" : 156328, "estimated_size" : "152.6kb", "overhead" : 2.0, "tripped" : 0 }, "accounting" : { "limit_size_in_bytes" : 4277534720, "limit_size" : "3.9gb", "estimated_size_in_bytes" : 99791912, "estimated_size" : "95.1mb", "overhead" : 1.0, "tripped" : 0 }, "parent" : { "limit_size_in_bytes" : 4063657984, "limit_size" : "3.7gb", "estimated_size_in_bytes" : 2878337296, "estimated_size" : "2.6gb", "overhead" : 1.0, "tripped" : 0 } }, "script" : { "compilations" : 0, "cache_evictions" : 0, "compilation_limit_triggered" : 0 }, "discovery" : { "cluster_state_queue" : { "total" : 0, "pending" : 0, "committed" : 0 }, "published_cluster_states" : { "full_states" : 1, "incompatible_diffs" : 0, "compatible_diffs" : 53420 } }, "ingest" : { "total" : { "count" : 0, "time_in_millis" : 0, "current" : 0, "failed" : 0 }, "pipelines" : { } }, "adaptive_selection" : { "zemdNAFGTyqs2-US_c07Hg" : { "outgoing_searches" : 0, "avg_queue_size" : 0, "avg_service_time_ns" : 879575, "avg_response_time_ns" : 20597656, "rank" : "20.6" }, "J8n1VQySQvmmc4x-J7IbPA" : { "outgoing_searches" : 0, "avg_queue_size" : 1, "avg_service_time_ns" : 204358, "avg_response_time_ns" : 1545732, "rank" : "35.8" } } } } } ```
braedon commented 4 years ago

Thanks @jonny-wg2, I think I've found the issue.

In the fs.data list for the nodes, it looks like AWS is dropping the path field for each data directory entry. The parser expects the path to be there as it uses it to populate the path label, identifying the metrics for each data directory.

AWS example:

      ...
      "fs" : {
        ...
        "data" : [
          {
            "type" : "ext4",
            "total_in_bytes" : 105152176128,
            "free_in_bytes" : 49135415296,
            "available_in_bytes" : 43749928960
          }
        ],
        ...
      },
      ...

Non-AWS example:

      ...
      "fs" : {
        ...
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/ (none)",
            "type" : "aufs",
            "total_in_bytes" : 91787288576,
            "free_in_bytes" : 15394492416,
            "available_in_bytes" : 10708357120
          }
        ],
        ...
      },
      ...

I'll put in a fallback to use the data directory's position in the fs.data list as the path label value e.g. path="0". I'm not sure if AWS ever uses more than one data directory, but I'd hope they keep the same order between calls if they do... Fingers crossed.

braedon commented 4 years ago

Fixed in 0.11.1

jonny-wg2 commented 4 years ago

Thanks @braedon !