Closed ghost closed 2 years ago
As for the timeout, looks to be the same issue as #579 - not sure what the solution is, but it seems to be a problem with AWS Elasticache clusters.
Redis_exporter is exceptionally slow
- if you run the cluster with full debug mode and get a timestamp for each log line you should be able to see where the time is being spend. I agree, 60s is too long, esp if you don't run larger SCANs as part of the scrape.
As for the changing shard - is it possible that you're being connected to a different host of your cluster, each time the scrape is run? This could happen if you connect via a DNS name that round-robins between nodes. Again, could be a Elasticache-only isse, not sure.
if you run the cluster with full debug mode and get a timestamp for each log line you should be able to see where the time is being spend. I agree, 60s is too long, esp if you don't run larger SCANs as part of the scrape.
It seems that only slowlog is possible on AWS . Would that suffice?
As for the changing shard - is it possible that you're being connected to a different host of your cluster, each time the scrape is run?
You are right by saying that this dns record may cause dns-rr-alike behaviour, but for me it seem that redis-cluster client should have some extra logic around. I did always used Elasticache configuration endpoint for golang microservices so redis-cluster client was able to autodetect cluster state and keep track of it as a whole. My question regarding exporter usage pattern comes from concern of resilience, efficiency and ability to react to cluster reconfigurations/failures without changing static configuration of exporter.
if you run the cluster with full debug mode and get a timestamp for each log line you should be able to see where the time is being spend. I agree, 60s is too long, esp if you don't run larger SCANs as part of the scrape.
It seems that only slowlog is possible on AWS . Would that suffice?
My bad, what I meant is, run the exporter in debug mode, it should output a log line per command with timestamps and you should be able to tell why it takes 60s for one scrape.
As for the changing shard - is it possible that you're being connected to a different host of your cluster, each time the scrape is run?
You are right by saying that this dns record may cause dns-rr-alike behaviour, but for me it seem that redis-cluster client should have some extra logic around. I did always used Elasticache configuration endpoint for golang microservices so redis-cluster client was able to autodetect cluster state and keep track of it as a whole. My question regarding exporter usage pattern comes from concern of resilience, efficiency and ability to react to cluster reconfigurations/failures without changing static configuration of exporter.
SD shouldn't in the exporter but be handled by the Prometheus server (plus additional plumbing if necessary)
Regarding exporter debug log, ive put whitespaces in order to make it more readable:
#curl invoked just before this line
time="2022-01-10T14:01:05Z" level=error msg="Cluster refresh failed: redisc: all nodes failed\nread tcp 10.0.37.233:52432->10.0.53.93:6379: i/o timeout"
time="2022-01-10T14:01:05Z" level=debug msg="Creating redis connection object"
time="2022-01-10T14:01:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:05Z" level=debug msg="keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:05Z" level=debug msg="e.singleKeys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:05Z" level=debug msg="e.keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:05Z" level=debug msg="allKeys: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: SLOWLOG [LEN]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: SLOWLOG [GET 1]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:05Z" level=debug msg="allStreams: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:05Z" level=debug msg="scrapeRedisHost() done"
time="2022-01-10T14:01:05Z" level=debug msg="Trying DialURL(): rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:05Z" level=debug msg="connected to: rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:05Z" level=debug msg="connecting took 0.009746 seconds"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: CLIENT [SETNAME redis_exporter]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: CONFIG [GET *]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="Redis CONFIG err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: INFO [ALL]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="Redis INFO ALL result: [\"# Server\\r\\nredis_version:6.0.5\\r\\nredis_git_sha1:0\\r\\nredis_git_dirty:0\\r\\nredis_build_id:0\\r\\nredis_mode:cluster\\r\\nos:Amazon ElastiCache\\r\\narch_bits:64\\r\\nmultiplexing_api:epoll\\r\\natomicvar_api:atomic-builtin\\r\\ngcc_version:0.0.0\\r\\nprocess_id:1\\r\\nrun_id:a20787f1c26718fd8c3b27a0c7dc33fce98ed847\\r\\ntcp_port:6379\\r\\nuptime_in_seconds:10641622\\r\\nuptime_in_days:123\\r\\nhz:10\\r\\nconfigured_hz:10\\r\\nlru_clock:14433313\\r\\nexecutable:-\\r\\nconfig_file:-\\r\\n\\r\\n# Clients\\r\\nconnected_clients:43\\r\\nclient_recent_max_input_buffer:2\\r\\nclient_recent_max_output_buffer:0\\r\\nblocked_clients:0\\r\\ntracking_clients:0\\r\\nclients_in_timeout_table:0\\r\\n\\r\\n# Memory\\r\\nused_memory:7250504\\r\\nused_memory_human:6.91M\\r\\nused_memory_rss:12759040\\r\\nused_memory_rss_human:12.17M\\r\\nused_memory_peak:44823168\\r\\nused_memory_peak_human:42.75M\\r\\nused_memory_peak_perc:16.18%\\r\\nused_memory_overhead:6872326\\r\\nused_memory_startup:5075656\\r\\nused_memory_dataset:378178\\r\\nused_memory_dataset_perc:17.39%\\r\\nallocator_allocated:15473208\\r\\nallocator_active:16818176\\r\\nallocator_resident:22024192\\r\\nused_memory_lua:37888\\r\\nused_memory_lua_human:37.00K\\r\\nused_memory_scripts:0\\r\\nused_memory_scripts_human:0B\\r\\nnumber_of_cached_scripts:0\\r\\nmaxmemory:436469760\\r\\nmaxmemory_human:416.25M\\r\\nmaxmemory_policy:volatile-lru\\r\\nallocator_frag_ratio:1.09\\r\\nallocator_frag_bytes:1344968\\r\\nallocator_rss_ratio:1.31\\r\\nallocator_rss_bytes:5206016\\r\\nrss_overhead_ratio:0.58\\r\\nrss_overhead_bytes:-9265152\\r\\nmem_fragmentation_ratio:1.78\\r\\nmem_fragmentation_bytes:5572288\\r\\nmem_not_counted_for_evict:0\\r\\nmem_replication_backlog:1048576\\r\\nmem_clients_slaves:17242\\r\\nmem_clients_normal:724164\\r\\nmem_aof_buffer:0\\r\\nmem_allocator:jemalloc-5.1.0\\r\\nactive_defrag_running:0\\r\\nlazyfree_pending_objects:0\\r\\n\\r\\n# Persistence\\r\\nloading:0\\r\\nrdb_changes_since_last_save:14698\\r\\nrdb_bgsave_in_progress:0\\r\\nrdb_last_save_time:1631181643\\r\\nrdb_last_bgsave_status:ok\\r\\nrdb_last_bgsave_time_sec:-1\\r\\nrdb_current_bgsave_time_sec:-1\\r\\nrdb_last_cow_size:380928\\r\\naof_enabled:0\\r\\naof_rewrite_in_progress:0\\r\\naof_rewrite_scheduled:0\\r\\naof_last_rewrite_time_sec:-1\\r\\naof_current_rewrite_time_sec:-1\\r\\naof_last_bgrewrite_status:ok\\r\\naof_last_write_status:ok\\r\\naof_last_cow_size:0\\r\\nmodule_fork_in_progress:0\\r\\nmodule_fork_last_cow_size:0\\r\\n\\r\\n# SSL\\r\\nssl_enabled:yes\\r\\nssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT\\r\\nssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT\\r\\nssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0\\r\\n\\r\\n# Stats\\r\\ntotal_connections_received:13901470\\r\\ntotal_commands_processed:27745346\\r\\ninstantaneous_ops_per_sec:0\\r\\ntotal_net_input_bytes:5126769661\\r\\ntotal_net_output_bytes:231166832944\\r\\ninstantaneous_input_kbps:0.31\\r\\ninstantaneous_output_kbps:21.66\\r\\nrejected_connections:0\\r\\nsync_full:1\\r\\nsync_partial_ok:0\\r\\nsync_partial_err:1\\r\\nexpired_keys:2186\\r\\nexpired_stale_perc:0.00\\r\\nexpired_time_cap_reached_count:0\\r\\nexpire_cycle_cpu_milliseconds:156512\\r\\nevicted_keys:0\\r\\nkeyspace_hits:9047\\r\\nkeyspace_misses:17280\\r\\npubsub_channels:0\\r\\npubsub_patterns:0\\r\\nlatest_fork_usec:265\\r\\nmigrate_cached_sockets:0\\r\\nslave_expires_tracked_keys:0\\r\\nactive_defrag_hits:0\\r\\nactive_defrag_misses:0\\r\\nactive_defrag_key_hits:0\\r\\nactive_defrag_key_misses:0\\r\\ntracking_total_keys:0\\r\\ntracking_total_items:0\\r\\ntracking_total_prefixes:0\\r\\nunexpected_error_replies:0\\r\\n\\r\\n# Replication\\r\\nrole:master\\r\\nconnected_slaves:1\\r\\nslave0:ip=10.17.1.161,port=6379,state=online,offset=606356374,lag=0\\r\\nmaster_replid:f940006536f8a5a1baf08c889b69c7ec1370af55\\r\\nmaster_replid2:0000000000000000000000000000000000000000\\r\\nmaster_repl_offset:606356374\\r\\nsecond_repl_offset:-1\\r\\nrepl_backlog_active:1\\r\\nrepl_backlog_size:1048576\\r\\nrepl_backlog_first_byte_offset:605307799\\r\\nrepl_backlog_histlen:1048576\\r\\n\\r\\n# CPU\\r\\nused_cpu_sys:14378.524887\\r\\nused_cpu_user:55102.294431\\r\\nused_cpu_sys_children:0.000000\\r\\nused_cpu_user_children:0.001069\\r\\n\\r\\n# Modules\\r\\n\\r\\n# Commandstats\\r\\ncmdstat_slowlog:calls=158,usec=2432,usec_per_call=15.39\\r\\ncmdstat_config:calls=9832419,usec=201002166,usec_per_call=20.44\\r\\ncmdstat_exists:calls=1770,usec=7886,usec_per_call=4.46\\r\\ncmdstat_command:calls=1,usec=1215,usec_per_call=1215.00\\r\\ncmdstat_info:calls=13183859,usec=5099318587,usec_per_call=386.79\\r\\ncmdstat_latency:calls=80,usec=1250,usec_per_call=15.62\\r\\ncmdstat_get:calls=24557,usec=93931,usec_per_call=3.83\\r\\ncmdstat_cluster:calls=4378,usec=442967,usec_per_call=101.18\\r\\ncmdstat_replconf:calls=10603786,usec=16838445,usec_per_call=1.59\\r\\ncmdstat_set:calls=14698,usec=89843,usec_per_call=6.11\\r\\ncmdstat_auth:calls=14256877,usec=104291297,usec_per_call=7.32\\r\\ncmdstat_psync:calls=1,usec=589,usec_per_call=589.00\\r\\ncmdstat_ping:calls=39778899,usec=35132612,usec_per_call=0.88\\r\\ncmdstat_client:calls=354807,usec=11947642,usec_per_call=33.67\\r\\ncmdstat_hello:calls=22,usec=306,usec_per_call=13.91\\r\\n\\r\\n# Cluster\\r\\ncluster_enabled:1\\r\\n\\r\\n# Keyspace\\r\\ndb0:keys=64,expires=0,avg_ttl=0\\r\\n\"]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: CLUSTER [INFO]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_state:ok"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_slots_assigned:16384"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_slots_ok:16384"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_slots_pfail:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_slots_fail:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_known_nodes:6"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_size:3"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_current_epoch:3"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_my_epoch:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_ping_sent:10603948"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_pong_sent:10584063"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_update_sent:117"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_sent:21188128"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_ping_received:10584060"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_pong_received:10609409"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_meet_received:3"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_update_received:263"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_stats_messages_received:21193735"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="dbCount: 1"
time="2022-01-10T14:01:05Z" level=debug msg="info: # Server"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Server"
time="2022-01-10T14:01:05Z" level=debug msg="info: redis_version:6.0.5"
time="2022-01-10T14:01:05Z" level=debug msg="info: redis_git_sha1:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: redis_git_dirty:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: redis_build_id:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: redis_mode:cluster"
time="2022-01-10T14:01:05Z" level=debug msg="info: os:Amazon ElastiCache"
time="2022-01-10T14:01:05Z" level=debug msg="info: arch_bits:64"
time="2022-01-10T14:01:05Z" level=debug msg="info: multiplexing_api:epoll"
time="2022-01-10T14:01:05Z" level=debug msg="info: atomicvar_api:atomic-builtin"
time="2022-01-10T14:01:05Z" level=debug msg="info: gcc_version:0.0.0"
time="2022-01-10T14:01:05Z" level=debug msg="info: process_id:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: run_id:a20787f1c26718fd8c3b27a0c7dc33fce98ed847"
time="2022-01-10T14:01:05Z" level=debug msg="info: tcp_port:6379"
time="2022-01-10T14:01:05Z" level=debug msg="info: uptime_in_seconds:10641622"
time="2022-01-10T14:01:05Z" level=debug msg="info: uptime_in_days:123"
time="2022-01-10T14:01:05Z" level=debug msg="info: hz:10"
time="2022-01-10T14:01:05Z" level=debug msg="info: configured_hz:10"
time="2022-01-10T14:01:05Z" level=debug msg="info: lru_clock:14433313"
time="2022-01-10T14:01:05Z" level=debug msg="info: executable:-"
time="2022-01-10T14:01:05Z" level=debug msg="info: config_file:-"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Clients"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Clients"
time="2022-01-10T14:01:05Z" level=debug msg="info: connected_clients:43"
time="2022-01-10T14:01:05Z" level=debug msg="info: client_recent_max_input_buffer:2"
time="2022-01-10T14:01:05Z" level=debug msg="info: client_recent_max_output_buffer:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: blocked_clients:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: tracking_clients:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: clients_in_timeout_table:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Memory"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Memory"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory:7250504"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_human:6.91M"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_rss:12759040"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_rss_human:12.17M"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_peak:44823168"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_peak_human:42.75M"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_peak_perc:16.18%"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_overhead:6872326"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_startup:5075656"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_dataset:378178"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_dataset_perc:17.39%"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_allocated:15473208"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_active:16818176"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_resident:22024192"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_lua:37888"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_lua_human:37.00K"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_scripts:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_memory_scripts_human:0B"
time="2022-01-10T14:01:05Z" level=debug msg="info: number_of_cached_scripts:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: maxmemory:436469760"
time="2022-01-10T14:01:05Z" level=debug msg="info: maxmemory_human:416.25M"
time="2022-01-10T14:01:05Z" level=debug msg="info: maxmemory_policy:volatile-lru"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_frag_ratio:1.09"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_frag_bytes:1344968"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_rss_ratio:1.31"
time="2022-01-10T14:01:05Z" level=debug msg="info: allocator_rss_bytes:5206016"
time="2022-01-10T14:01:05Z" level=debug msg="info: rss_overhead_ratio:0.58"
time="2022-01-10T14:01:05Z" level=debug msg="info: rss_overhead_bytes:-9265152"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_fragmentation_ratio:1.78"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_fragmentation_bytes:5572288"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_not_counted_for_evict:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_replication_backlog:1048576"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_clients_slaves:17242"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_clients_normal:724164"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_aof_buffer:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: mem_allocator:jemalloc-5.1.0"
time="2022-01-10T14:01:05Z" level=debug msg="info: active_defrag_running:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: lazyfree_pending_objects:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Persistence"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Persistence"
time="2022-01-10T14:01:05Z" level=debug msg="info: loading:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_changes_since_last_save:14698"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_bgsave_in_progress:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_last_save_time:1631181643"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_last_bgsave_status:ok"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_last_bgsave_time_sec:-1"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_current_bgsave_time_sec:-1"
time="2022-01-10T14:01:05Z" level=debug msg="info: rdb_last_cow_size:380928"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_enabled:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_rewrite_in_progress:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_rewrite_scheduled:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_last_rewrite_time_sec:-1"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_current_rewrite_time_sec:-1"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_last_bgrewrite_status:ok"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_last_write_status:ok"
time="2022-01-10T14:01:05Z" level=debug msg="info: aof_last_cow_size:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: module_fork_in_progress:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: module_fork_last_cow_size:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # SSL"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: SSL"
time="2022-01-10T14:01:05Z" level=debug msg="info: ssl_enabled:yes"
time="2022-01-10T14:01:05Z" level=debug msg="info: ssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT"
time="2022-01-10T14:01:05Z" level=debug msg="info: ssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT"
time="2022-01-10T14:01:05Z" level=debug msg="info: ssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Stats"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Stats"
time="2022-01-10T14:01:05Z" level=debug msg="info: total_connections_received:13901470"
time="2022-01-10T14:01:05Z" level=debug msg="info: total_commands_processed:27745346"
time="2022-01-10T14:01:05Z" level=debug msg="info: instantaneous_ops_per_sec:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: total_net_input_bytes:5126769661"
time="2022-01-10T14:01:05Z" level=debug msg="info: total_net_output_bytes:231166832944"
time="2022-01-10T14:01:05Z" level=debug msg="info: instantaneous_input_kbps:0.31"
time="2022-01-10T14:01:05Z" level=debug msg="info: instantaneous_output_kbps:21.66"
time="2022-01-10T14:01:05Z" level=debug msg="info: rejected_connections:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: sync_full:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: sync_partial_ok:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: sync_partial_err:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: expired_keys:2186"
time="2022-01-10T14:01:05Z" level=debug msg="info: expired_stale_perc:0.00"
time="2022-01-10T14:01:05Z" level=debug msg="info: expired_time_cap_reached_count:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: expire_cycle_cpu_milliseconds:156512"
time="2022-01-10T14:01:05Z" level=debug msg="info: evicted_keys:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: keyspace_hits:9047"
time="2022-01-10T14:01:05Z" level=debug msg="info: keyspace_misses:17280"
time="2022-01-10T14:01:05Z" level=debug msg="info: pubsub_channels:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: pubsub_patterns:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: latest_fork_usec:265"
time="2022-01-10T14:01:05Z" level=debug msg="info: migrate_cached_sockets:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: slave_expires_tracked_keys:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: active_defrag_hits:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: active_defrag_misses:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: active_defrag_key_hits:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: active_defrag_key_misses:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: tracking_total_keys:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: tracking_total_items:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: tracking_total_prefixes:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: unexpected_error_replies:0"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Replication"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Replication"
time="2022-01-10T14:01:05Z" level=debug msg="info: role:master"
time="2022-01-10T14:01:05Z" level=debug msg="info: connected_slaves:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: slave0:ip=10.17.1.161,port=6379,state=online,offset=606356374,lag=0"
time="2022-01-10T14:01:05Z" level=debug msg="info: master_replid:f940006536f8a5a1baf08c889b69c7ec1370af55"
time="2022-01-10T14:01:05Z" level=debug msg="info: master_replid2:0000000000000000000000000000000000000000"
time="2022-01-10T14:01:05Z" level=debug msg="info: master_repl_offset:606356374"
time="2022-01-10T14:01:05Z" level=debug msg="info: second_repl_offset:-1"
time="2022-01-10T14:01:05Z" level=debug msg="info: repl_backlog_active:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: repl_backlog_size:1048576"
time="2022-01-10T14:01:05Z" level=debug msg="info: repl_backlog_first_byte_offset:605307799"
time="2022-01-10T14:01:05Z" level=debug msg="info: repl_backlog_histlen:1048576"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # CPU"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: CPU"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_cpu_sys:14378.524887"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_cpu_user:55102.294431"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_cpu_sys_children:0.000000"
time="2022-01-10T14:01:05Z" level=debug msg="info: used_cpu_user_children:0.001069"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Modules"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Modules"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Commandstats"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Commandstats"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_slowlog:calls=158,usec=2432,usec_per_call=15.39"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_config:calls=9832419,usec=201002166,usec_per_call=20.44"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_exists:calls=1770,usec=7886,usec_per_call=4.46"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_command:calls=1,usec=1215,usec_per_call=1215.00"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_info:calls=13183859,usec=5099318587,usec_per_call=386.79"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_latency:calls=80,usec=1250,usec_per_call=15.62"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_get:calls=24557,usec=93931,usec_per_call=3.83"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_cluster:calls=4378,usec=442967,usec_per_call=101.18"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_replconf:calls=10603786,usec=16838445,usec_per_call=1.59"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_set:calls=14698,usec=89843,usec_per_call=6.11"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_auth:calls=14256877,usec=104291297,usec_per_call=7.32"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_psync:calls=1,usec=589,usec_per_call=589.00"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_ping:calls=39778899,usec=35132612,usec_per_call=0.88"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_client:calls=354807,usec=11947642,usec_per_call=33.67"
time="2022-01-10T14:01:05Z" level=debug msg="info: cmdstat_hello:calls=22,usec=306,usec_per_call=13.91"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Cluster"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Cluster"
time="2022-01-10T14:01:05Z" level=debug msg="info: cluster_enabled:1"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="info: # Keyspace"
time="2022-01-10T14:01:05Z" level=debug msg="set fieldClass: Keyspace"
time="2022-01-10T14:01:05Z" level=debug msg="info: db0:keys=64,expires=0,avg_ttl=0"
time="2022-01-10T14:01:05Z" level=debug msg="parseDBKeyspaceString inputKey: [db0] inputVal: [keys=64,expires=0,avg_ttl=0]"
time="2022-01-10T14:01:05Z" level=debug msg="info: "
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - running command: LATENCY [LATEST]"
time="2022-01-10T14:01:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:05Z" level=debug msg="Creating cluster object"
time="2022-01-10T14:01:05Z" level=debug msg="Running refresh on cluster object"
time="2022-01-10T14:01:20Z" level=error msg="Cluster refresh failed: redisc: all nodes failed\nread tcp 10.0.37.233:52556->10.0.53.93:6379: i/o timeout"
time="2022-01-10T14:01:20Z" level=debug msg="Creating redis connection object"
time="2022-01-10T14:01:20Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:20Z" level=debug msg="keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:20Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:20Z" level=debug msg="e.singleKeys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:20Z" level=debug msg="e.keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:20Z" level=debug msg="allKeys: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: SLOWLOG [LEN]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: SLOWLOG [GET 1]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:20Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:20Z" level=debug msg="allStreams: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:20Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:20Z" level=debug msg="scrapeRedisHost() done"
time="2022-01-10T14:01:20Z" level=debug msg="Trying DialURL(): rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:20Z" level=debug msg="connected to: rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:20Z" level=debug msg="connecting took 0.006506 seconds"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: CLIENT [SETNAME redis_exporter]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: CONFIG [GET *]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="Redis CONFIG err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: INFO [ALL]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="Redis INFO ALL result: [\"# Server\\r\\nredis_version:6.0.5\\r\\nredis_git_sha1:0\\r\\nredis_git_dirty:0\\r\\nredis_build_id:0\\r\\nredis_mode:cluster\\r\\nos:Amazon ElastiCache\\r\\narch_bits:64\\r\\nmultiplexing_api:epoll\\r\\natomicvar_api:atomic-builtin\\r\\ngcc_version:0.0.0\\r\\nprocess_id:1\\r\\nrun_id:a20787f1c26718fd8c3b27a0c7dc33fce98ed847\\r\\ntcp_port:6379\\r\\nuptime_in_seconds:10641637\\r\\nuptime_in_days:123\\r\\nhz:10\\r\\nconfigured_hz:10\\r\\nlru_clock:14433328\\r\\nexecutable:-\\r\\nconfig_file:-\\r\\n\\r\\n# Clients\\r\\nconnected_clients:43\\r\\nclient_recent_max_input_buffer:2\\r\\nclient_recent_max_output_buffer:0\\r\\nblocked_clients:0\\r\\ntracking_clients:0\\r\\nclients_in_timeout_table:0\\r\\n\\r\\n# Memory\\r\\nused_memory:7250504\\r\\nused_memory_human:6.91M\\r\\nused_memory_rss:12255232\\r\\nused_memory_rss_human:11.69M\\r\\nused_memory_peak:44823168\\r\\nused_memory_peak_human:42.75M\\r\\nused_memory_peak_perc:16.18%\\r\\nused_memory_overhead:6872326\\r\\nused_memory_startup:5075656\\r\\nused_memory_dataset:378178\\r\\nused_memory_dataset_perc:17.39%\\r\\nallocator_allocated:15436760\\r\\nallocator_active:16760832\\r\\nallocator_resident:20316160\\r\\nused_memory_lua:37888\\r\\nused_memory_lua_human:37.00K\\r\\nused_memory_scripts:0\\r\\nused_memory_scripts_human:0B\\r\\nnumber_of_cached_scripts:0\\r\\nmaxmemory:436469760\\r\\nmaxmemory_human:416.25M\\r\\nmaxmemory_policy:volatile-lru\\r\\nallocator_frag_ratio:1.09\\r\\nallocator_frag_bytes:1324072\\r\\nallocator_rss_ratio:1.21\\r\\nallocator_rss_bytes:3555328\\r\\nrss_overhead_ratio:0.60\\r\\nrss_overhead_bytes:-8060928\\r\\nmem_fragmentation_ratio:1.70\\r\\nmem_fragmentation_bytes:5047264\\r\\nmem_not_counted_for_evict:0\\r\\nmem_replication_backlog:1048576\\r\\nmem_clients_slaves:17242\\r\\nmem_clients_normal:724164\\r\\nmem_aof_buffer:0\\r\\nmem_allocator:jemalloc-5.1.0\\r\\nactive_defrag_running:0\\r\\nlazyfree_pending_objects:0\\r\\n\\r\\n# Persistence\\r\\nloading:0\\r\\nrdb_changes_since_last_save:14698\\r\\nrdb_bgsave_in_progress:0\\r\\nrdb_last_save_time:1631181643\\r\\nrdb_last_bgsave_status:ok\\r\\nrdb_last_bgsave_time_sec:-1\\r\\nrdb_current_bgsave_time_sec:-1\\r\\nrdb_last_cow_size:380928\\r\\naof_enabled:0\\r\\naof_rewrite_in_progress:0\\r\\naof_rewrite_scheduled:0\\r\\naof_last_rewrite_time_sec:-1\\r\\naof_current_rewrite_time_sec:-1\\r\\naof_last_bgrewrite_status:ok\\r\\naof_last_write_status:ok\\r\\naof_last_cow_size:0\\r\\nmodule_fork_in_progress:0\\r\\nmodule_fork_last_cow_size:0\\r\\n\\r\\n# SSL\\r\\nssl_enabled:yes\\r\\nssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT\\r\\nssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT\\r\\nssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0\\r\\n\\r\\n# Stats\\r\\ntotal_connections_received:13901471\\r\\ntotal_commands_processed:27745353\\r\\ninstantaneous_ops_per_sec:0\\r\\ntotal_net_input_bytes:5126776232\\r\\ntotal_net_output_bytes:231167158905\\r\\ninstantaneous_input_kbps:0.34\\r\\ninstantaneous_output_kbps:21.70\\r\\nrejected_connections:0\\r\\nsync_full:1\\r\\nsync_partial_ok:0\\r\\nsync_partial_err:1\\r\\nexpired_keys:2186\\r\\nexpired_stale_perc:0.00\\r\\nexpired_time_cap_reached_count:0\\r\\nexpire_cycle_cpu_milliseconds:156512\\r\\nevicted_keys:0\\r\\nkeyspace_hits:9047\\r\\nkeyspace_misses:17280\\r\\npubsub_channels:0\\r\\npubsub_patterns:0\\r\\nlatest_fork_usec:265\\r\\nmigrate_cached_sockets:0\\r\\nslave_expires_tracked_keys:0\\r\\nactive_defrag_hits:0\\r\\nactive_defrag_misses:0\\r\\nactive_defrag_key_hits:0\\r\\nactive_defrag_key_misses:0\\r\\ntracking_total_keys:0\\r\\ntracking_total_items:0\\r\\ntracking_total_prefixes:0\\r\\nunexpected_error_replies:0\\r\\n\\r\\n# Replication\\r\\nrole:master\\r\\nconnected_slaves:1\\r\\nslave0:ip=10.17.1.161,port=6379,state=online,offset=606357183,lag=0\\r\\nmaster_replid:f940006536f8a5a1baf08c889b69c7ec1370af55\\r\\nmaster_replid2:0000000000000000000000000000000000000000\\r\\nmaster_repl_offset:606357183\\r\\nsecond_repl_offset:-1\\r\\nrepl_backlog_active:1\\r\\nrepl_backlog_size:1048576\\r\\nrepl_backlog_first_byte_offset:605308608\\r\\nrepl_backlog_histlen:1048576\\r\\n\\r\\n# CPU\\r\\nused_cpu_sys:14378.543138\\r\\nused_cpu_user:55102.339845\\r\\nused_cpu_sys_children:0.000000\\r\\nused_cpu_user_children:0.001069\\r\\n\\r\\n# Modules\\r\\n\\r\\n# Commandstats\\r\\ncmdstat_slowlog:calls=160,usec=2440,usec_per_call=15.25\\r\\ncmdstat_config:calls=9832452,usec=201002820,usec_per_call=20.44\\r\\ncmdstat_exists:calls=1770,usec=7886,usec_per_call=4.46\\r\\ncmdstat_command:calls=1,usec=1215,usec_per_call=1215.00\\r\\ncmdstat_info:calls=13183878,usec=5099324971,usec_per_call=386.78\\r\\ncmdstat_latency:calls=81,usec=1252,usec_per_call=15.46\\r\\ncmdstat_get:calls=24557,usec=93931,usec_per_call=3.83\\r\\ncmdstat_cluster:calls=4379,usec=443004,usec_per_call=101.17\\r\\ncmdstat_replconf:calls=10603801,usec=16838468,usec_per_call=1.59\\r\\ncmdstat_set:calls=14698,usec=89843,usec_per_call=6.11\\r\\ncmdstat_auth:calls=14256878,usec=104291305,usec_per_call=7.32\\r\\ncmdstat_psync:calls=1,usec=589,usec_per_call=589.00\\r\\ncmdstat_ping:calls=39778952,usec=35132673,usec_per_call=0.88\\r\\ncmdstat_client:calls=354808,usec=11947643,usec_per_call=33.67\\r\\ncmdstat_hello:calls=22,usec=306,usec_per_call=13.91\\r\\n\\r\\n# Cluster\\r\\ncluster_enabled:1\\r\\n\\r\\n# Keyspace\\r\\ndb0:keys=64,expires=0,avg_ttl=0\\r\\n\"]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: CLUSTER [INFO]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_state:ok"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_slots_assigned:16384"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_slots_ok:16384"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_slots_pfail:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_slots_fail:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_known_nodes:6"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_size:3"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_current_epoch:3"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_my_epoch:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_ping_sent:10603963"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_pong_sent:10584077"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_update_sent:117"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_sent:21188157"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_ping_received:10584074"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_pong_received:10609424"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_meet_received:3"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_update_received:263"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_stats_messages_received:21193764"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="dbCount: 1"
time="2022-01-10T14:01:20Z" level=debug msg="info: # Server"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Server"
time="2022-01-10T14:01:20Z" level=debug msg="info: redis_version:6.0.5"
time="2022-01-10T14:01:20Z" level=debug msg="info: redis_git_sha1:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: redis_git_dirty:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: redis_build_id:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: redis_mode:cluster"
time="2022-01-10T14:01:20Z" level=debug msg="info: os:Amazon ElastiCache"
time="2022-01-10T14:01:20Z" level=debug msg="info: arch_bits:64"
time="2022-01-10T14:01:20Z" level=debug msg="info: multiplexing_api:epoll"
time="2022-01-10T14:01:20Z" level=debug msg="info: atomicvar_api:atomic-builtin"
time="2022-01-10T14:01:20Z" level=debug msg="info: gcc_version:0.0.0"
time="2022-01-10T14:01:20Z" level=debug msg="info: process_id:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: run_id:a20787f1c26718fd8c3b27a0c7dc33fce98ed847"
time="2022-01-10T14:01:20Z" level=debug msg="info: tcp_port:6379"
time="2022-01-10T14:01:20Z" level=debug msg="info: uptime_in_seconds:10641637"
time="2022-01-10T14:01:20Z" level=debug msg="info: uptime_in_days:123"
time="2022-01-10T14:01:20Z" level=debug msg="info: hz:10"
time="2022-01-10T14:01:20Z" level=debug msg="info: configured_hz:10"
time="2022-01-10T14:01:20Z" level=debug msg="info: lru_clock:14433328"
time="2022-01-10T14:01:20Z" level=debug msg="info: executable:-"
time="2022-01-10T14:01:20Z" level=debug msg="info: config_file:-"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Clients"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Clients"
time="2022-01-10T14:01:20Z" level=debug msg="info: connected_clients:43"
time="2022-01-10T14:01:20Z" level=debug msg="info: client_recent_max_input_buffer:2"
time="2022-01-10T14:01:20Z" level=debug msg="info: client_recent_max_output_buffer:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: blocked_clients:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: tracking_clients:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: clients_in_timeout_table:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Memory"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Memory"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory:7250504"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_human:6.91M"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_rss:12255232"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_rss_human:11.69M"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_peak:44823168"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_peak_human:42.75M"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_peak_perc:16.18%"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_overhead:6872326"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_startup:5075656"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_dataset:378178"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_dataset_perc:17.39%"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_allocated:15436760"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_active:16760832"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_resident:20316160"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_lua:37888"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_lua_human:37.00K"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_scripts:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_memory_scripts_human:0B"
time="2022-01-10T14:01:20Z" level=debug msg="info: number_of_cached_scripts:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: maxmemory:436469760"
time="2022-01-10T14:01:20Z" level=debug msg="info: maxmemory_human:416.25M"
time="2022-01-10T14:01:20Z" level=debug msg="info: maxmemory_policy:volatile-lru"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_frag_ratio:1.09"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_frag_bytes:1324072"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_rss_ratio:1.21"
time="2022-01-10T14:01:20Z" level=debug msg="info: allocator_rss_bytes:3555328"
time="2022-01-10T14:01:20Z" level=debug msg="info: rss_overhead_ratio:0.60"
time="2022-01-10T14:01:20Z" level=debug msg="info: rss_overhead_bytes:-8060928"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_fragmentation_ratio:1.70"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_fragmentation_bytes:5047264"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_not_counted_for_evict:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_replication_backlog:1048576"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_clients_slaves:17242"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_clients_normal:724164"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_aof_buffer:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: mem_allocator:jemalloc-5.1.0"
time="2022-01-10T14:01:20Z" level=debug msg="info: active_defrag_running:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: lazyfree_pending_objects:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Persistence"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Persistence"
time="2022-01-10T14:01:20Z" level=debug msg="info: loading:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_changes_since_last_save:14698"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_bgsave_in_progress:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_last_save_time:1631181643"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_last_bgsave_status:ok"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_last_bgsave_time_sec:-1"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_current_bgsave_time_sec:-1"
time="2022-01-10T14:01:20Z" level=debug msg="info: rdb_last_cow_size:380928"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_enabled:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_rewrite_in_progress:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_rewrite_scheduled:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_last_rewrite_time_sec:-1"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_current_rewrite_time_sec:-1"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_last_bgrewrite_status:ok"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_last_write_status:ok"
time="2022-01-10T14:01:20Z" level=debug msg="info: aof_last_cow_size:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: module_fork_in_progress:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: module_fork_last_cow_size:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # SSL"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: SSL"
time="2022-01-10T14:01:20Z" level=debug msg="info: ssl_enabled:yes"
time="2022-01-10T14:01:20Z" level=debug msg="info: ssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT"
time="2022-01-10T14:01:20Z" level=debug msg="info: ssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT"
time="2022-01-10T14:01:20Z" level=debug msg="info: ssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Stats"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Stats"
time="2022-01-10T14:01:20Z" level=debug msg="info: total_connections_received:13901471"
time="2022-01-10T14:01:20Z" level=debug msg="info: total_commands_processed:27745353"
time="2022-01-10T14:01:20Z" level=debug msg="info: instantaneous_ops_per_sec:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: total_net_input_bytes:5126776232"
time="2022-01-10T14:01:20Z" level=debug msg="info: total_net_output_bytes:231167158905"
time="2022-01-10T14:01:20Z" level=debug msg="info: instantaneous_input_kbps:0.34"
time="2022-01-10T14:01:20Z" level=debug msg="info: instantaneous_output_kbps:21.70"
time="2022-01-10T14:01:20Z" level=debug msg="info: rejected_connections:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: sync_full:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: sync_partial_ok:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: sync_partial_err:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: expired_keys:2186"
time="2022-01-10T14:01:20Z" level=debug msg="info: expired_stale_perc:0.00"
time="2022-01-10T14:01:20Z" level=debug msg="info: expired_time_cap_reached_count:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: expire_cycle_cpu_milliseconds:156512"
time="2022-01-10T14:01:20Z" level=debug msg="info: evicted_keys:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: keyspace_hits:9047"
time="2022-01-10T14:01:20Z" level=debug msg="info: keyspace_misses:17280"
time="2022-01-10T14:01:20Z" level=debug msg="info: pubsub_channels:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: pubsub_patterns:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: latest_fork_usec:265"
time="2022-01-10T14:01:20Z" level=debug msg="info: migrate_cached_sockets:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: slave_expires_tracked_keys:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: active_defrag_hits:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: active_defrag_misses:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: active_defrag_key_hits:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: active_defrag_key_misses:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: tracking_total_keys:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: tracking_total_items:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: tracking_total_prefixes:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: unexpected_error_replies:0"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Replication"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Replication"
time="2022-01-10T14:01:20Z" level=debug msg="info: role:master"
time="2022-01-10T14:01:20Z" level=debug msg="info: connected_slaves:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: slave0:ip=10.17.1.161,port=6379,state=online,offset=606357183,lag=0"
time="2022-01-10T14:01:20Z" level=debug msg="info: master_replid:f940006536f8a5a1baf08c889b69c7ec1370af55"
time="2022-01-10T14:01:20Z" level=debug msg="info: master_replid2:0000000000000000000000000000000000000000"
time="2022-01-10T14:01:20Z" level=debug msg="info: master_repl_offset:606357183"
time="2022-01-10T14:01:20Z" level=debug msg="info: second_repl_offset:-1"
time="2022-01-10T14:01:20Z" level=debug msg="info: repl_backlog_active:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: repl_backlog_size:1048576"
time="2022-01-10T14:01:20Z" level=debug msg="info: repl_backlog_first_byte_offset:605308608"
time="2022-01-10T14:01:20Z" level=debug msg="info: repl_backlog_histlen:1048576"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # CPU"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: CPU"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_cpu_sys:14378.543138"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_cpu_user:55102.339845"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_cpu_sys_children:0.000000"
time="2022-01-10T14:01:20Z" level=debug msg="info: used_cpu_user_children:0.001069"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Modules"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Modules"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Commandstats"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Commandstats"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_slowlog:calls=160,usec=2440,usec_per_call=15.25"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_config:calls=9832452,usec=201002820,usec_per_call=20.44"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_exists:calls=1770,usec=7886,usec_per_call=4.46"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_command:calls=1,usec=1215,usec_per_call=1215.00"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_info:calls=13183878,usec=5099324971,usec_per_call=386.78"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_latency:calls=81,usec=1252,usec_per_call=15.46"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_get:calls=24557,usec=93931,usec_per_call=3.83"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_cluster:calls=4379,usec=443004,usec_per_call=101.17"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_replconf:calls=10603801,usec=16838468,usec_per_call=1.59"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_set:calls=14698,usec=89843,usec_per_call=6.11"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_auth:calls=14256878,usec=104291305,usec_per_call=7.32"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_psync:calls=1,usec=589,usec_per_call=589.00"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_ping:calls=39778952,usec=35132673,usec_per_call=0.88"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_client:calls=354808,usec=11947643,usec_per_call=33.67"
time="2022-01-10T14:01:20Z" level=debug msg="info: cmdstat_hello:calls=22,usec=306,usec_per_call=13.91"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Cluster"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Cluster"
time="2022-01-10T14:01:20Z" level=debug msg="info: cluster_enabled:1"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="info: # Keyspace"
time="2022-01-10T14:01:20Z" level=debug msg="set fieldClass: Keyspace"
time="2022-01-10T14:01:20Z" level=debug msg="info: db0:keys=64,expires=0,avg_ttl=0"
time="2022-01-10T14:01:20Z" level=debug msg="parseDBKeyspaceString inputKey: [db0] inputVal: [keys=64,expires=0,avg_ttl=0]"
time="2022-01-10T14:01:20Z" level=debug msg="info: "
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - running command: LATENCY [LATEST]"
time="2022-01-10T14:01:20Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:20Z" level=debug msg="Creating cluster object"
time="2022-01-10T14:01:20Z" level=debug msg="Running refresh on cluster object"
time="2022-01-10T14:01:35Z" level=error msg="Cluster refresh failed: redisc: all nodes failed\nread tcp 10.0.37.233:59672->10.0.55.138:6379: i/o timeout"
time="2022-01-10T14:01:35Z" level=debug msg="Creating redis connection object"
time="2022-01-10T14:01:35Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:35Z" level=debug msg="keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:35Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:35Z" level=debug msg="e.singleKeys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:35Z" level=debug msg="e.keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:35Z" level=debug msg="allKeys: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: SLOWLOG [LEN]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: SLOWLOG [GET 1]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:35Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:35Z" level=debug msg="allStreams: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:35Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:35Z" level=debug msg="scrapeRedisHost() done"
time="2022-01-10T14:01:35Z" level=debug msg="Trying DialURL(): rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:35Z" level=debug msg="connected to: rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:35Z" level=debug msg="connecting took 0.005641 seconds"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: CLIENT [SETNAME redis_exporter]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: CONFIG [GET *]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="Redis CONFIG err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: INFO [ALL]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="Redis INFO ALL result: [\"# Server\\r\\nredis_version:6.0.5\\r\\nredis_git_sha1:0\\r\\nredis_git_dirty:0\\r\\nredis_build_id:0\\r\\nredis_mode:cluster\\r\\nos:Amazon ElastiCache\\r\\narch_bits:64\\r\\nmultiplexing_api:epoll\\r\\natomicvar_api:atomic-builtin\\r\\ngcc_version:0.0.0\\r\\nprocess_id:1\\r\\nrun_id:f71639f5727cf3e762af7206e0ed0488cfb29e21\\r\\ntcp_port:6379\\r\\nuptime_in_seconds:10641630\\r\\nuptime_in_days:123\\r\\nhz:10\\r\\nconfigured_hz:10\\r\\nlru_clock:14433343\\r\\nexecutable:-\\r\\nconfig_file:-\\r\\n\\r\\n# Clients\\r\\nconnected_clients:37\\r\\nclient_recent_max_input_buffer:4\\r\\nclient_recent_max_output_buffer:0\\r\\nblocked_clients:0\\r\\ntracking_clients:0\\r\\nclients_in_timeout_table:0\\r\\n\\r\\n# Memory\\r\\nused_memory:8302536\\r\\nused_memory_human:7.92M\\r\\nused_memory_rss:10616832\\r\\nused_memory_rss_human:10.12M\\r\\nused_memory_peak:44823792\\r\\nused_memory_peak_human:42.75M\\r\\nused_memory_peak_perc:18.52%\\r\\nused_memory_overhead:6390074\\r\\nused_memory_startup:5075656\\r\\nused_memory_dataset:1912462\\r\\nused_memory_dataset_perc:59.27%\\r\\nallocator_allocated:12023680\\r\\nallocator_active:13217792\\r\\nallocator_resident:16855040\\r\\nused_memory_lua:37888\\r\\nused_memory_lua_human:37.00K\\r\\nused_memory_scripts:0\\r\\nused_memory_scripts_human:0B\\r\\nnumber_of_cached_scripts:0\\r\\nmaxmemory:436469760\\r\\nmaxmemory_human:416.25M\\r\\nmaxmemory_policy:volatile-lru\\r\\nallocator_frag_ratio:1.10\\r\\nallocator_frag_bytes:1194112\\r\\nallocator_rss_ratio:1.28\\r\\nallocator_rss_bytes:3637248\\r\\nrss_overhead_ratio:0.63\\r\\nrss_overhead_bytes:-6238208\\r\\nmem_fragmentation_ratio:1.33\\r\\nmem_fragmentation_bytes:2607856\\r\\nmem_not_counted_for_evict:0\\r\\nmem_replication_backlog:1048576\\r\\nmem_clients_slaves:0\\r\\nmem_clients_normal:258634\\r\\nmem_aof_buffer:0\\r\\nmem_allocator:jemalloc-5.1.0\\r\\nactive_defrag_running:0\\r\\nlazyfree_pending_objects:0\\r\\n\\r\\n# Persistence\\r\\nloading:0\\r\\nrdb_changes_since_last_save:0\\r\\nrdb_bgsave_in_progress:0\\r\\nrdb_last_save_time:1641772915\\r\\nrdb_last_bgsave_status:ok\\r\\nrdb_last_bgsave_time_sec:0\\r\\nrdb_current_bgsave_time_sec:-1\\r\\nrdb_last_cow_size:802816\\r\\naof_enabled:0\\r\\naof_rewrite_in_progress:0\\r\\naof_rewrite_scheduled:0\\r\\naof_last_rewrite_time_sec:-1\\r\\naof_current_rewrite_time_sec:-1\\r\\naof_last_bgrewrite_status:ok\\r\\naof_last_write_status:ok\\r\\naof_last_cow_size:0\\r\\nmodule_fork_in_progress:0\\r\\nmodule_fork_last_cow_size:0\\r\\n\\r\\n# SSL\\r\\nssl_enabled:yes\\r\\nssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT\\r\\nssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT\\r\\nssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0\\r\\n\\r\\n# Stats\\r\\ntotal_connections_received:13901482\\r\\ntotal_commands_processed:27660746\\r\\ninstantaneous_ops_per_sec:16\\r\\ntotal_net_input_bytes:5261135013\\r\\ntotal_net_output_bytes:219405965118\\r\\ninstantaneous_input_kbps:2.43\\r\\ninstantaneous_output_kbps:20.79\\r\\nrejected_connections:0\\r\\nsync_full:0\\r\\nsync_partial_ok:0\\r\\nsync_partial_err:0\\r\\nexpired_keys:0\\r\\nexpired_stale_perc:0.00\\r\\nexpired_time_cap_reached_count:0\\r\\nexpire_cycle_cpu_milliseconds:1\\r\\nevicted_keys:0\\r\\nkeyspace_hits:0\\r\\nkeyspace_misses:0\\r\\npubsub_channels:0\\r\\npubsub_patterns:0\\r\\nlatest_fork_usec:541\\r\\nmigrate_cached_sockets:0\\r\\nslave_expires_tracked_keys:0\\r\\nactive_defrag_hits:0\\r\\nactive_defrag_misses:0\\r\\nactive_defrag_key_hits:0\\r\\nactive_defrag_key_misses:0\\r\\ntracking_total_keys:0\\r\\ntracking_total_items:0\\r\\ntracking_total_prefixes:0\\r\\nunexpected_error_replies:0\\r\\n\\r\\n# Replication\\r\\nrole:slave\\r\\nmaster_host:redis-hub-dev-0001-001.xxxxxx.cache.amazonaws.com\\r\\nmaster_port:6379\\r\\nmaster_link_status:up\\r\\nmaster_last_io_seconds_ago:0\\r\\nmaster_sync_in_progress:0\\r\\nslave_repl_offset:605375033\\r\\nrepl_sync_enabled:1\\r\\nslave_read_reploff:605375033\\r\\nslave_priority:100\\r\\nslave_read_only:1\\r\\nconnected_slaves:0\\r\\nmaster_replid:9e9a3de7020afedc390a4f5111ac47393bb94300\\r\\nmaster_replid2:0000000000000000000000000000000000000000\\r\\nmaster_repl_offset:605375033\\r\\nsecond_repl_offset:-1\\r\\nrepl_backlog_active:1\\r\\nrepl_backlog_size:1048576\\r\\nrepl_backlog_first_byte_offset:604326458\\r\\nrepl_backlog_histlen:1048576\\r\\n\\r\\n# CPU\\r\\nused_cpu_sys:16801.942116\\r\\nused_cpu_user:51935.326839\\r\\nused_cpu_sys_children:0.122924\\r\\nused_cpu_user_children:0.224317\\r\\n\\r\\n# Modules\\r\\n\\r\\n# Commandstats\\r\\ncmdstat_latency:calls=87,usec=183,usec_per_call=2.10\\r\\ncmdstat_del:calls=2097,usec=20281,usec_per_call=9.67\\r\\ncmdstat_select:calls=1,usec=1,usec_per_call=1.00\\r\\ncmdstat_hello:calls=27,usec=360,usec_per_call=13.33\\r\\ncmdstat_bgsave:calls=125,usec=91871,usec_per_call=734.97\\r\\ncmdstat_client:calls=354818,usec=10932652,usec_per_call=30.81\\r\\ncmdstat_info:calls=13184252,usec=4687009525,usec_per_call=355.50\\r\\ncmdstat_config:calls=9869057,usec=207443039,usec_per_call=21.02\\r\\ncmdstat_auth:calls=14261018,usec=100290795,usec_per_call=7.03\\r\\ncmdstat_slowlog:calls=172,usec=686,usec_per_call=3.99\\r\\ncmdstat_ping:calls=40822130,usec=33705617,usec_per_call=0.83\\r\\ncmdstat_cluster:calls=4543,usec=458247,usec_per_call=100.87\\r\\ncmdstat_replconf:calls=10602564,usec=18951385,usec_per_call=1.79\\r\\ncmdstat_set:calls=13613,usec=100207,usec_per_call=7.36\\r\\n\\r\\n# Cluster\\r\\ncluster_enabled:1\\r\\n\\r\\n# Keyspace\\r\\ndb0:keys=77,expires=0,avg_ttl=0\\r\\n\"]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: CLUSTER [INFO]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_state:ok"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_slots_assigned:16384"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_slots_ok:16384"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_slots_pfail:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_slots_fail:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_known_nodes:6"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_size:3"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_current_epoch:3"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_my_epoch:2"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_ping_sent:10606432"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_pong_sent:10664207"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_meet_sent:2"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_update_sent:95"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_sent:21270736"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_ping_received:10664207"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_pong_received:10611895"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_update_received:166"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_stats_messages_received:21276268"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="dbCount: 1"
time="2022-01-10T14:01:35Z" level=debug msg="info: # Server"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Server"
time="2022-01-10T14:01:35Z" level=debug msg="info: redis_version:6.0.5"
time="2022-01-10T14:01:35Z" level=debug msg="info: redis_git_sha1:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: redis_git_dirty:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: redis_build_id:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: redis_mode:cluster"
time="2022-01-10T14:01:35Z" level=debug msg="info: os:Amazon ElastiCache"
time="2022-01-10T14:01:35Z" level=debug msg="info: arch_bits:64"
time="2022-01-10T14:01:35Z" level=debug msg="info: multiplexing_api:epoll"
time="2022-01-10T14:01:35Z" level=debug msg="info: atomicvar_api:atomic-builtin"
time="2022-01-10T14:01:35Z" level=debug msg="info: gcc_version:0.0.0"
time="2022-01-10T14:01:35Z" level=debug msg="info: process_id:1"
time="2022-01-10T14:01:35Z" level=debug msg="info: run_id:f71639f5727cf3e762af7206e0ed0488cfb29e21"
time="2022-01-10T14:01:35Z" level=debug msg="info: tcp_port:6379"
time="2022-01-10T14:01:35Z" level=debug msg="info: uptime_in_seconds:10641630"
time="2022-01-10T14:01:35Z" level=debug msg="info: uptime_in_days:123"
time="2022-01-10T14:01:35Z" level=debug msg="info: hz:10"
time="2022-01-10T14:01:35Z" level=debug msg="info: configured_hz:10"
time="2022-01-10T14:01:35Z" level=debug msg="info: lru_clock:14433343"
time="2022-01-10T14:01:35Z" level=debug msg="info: executable:-"
time="2022-01-10T14:01:35Z" level=debug msg="info: config_file:-"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Clients"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Clients"
time="2022-01-10T14:01:35Z" level=debug msg="info: connected_clients:37"
time="2022-01-10T14:01:35Z" level=debug msg="info: client_recent_max_input_buffer:4"
time="2022-01-10T14:01:35Z" level=debug msg="info: client_recent_max_output_buffer:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: blocked_clients:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: tracking_clients:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: clients_in_timeout_table:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Memory"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Memory"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory:8302536"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_human:7.92M"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_rss:10616832"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_rss_human:10.12M"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_peak:44823792"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_peak_human:42.75M"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_peak_perc:18.52%"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_overhead:6390074"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_startup:5075656"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_dataset:1912462"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_dataset_perc:59.27%"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_allocated:12023680"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_active:13217792"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_resident:16855040"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_lua:37888"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_lua_human:37.00K"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_scripts:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_memory_scripts_human:0B"
time="2022-01-10T14:01:35Z" level=debug msg="info: number_of_cached_scripts:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: maxmemory:436469760"
time="2022-01-10T14:01:35Z" level=debug msg="info: maxmemory_human:416.25M"
time="2022-01-10T14:01:35Z" level=debug msg="info: maxmemory_policy:volatile-lru"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_frag_ratio:1.10"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_frag_bytes:1194112"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_rss_ratio:1.28"
time="2022-01-10T14:01:35Z" level=debug msg="info: allocator_rss_bytes:3637248"
time="2022-01-10T14:01:35Z" level=debug msg="info: rss_overhead_ratio:0.63"
time="2022-01-10T14:01:35Z" level=debug msg="info: rss_overhead_bytes:-6238208"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_fragmentation_ratio:1.33"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_fragmentation_bytes:2607856"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_not_counted_for_evict:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_replication_backlog:1048576"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_clients_slaves:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_clients_normal:258634"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_aof_buffer:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: mem_allocator:jemalloc-5.1.0"
time="2022-01-10T14:01:35Z" level=debug msg="info: active_defrag_running:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: lazyfree_pending_objects:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Persistence"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Persistence"
time="2022-01-10T14:01:35Z" level=debug msg="info: loading:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_changes_since_last_save:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_bgsave_in_progress:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_last_save_time:1641772915"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_last_bgsave_status:ok"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_last_bgsave_time_sec:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_current_bgsave_time_sec:-1"
time="2022-01-10T14:01:35Z" level=debug msg="info: rdb_last_cow_size:802816"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_enabled:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_rewrite_in_progress:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_rewrite_scheduled:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_last_rewrite_time_sec:-1"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_current_rewrite_time_sec:-1"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_last_bgrewrite_status:ok"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_last_write_status:ok"
time="2022-01-10T14:01:35Z" level=debug msg="info: aof_last_cow_size:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: module_fork_in_progress:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: module_fork_last_cow_size:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # SSL"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: SSL"
time="2022-01-10T14:01:35Z" level=debug msg="info: ssl_enabled:yes"
time="2022-01-10T14:01:35Z" level=debug msg="info: ssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT"
time="2022-01-10T14:01:35Z" level=debug msg="info: ssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT"
time="2022-01-10T14:01:35Z" level=debug msg="info: ssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Stats"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Stats"
time="2022-01-10T14:01:35Z" level=debug msg="info: total_connections_received:13901482"
time="2022-01-10T14:01:35Z" level=debug msg="info: total_commands_processed:27660746"
time="2022-01-10T14:01:35Z" level=debug msg="info: instantaneous_ops_per_sec:16"
time="2022-01-10T14:01:35Z" level=debug msg="info: total_net_input_bytes:5261135013"
time="2022-01-10T14:01:35Z" level=debug msg="info: total_net_output_bytes:219405965118"
time="2022-01-10T14:01:35Z" level=debug msg="info: instantaneous_input_kbps:2.43"
time="2022-01-10T14:01:35Z" level=debug msg="info: instantaneous_output_kbps:20.79"
time="2022-01-10T14:01:35Z" level=debug msg="info: rejected_connections:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: sync_full:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: sync_partial_ok:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: sync_partial_err:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: expired_keys:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: expired_stale_perc:0.00"
time="2022-01-10T14:01:35Z" level=debug msg="info: expired_time_cap_reached_count:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: expire_cycle_cpu_milliseconds:1"
time="2022-01-10T14:01:35Z" level=debug msg="info: evicted_keys:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: keyspace_hits:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: keyspace_misses:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: pubsub_channels:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: pubsub_patterns:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: latest_fork_usec:541"
time="2022-01-10T14:01:35Z" level=debug msg="info: migrate_cached_sockets:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: slave_expires_tracked_keys:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: active_defrag_hits:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: active_defrag_misses:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: active_defrag_key_hits:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: active_defrag_key_misses:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: tracking_total_keys:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: tracking_total_items:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: tracking_total_prefixes:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: unexpected_error_replies:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Replication"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Replication"
time="2022-01-10T14:01:35Z" level=debug msg="info: role:slave"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_host:redis-hub-dev-0001-001.xxxxxx.cache.amazonaws.com"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_port:6379"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_link_status:up"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_last_io_seconds_ago:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_sync_in_progress:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: slave_repl_offset:605375033"
time="2022-01-10T14:01:35Z" level=debug msg="info: repl_sync_enabled:1"
time="2022-01-10T14:01:35Z" level=debug msg="info: slave_read_reploff:605375033"
time="2022-01-10T14:01:35Z" level=debug msg="info: slave_priority:100"
time="2022-01-10T14:01:35Z" level=debug msg="info: slave_read_only:1"
time="2022-01-10T14:01:35Z" level=debug msg="info: connected_slaves:0"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_replid:9e9a3de7020afedc390a4f5111ac47393bb94300"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_replid2:0000000000000000000000000000000000000000"
time="2022-01-10T14:01:35Z" level=debug msg="info: master_repl_offset:605375033"
time="2022-01-10T14:01:35Z" level=debug msg="info: second_repl_offset:-1"
time="2022-01-10T14:01:35Z" level=debug msg="info: repl_backlog_active:1"
time="2022-01-10T14:01:35Z" level=debug msg="info: repl_backlog_size:1048576"
time="2022-01-10T14:01:35Z" level=debug msg="info: repl_backlog_first_byte_offset:604326458"
time="2022-01-10T14:01:35Z" level=debug msg="info: repl_backlog_histlen:1048576"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # CPU"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: CPU"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_cpu_sys:16801.942116"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_cpu_user:51935.326839"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_cpu_sys_children:0.122924"
time="2022-01-10T14:01:35Z" level=debug msg="info: used_cpu_user_children:0.224317"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Modules"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Modules"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Commandstats"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Commandstats"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_latency:calls=87,usec=183,usec_per_call=2.10"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_del:calls=2097,usec=20281,usec_per_call=9.67"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_select:calls=1,usec=1,usec_per_call=1.00"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_hello:calls=27,usec=360,usec_per_call=13.33"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_bgsave:calls=125,usec=91871,usec_per_call=734.97"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_client:calls=354818,usec=10932652,usec_per_call=30.81"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_info:calls=13184252,usec=4687009525,usec_per_call=355.50"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_config:calls=9869057,usec=207443039,usec_per_call=21.02"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_auth:calls=14261018,usec=100290795,usec_per_call=7.03"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_slowlog:calls=172,usec=686,usec_per_call=3.99"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_ping:calls=40822130,usec=33705617,usec_per_call=0.83"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_cluster:calls=4543,usec=458247,usec_per_call=100.87"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_replconf:calls=10602564,usec=18951385,usec_per_call=1.79"
time="2022-01-10T14:01:35Z" level=debug msg="info: cmdstat_set:calls=13613,usec=100207,usec_per_call=7.36"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Cluster"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Cluster"
time="2022-01-10T14:01:35Z" level=debug msg="info: cluster_enabled:1"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="info: # Keyspace"
time="2022-01-10T14:01:35Z" level=debug msg="set fieldClass: Keyspace"
time="2022-01-10T14:01:35Z" level=debug msg="info: db0:keys=77,expires=0,avg_ttl=0"
time="2022-01-10T14:01:35Z" level=debug msg="parseDBKeyspaceString inputKey: [db0] inputVal: [keys=77,expires=0,avg_ttl=0]"
time="2022-01-10T14:01:35Z" level=debug msg="info: "
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - running command: LATENCY [LATEST]"
time="2022-01-10T14:01:35Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:35Z" level=debug msg="Creating cluster object"
time="2022-01-10T14:01:35Z" level=debug msg="Running refresh on cluster object"
time="2022-01-10T14:01:50Z" level=error msg="Cluster refresh failed: redisc: all nodes failed\nread tcp 10.0.37.233:52814->10.0.53.93:6379: i/o timeout"
time="2022-01-10T14:01:50Z" level=debug msg="Creating redis connection object"
time="2022-01-10T14:01:50Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:50Z" level=debug msg="keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:50Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:50Z" level=debug msg="e.singleKeys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:50Z" level=debug msg="e.keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:01:50Z" level=debug msg="allKeys: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: SLOWLOG [LEN]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: SLOWLOG [GET 1]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:50Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:50Z" level=debug msg="allStreams: []exporter.dbKeyPair{}"
time="2022-01-10T14:01:50Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:01:50Z" level=debug msg="scrapeRedisHost() done"
time="2022-01-10T14:01:50Z" level=debug msg="Trying DialURL(): rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:50Z" level=debug msg="connected to: rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:01:50Z" level=debug msg="connecting took 0.006006 seconds"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: CLIENT [SETNAME redis_exporter]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: CONFIG [GET *]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="Redis CONFIG err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: INFO [ALL]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="Redis INFO ALL result: [\"# Server\\r\\nredis_version:6.0.5\\r\\nredis_git_sha1:0\\r\\nredis_git_dirty:0\\r\\nredis_build_id:0\\r\\nredis_mode:cluster\\r\\nos:Amazon ElastiCache\\r\\narch_bits:64\\r\\nmultiplexing_api:epoll\\r\\natomicvar_api:atomic-builtin\\r\\ngcc_version:0.0.0\\r\\nprocess_id:1\\r\\nrun_id:55709d8ac32ffa5cdc0e46f1a64b995499526e1e\\r\\ntcp_port:6379\\r\\nuptime_in_seconds:10641052\\r\\nuptime_in_days:123\\r\\nhz:10\\r\\nconfigured_hz:10\\r\\nlru_clock:14433358\\r\\nexecutable:-\\r\\nconfig_file:-\\r\\n\\r\\n# Clients\\r\\nconnected_clients:44\\r\\nclient_recent_max_input_buffer:2\\r\\nclient_recent_max_output_buffer:0\\r\\nblocked_clients:0\\r\\ntracking_clients:0\\r\\nclients_in_timeout_table:0\\r\\n\\r\\n# Memory\\r\\nused_memory:7448496\\r\\nused_memory_human:7.10M\\r\\nused_memory_rss:12386304\\r\\nused_memory_rss_human:11.81M\\r\\nused_memory_peak:45704840\\r\\nused_memory_peak_human:43.59M\\r\\nused_memory_peak_perc:16.30%\\r\\nused_memory_overhead:6869718\\r\\nused_memory_startup:5075656\\r\\nused_memory_dataset:578778\\r\\nused_memory_dataset_perc:24.39%\\r\\nallocator_allocated:12687592\\r\\nallocator_active:14028800\\r\\nallocator_resident:17448960\\r\\nused_memory_lua:37888\\r\\nused_memory_lua_human:37.00K\\r\\nused_memory_scripts:0\\r\\nused_memory_scripts_human:0B\\r\\nnumber_of_cached_scripts:0\\r\\nmaxmemory:436469760\\r\\nmaxmemory_human:416.25M\\r\\nmaxmemory_policy:volatile-lru\\r\\nallocator_frag_ratio:1.11\\r\\nallocator_frag_bytes:1341208\\r\\nallocator_rss_ratio:1.24\\r\\nallocator_rss_bytes:3420160\\r\\nrss_overhead_ratio:0.71\\r\\nrss_overhead_bytes:-5062656\\r\\nmem_fragmentation_ratio:1.68\\r\\nmem_fragmentation_bytes:5001560\\r\\nmem_not_counted_for_evict:0\\r\\nmem_replication_backlog:1048576\\r\\nmem_clients_slaves:0\\r\\nmem_clients_normal:741406\\r\\nmem_aof_buffer:0\\r\\nmem_allocator:jemalloc-5.1.0\\r\\nactive_defrag_running:0\\r\\nlazyfree_pending_objects:0\\r\\n\\r\\n# Persistence\\r\\nloading:0\\r\\nrdb_changes_since_last_save:0\\r\\nrdb_bgsave_in_progress:0\\r\\nrdb_last_save_time:1641772915\\r\\nrdb_last_bgsave_status:ok\\r\\nrdb_last_bgsave_time_sec:0\\r\\nrdb_current_bgsave_time_sec:-1\\r\\nrdb_last_cow_size:712704\\r\\naof_enabled:0\\r\\naof_rewrite_in_progress:0\\r\\naof_rewrite_scheduled:0\\r\\naof_last_rewrite_time_sec:-1\\r\\naof_current_rewrite_time_sec:-1\\r\\naof_last_bgrewrite_status:ok\\r\\naof_last_write_status:ok\\r\\naof_last_cow_size:0\\r\\nmodule_fork_in_progress:0\\r\\nmodule_fork_last_cow_size:0\\r\\n\\r\\n# SSL\\r\\nssl_enabled:yes\\r\\nssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT\\r\\nssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT\\r\\nssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0\\r\\n\\r\\n# Stats\\r\\ntotal_connections_received:13903447\\r\\ntotal_commands_processed:27661864\\r\\ninstantaneous_ops_per_sec:0\\r\\ntotal_net_input_bytes:5279392187\\r\\ntotal_net_output_bytes:219021069558\\r\\ninstantaneous_input_kbps:0.32\\r\\ninstantaneous_output_kbps:20.53\\r\\nrejected_connections:0\\r\\nsync_full:0\\r\\nsync_partial_ok:0\\r\\nsync_partial_err:0\\r\\nexpired_keys:0\\r\\nexpired_stale_perc:0.00\\r\\nexpired_time_cap_reached_count:0\\r\\nexpire_cycle_cpu_milliseconds:0\\r\\nevicted_keys:0\\r\\nkeyspace_hits:0\\r\\nkeyspace_misses:0\\r\\npubsub_channels:0\\r\\npubsub_patterns:0\\r\\nlatest_fork_usec:479\\r\\nmigrate_cached_sockets:0\\r\\nslave_expires_tracked_keys:0\\r\\nactive_defrag_hits:0\\r\\nactive_defrag_misses:0\\r\\nactive_defrag_key_hits:0\\r\\nactive_defrag_key_misses:0\\r\\ntracking_total_keys:0\\r\\ntracking_total_items:0\\r\\ntracking_total_prefixes:0\\r\\nunexpected_error_replies:0\\r\\n\\r\\n# Replication\\r\\nrole:slave\\r\\nmaster_host:redis-hub-dev-0003-001.xxxxxx.cache.amazonaws.com\\r\\nmaster_port:6379\\r\\nmaster_link_status:up\\r\\nmaster_last_io_seconds_ago:0\\r\\nmaster_sync_in_progress:0\\r\\nslave_repl_offset:623093932\\r\\nrepl_sync_enabled:1\\r\\nslave_read_reploff:623093932\\r\\nslave_priority:100\\r\\nslave_read_only:1\\r\\nconnected_slaves:0\\r\\nmaster_replid:0596b5be88ffd82fa7a0aac823c15c815c3ff677\\r\\nmaster_replid2:0000000000000000000000000000000000000000\\r\\nmaster_repl_offset:623093932\\r\\nsecond_repl_offset:-1\\r\\nrepl_backlog_active:1\\r\\nrepl_backlog_size:1048576\\r\\nrepl_backlog_first_byte_offset:622045357\\r\\nrepl_backlog_histlen:1048576\\r\\n\\r\\n# CPU\\r\\nused_cpu_sys:17218.193885\\r\\nused_cpu_user:52890.662743\\r\\nused_cpu_sys_children:0.100367\\r\\nused_cpu_user_children:0.259001\\r\\n\\r\\n# Modules\\r\\n\\r\\n# Commandstats\\r\\ncmdstat_del:calls=2313,usec=22428,usec_per_call=9.70\\r\\ncmdstat_hello:calls=37,usec=486,usec_per_call=13.14\\r\\ncmdstat_config:calls=9869013,usec=209263741,usec_per_call=21.20\\r\\ncmdstat_info:calls=13183530,usec=5175285363,usec_per_call=392.56\\r\\ncmdstat_auth:calls=14264325,usec=106216289,usec_per_call=7.45\\r\\ncmdstat_ping:calls=40826039,usec=38016421,usec_per_call=0.93\\r\\ncmdstat_bgsave:calls=125,usec=87244,usec_per_call=697.95\\r\\ncmdstat_set:calls=13961,usec=106754,usec_per_call=7.65\\r\\ncmdstat_latency:calls=72,usec=154,usec_per_call=2.14\\r\\ncmdstat_client:calls=354789,usec=13899222,usec_per_call=39.18\\r\\ncmdstat_replconf:calls=10601795,usec=17881705,usec_per_call=1.69\\r\\ncmdstat_select:calls=1,usec=1,usec_per_call=1.00\\r\\ncmdstat_cluster:calls=4610,usec=470891,usec_per_call=102.15\\r\\ncmdstat_flushall:calls=1,usec=1246,usec_per_call=1246.00\\r\\ncmdstat_slowlog:calls=144,usec=888,usec_per_call=6.17\\r\\n\\r\\n# Cluster\\r\\ncluster_enabled:1\\r\\n\\r\\n# Keyspace\\r\\ndb0:keys=50,expires=0,avg_ttl=0\\r\\n\"]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: CLUSTER [INFO]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_state:ok"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_slots_assigned:16384"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_slots_ok:16384"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_slots_pfail:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_slots_fail:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_known_nodes:6"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_size:3"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_current_epoch:3"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_my_epoch:3"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_ping_sent:10604705"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_pong_sent:10126501"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_meet_sent:2"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_update_sent:182"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_sent:20731390"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_ping_received:10126501"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_pong_received:10610167"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_stats_messages_received:20736668"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="dbCount: 1"
time="2022-01-10T14:01:50Z" level=debug msg="info: # Server"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Server"
time="2022-01-10T14:01:50Z" level=debug msg="info: redis_version:6.0.5"
time="2022-01-10T14:01:50Z" level=debug msg="info: redis_git_sha1:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: redis_git_dirty:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: redis_build_id:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: redis_mode:cluster"
time="2022-01-10T14:01:50Z" level=debug msg="info: os:Amazon ElastiCache"
time="2022-01-10T14:01:50Z" level=debug msg="info: arch_bits:64"
time="2022-01-10T14:01:50Z" level=debug msg="info: multiplexing_api:epoll"
time="2022-01-10T14:01:50Z" level=debug msg="info: atomicvar_api:atomic-builtin"
time="2022-01-10T14:01:50Z" level=debug msg="info: gcc_version:0.0.0"
time="2022-01-10T14:01:50Z" level=debug msg="info: process_id:1"
time="2022-01-10T14:01:50Z" level=debug msg="info: run_id:55709d8ac32ffa5cdc0e46f1a64b995499526e1e"
time="2022-01-10T14:01:50Z" level=debug msg="info: tcp_port:6379"
time="2022-01-10T14:01:50Z" level=debug msg="info: uptime_in_seconds:10641052"
time="2022-01-10T14:01:50Z" level=debug msg="info: uptime_in_days:123"
time="2022-01-10T14:01:50Z" level=debug msg="info: hz:10"
time="2022-01-10T14:01:50Z" level=debug msg="info: configured_hz:10"
time="2022-01-10T14:01:50Z" level=debug msg="info: lru_clock:14433358"
time="2022-01-10T14:01:50Z" level=debug msg="info: executable:-"
time="2022-01-10T14:01:50Z" level=debug msg="info: config_file:-"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Clients"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Clients"
time="2022-01-10T14:01:50Z" level=debug msg="info: connected_clients:44"
time="2022-01-10T14:01:50Z" level=debug msg="info: client_recent_max_input_buffer:2"
time="2022-01-10T14:01:50Z" level=debug msg="info: client_recent_max_output_buffer:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: blocked_clients:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: tracking_clients:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: clients_in_timeout_table:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Memory"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Memory"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory:7448496"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_human:7.10M"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_rss:12386304"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_rss_human:11.81M"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_peak:45704840"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_peak_human:43.59M"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_peak_perc:16.30%"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_overhead:6869718"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_startup:5075656"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_dataset:578778"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_dataset_perc:24.39%"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_allocated:12687592"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_active:14028800"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_resident:17448960"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_lua:37888"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_lua_human:37.00K"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_scripts:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_memory_scripts_human:0B"
time="2022-01-10T14:01:50Z" level=debug msg="info: number_of_cached_scripts:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: maxmemory:436469760"
time="2022-01-10T14:01:50Z" level=debug msg="info: maxmemory_human:416.25M"
time="2022-01-10T14:01:50Z" level=debug msg="info: maxmemory_policy:volatile-lru"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_frag_ratio:1.11"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_frag_bytes:1341208"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_rss_ratio:1.24"
time="2022-01-10T14:01:50Z" level=debug msg="info: allocator_rss_bytes:3420160"
time="2022-01-10T14:01:50Z" level=debug msg="info: rss_overhead_ratio:0.71"
time="2022-01-10T14:01:50Z" level=debug msg="info: rss_overhead_bytes:-5062656"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_fragmentation_ratio:1.68"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_fragmentation_bytes:5001560"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_not_counted_for_evict:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_replication_backlog:1048576"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_clients_slaves:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_clients_normal:741406"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_aof_buffer:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: mem_allocator:jemalloc-5.1.0"
time="2022-01-10T14:01:50Z" level=debug msg="info: active_defrag_running:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: lazyfree_pending_objects:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Persistence"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Persistence"
time="2022-01-10T14:01:50Z" level=debug msg="info: loading:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_changes_since_last_save:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_bgsave_in_progress:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_last_save_time:1641772915"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_last_bgsave_status:ok"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_last_bgsave_time_sec:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_current_bgsave_time_sec:-1"
time="2022-01-10T14:01:50Z" level=debug msg="info: rdb_last_cow_size:712704"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_enabled:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_rewrite_in_progress:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_rewrite_scheduled:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_last_rewrite_time_sec:-1"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_current_rewrite_time_sec:-1"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_last_bgrewrite_status:ok"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_last_write_status:ok"
time="2022-01-10T14:01:50Z" level=debug msg="info: aof_last_cow_size:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: module_fork_in_progress:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: module_fork_last_cow_size:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # SSL"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: SSL"
time="2022-01-10T14:01:50Z" level=debug msg="info: ssl_enabled:yes"
time="2022-01-10T14:01:50Z" level=debug msg="info: ssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT"
time="2022-01-10T14:01:50Z" level=debug msg="info: ssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT"
time="2022-01-10T14:01:50Z" level=debug msg="info: ssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Stats"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Stats"
time="2022-01-10T14:01:50Z" level=debug msg="info: total_connections_received:13903447"
time="2022-01-10T14:01:50Z" level=debug msg="info: total_commands_processed:27661864"
time="2022-01-10T14:01:50Z" level=debug msg="info: instantaneous_ops_per_sec:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: total_net_input_bytes:5279392187"
time="2022-01-10T14:01:50Z" level=debug msg="info: total_net_output_bytes:219021069558"
time="2022-01-10T14:01:50Z" level=debug msg="info: instantaneous_input_kbps:0.32"
time="2022-01-10T14:01:50Z" level=debug msg="info: instantaneous_output_kbps:20.53"
time="2022-01-10T14:01:50Z" level=debug msg="info: rejected_connections:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: sync_full:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: sync_partial_ok:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: sync_partial_err:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: expired_keys:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: expired_stale_perc:0.00"
time="2022-01-10T14:01:50Z" level=debug msg="info: expired_time_cap_reached_count:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: expire_cycle_cpu_milliseconds:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: evicted_keys:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: keyspace_hits:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: keyspace_misses:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: pubsub_channels:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: pubsub_patterns:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: latest_fork_usec:479"
time="2022-01-10T14:01:50Z" level=debug msg="info: migrate_cached_sockets:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: slave_expires_tracked_keys:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: active_defrag_hits:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: active_defrag_misses:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: active_defrag_key_hits:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: active_defrag_key_misses:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: tracking_total_keys:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: tracking_total_items:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: tracking_total_prefixes:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: unexpected_error_replies:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Replication"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Replication"
time="2022-01-10T14:01:50Z" level=debug msg="info: role:slave"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_host:redis-hub-dev-0003-001.xxxxxx.cache.amazonaws.com"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_port:6379"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_link_status:up"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_last_io_seconds_ago:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_sync_in_progress:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: slave_repl_offset:623093932"
time="2022-01-10T14:01:50Z" level=debug msg="info: repl_sync_enabled:1"
time="2022-01-10T14:01:50Z" level=debug msg="info: slave_read_reploff:623093932"
time="2022-01-10T14:01:50Z" level=debug msg="info: slave_priority:100"
time="2022-01-10T14:01:50Z" level=debug msg="info: slave_read_only:1"
time="2022-01-10T14:01:50Z" level=debug msg="info: connected_slaves:0"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_replid:0596b5be88ffd82fa7a0aac823c15c815c3ff677"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_replid2:0000000000000000000000000000000000000000"
time="2022-01-10T14:01:50Z" level=debug msg="info: master_repl_offset:623093932"
time="2022-01-10T14:01:50Z" level=debug msg="info: second_repl_offset:-1"
time="2022-01-10T14:01:50Z" level=debug msg="info: repl_backlog_active:1"
time="2022-01-10T14:01:50Z" level=debug msg="info: repl_backlog_size:1048576"
time="2022-01-10T14:01:50Z" level=debug msg="info: repl_backlog_first_byte_offset:622045357"
time="2022-01-10T14:01:50Z" level=debug msg="info: repl_backlog_histlen:1048576"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # CPU"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: CPU"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_cpu_sys:17218.193885"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_cpu_user:52890.662743"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_cpu_sys_children:0.100367"
time="2022-01-10T14:01:50Z" level=debug msg="info: used_cpu_user_children:0.259001"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Modules"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Modules"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Commandstats"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Commandstats"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_del:calls=2313,usec=22428,usec_per_call=9.70"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_hello:calls=37,usec=486,usec_per_call=13.14"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_config:calls=9869013,usec=209263741,usec_per_call=21.20"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_info:calls=13183530,usec=5175285363,usec_per_call=392.56"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_auth:calls=14264325,usec=106216289,usec_per_call=7.45"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_ping:calls=40826039,usec=38016421,usec_per_call=0.93"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_bgsave:calls=125,usec=87244,usec_per_call=697.95"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_set:calls=13961,usec=106754,usec_per_call=7.65"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_latency:calls=72,usec=154,usec_per_call=2.14"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_client:calls=354789,usec=13899222,usec_per_call=39.18"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_replconf:calls=10601795,usec=17881705,usec_per_call=1.69"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_select:calls=1,usec=1,usec_per_call=1.00"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_cluster:calls=4610,usec=470891,usec_per_call=102.15"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_flushall:calls=1,usec=1246,usec_per_call=1246.00"
time="2022-01-10T14:01:50Z" level=debug msg="info: cmdstat_slowlog:calls=144,usec=888,usec_per_call=6.17"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Cluster"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Cluster"
time="2022-01-10T14:01:50Z" level=debug msg="info: cluster_enabled:1"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="info: # Keyspace"
time="2022-01-10T14:01:50Z" level=debug msg="set fieldClass: Keyspace"
time="2022-01-10T14:01:50Z" level=debug msg="info: db0:keys=50,expires=0,avg_ttl=0"
time="2022-01-10T14:01:50Z" level=debug msg="parseDBKeyspaceString inputKey: [db0] inputVal: [keys=50,expires=0,avg_ttl=0]"
time="2022-01-10T14:01:50Z" level=debug msg="info: "
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - running command: LATENCY [LATEST]"
time="2022-01-10T14:01:50Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:01:50Z" level=debug msg="Creating cluster object"
time="2022-01-10T14:01:50Z" level=debug msg="Running refresh on cluster object"
time="2022-01-10T14:02:05Z" level=error msg="Cluster refresh failed: redisc: all nodes failed\nread tcp 10.0.37.233:56468->10.0.35.94:6379: i/o timeout"
time="2022-01-10T14:02:05Z" level=debug msg="Creating redis connection object"
time="2022-01-10T14:02:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:02:05Z" level=debug msg="keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:02:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:02:05Z" level=debug msg="e.singleKeys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:02:05Z" level=debug msg="e.keys: []exporter.dbKeyPair(nil)"
time="2022-01-10T14:02:05Z" level=debug msg="allKeys: []exporter.dbKeyPair{}"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: SLOWLOG [LEN]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: SLOWLOG [GET 1]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:02:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:02:05Z" level=debug msg="allStreams: []exporter.dbKeyPair{}"
time="2022-01-10T14:02:05Z" level=debug msg="parseKeyArg(): Got empty key arguments, parsing skipped"
time="2022-01-10T14:02:05Z" level=debug msg="scrapeRedisHost() done"
time="2022-01-10T14:02:05Z" level=debug msg="Trying DialURL(): rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:02:05Z" level=debug msg="connected to: rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
time="2022-01-10T14:02:05Z" level=debug msg="connecting took 0.006787 seconds"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: CLIENT [SETNAME redis_exporter]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: CONFIG [GET *]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="Redis CONFIG err: ERR unknown command `CONFIG`, with args beginning with: `GET`, `*`, "
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: INFO [ALL]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="Redis INFO ALL result: [\"# Server\\r\\nredis_version:6.0.5\\r\\nredis_git_sha1:0\\r\\nredis_git_dirty:0\\r\\nredis_build_id:0\\r\\nredis_mode:cluster\\r\\nos:Amazon ElastiCache\\r\\narch_bits:64\\r\\nmultiplexing_api:epoll\\r\\natomicvar_api:atomic-builtin\\r\\ngcc_version:0.0.0\\r\\nprocess_id:1\\r\\nrun_id:a20787f1c26718fd8c3b27a0c7dc33fce98ed847\\r\\ntcp_port:6379\\r\\nuptime_in_seconds:10641682\\r\\nuptime_in_days:123\\r\\nhz:10\\r\\nconfigured_hz:10\\r\\nlru_clock:14433373\\r\\nexecutable:-\\r\\nconfig_file:-\\r\\n\\r\\n# Clients\\r\\nconnected_clients:43\\r\\nclient_recent_max_input_buffer:2\\r\\nclient_recent_max_output_buffer:0\\r\\nblocked_clients:0\\r\\ntracking_clients:0\\r\\nclients_in_timeout_table:0\\r\\n\\r\\n# Memory\\r\\nused_memory:7332408\\r\\nused_memory_human:6.99M\\r\\nused_memory_rss:12472320\\r\\nused_memory_rss_human:11.89M\\r\\nused_memory_peak:44823168\\r\\nused_memory_peak_human:42.75M\\r\\nused_memory_peak_perc:16.36%\\r\\nused_memory_overhead:6872326\\r\\nused_memory_startup:5075656\\r\\nused_memory_dataset:460082\\r\\nused_memory_dataset_perc:20.39%\\r\\nallocator_allocated:15722448\\r\\nallocator_active:17088512\\r\\nallocator_resident:22155264\\r\\nused_memory_lua:37888\\r\\nused_memory_lua_human:37.00K\\r\\nused_memory_scripts:0\\r\\nused_memory_scripts_human:0B\\r\\nnumber_of_cached_scripts:0\\r\\nmaxmemory:436469760\\r\\nmaxmemory_human:416.25M\\r\\nmaxmemory_policy:volatile-lru\\r\\nallocator_frag_ratio:1.09\\r\\nallocator_frag_bytes:1366064\\r\\nallocator_rss_ratio:1.30\\r\\nallocator_rss_bytes:5066752\\r\\nrss_overhead_ratio:0.56\\r\\nrss_overhead_bytes:-9682944\\r\\nmem_fragmentation_ratio:1.72\\r\\nmem_fragmentation_bytes:5203664\\r\\nmem_not_counted_for_evict:0\\r\\nmem_replication_backlog:1048576\\r\\nmem_clients_slaves:17242\\r\\nmem_clients_normal:724164\\r\\nmem_aof_buffer:0\\r\\nmem_allocator:jemalloc-5.1.0\\r\\nactive_defrag_running:0\\r\\nlazyfree_pending_objects:0\\r\\n\\r\\n# Persistence\\r\\nloading:0\\r\\nrdb_changes_since_last_save:14698\\r\\nrdb_bgsave_in_progress:0\\r\\nrdb_last_save_time:1631181643\\r\\nrdb_last_bgsave_status:ok\\r\\nrdb_last_bgsave_time_sec:-1\\r\\nrdb_current_bgsave_time_sec:-1\\r\\nrdb_last_cow_size:380928\\r\\naof_enabled:0\\r\\naof_rewrite_in_progress:0\\r\\naof_rewrite_scheduled:0\\r\\naof_last_rewrite_time_sec:-1\\r\\naof_current_rewrite_time_sec:-1\\r\\naof_last_bgrewrite_status:ok\\r\\naof_last_write_status:ok\\r\\naof_last_cow_size:0\\r\\nmodule_fork_in_progress:0\\r\\nmodule_fork_last_cow_size:0\\r\\n\\r\\n# SSL\\r\\nssl_enabled:yes\\r\\nssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT\\r\\nssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT\\r\\nssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0\\r\\n\\r\\n# Stats\\r\\ntotal_connections_received:13901512\\r\\ntotal_commands_processed:27745440\\r\\ninstantaneous_ops_per_sec:25\\r\\ntotal_net_input_bytes:5126801182\\r\\ntotal_net_output_bytes:231168148282\\r\\ninstantaneous_input_kbps:0.85\\r\\ninstantaneous_output_kbps:21.87\\r\\nrejected_connections:0\\r\\nsync_full:1\\r\\nsync_partial_ok:0\\r\\nsync_partial_err:1\\r\\nexpired_keys:2186\\r\\nexpired_stale_perc:0.00\\r\\nexpired_time_cap_reached_count:0\\r\\nexpire_cycle_cpu_milliseconds:156513\\r\\nevicted_keys:0\\r\\nkeyspace_hits:9047\\r\\nkeyspace_misses:17280\\r\\npubsub_channels:0\\r\\npubsub_patterns:0\\r\\nlatest_fork_usec:265\\r\\nmigrate_cached_sockets:0\\r\\nslave_expires_tracked_keys:0\\r\\nactive_defrag_hits:0\\r\\nactive_defrag_misses:0\\r\\nactive_defrag_key_hits:0\\r\\nactive_defrag_key_misses:0\\r\\ntracking_total_keys:0\\r\\ntracking_total_items:0\\r\\ntracking_total_prefixes:0\\r\\nunexpected_error_replies:0\\r\\n\\r\\n# Replication\\r\\nrole:master\\r\\nconnected_slaves:1\\r\\nslave0:ip=10.17.1.161,port=6379,state=online,offset=606359638,lag=0\\r\\nmaster_replid:f940006536f8a5a1baf08c889b69c7ec1370af55\\r\\nmaster_replid2:0000000000000000000000000000000000000000\\r\\nmaster_repl_offset:606359638\\r\\nsecond_repl_offset:-1\\r\\nrepl_backlog_active:1\\r\\nrepl_backlog_size:1048576\\r\\nrepl_backlog_first_byte_offset:605311063\\r\\nrepl_backlog_histlen:1048576\\r\\n\\r\\n# CPU\\r\\nused_cpu_sys:14378.554467\\r\\nused_cpu_user:55102.588953\\r\\nused_cpu_sys_children:0.000000\\r\\nused_cpu_user_children:0.001069\\r\\n\\r\\n# Modules\\r\\n\\r\\n# Commandstats\\r\\ncmdstat_slowlog:calls=162,usec=2447,usec_per_call=15.10\\r\\ncmdstat_config:calls=9832551,usec=201004759,usec_per_call=20.44\\r\\ncmdstat_exists:calls=1770,usec=7886,usec_per_call=4.46\\r\\ncmdstat_command:calls=1,usec=1215,usec_per_call=1215.00\\r\\ncmdstat_info:calls=13183935,usec=5099344640,usec_per_call=386.78\\r\\ncmdstat_latency:calls=82,usec=1253,usec_per_call=15.28\\r\\ncmdstat_get:calls=24557,usec=93931,usec_per_call=3.83\\r\\ncmdstat_cluster:calls=4380,usec=443042,usec_per_call=101.15\\r\\ncmdstat_replconf:calls=10603846,usec=16838538,usec_per_call=1.59\\r\\ncmdstat_set:calls=14698,usec=89843,usec_per_call=6.11\\r\\ncmdstat_auth:calls=14256921,usec=104291627,usec_per_call=7.32\\r\\ncmdstat_psync:calls=1,usec=589,usec_per_call=589.00\\r\\ncmdstat_ping:calls=39779161,usec=35132849,usec_per_call=0.88\\r\\ncmdstat_client:calls=354811,usec=11947663,usec_per_call=33.67\\r\\ncmdstat_hello:calls=22,usec=306,usec_per_call=13.91\\r\\n\\r\\n# Cluster\\r\\ncluster_enabled:1\\r\\n\\r\\n# Keyspace\\r\\ndb0:keys=64,expires=0,avg_ttl=0\\r\\n\"]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: CLUSTER [INFO]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_state:ok"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_slots_assigned:16384"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_slots_ok:16384"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_slots_pfail:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_slots_fail:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_known_nodes:6"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_size:3"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_current_epoch:3"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_my_epoch:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_ping_sent:10604008"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_pong_sent:10584118"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_update_sent:117"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_sent:21188243"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_ping_received:10584115"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_pong_received:10609469"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_meet_received:3"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_update_received:263"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_stats_messages_received:21193850"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="dbCount: 1"
time="2022-01-10T14:02:05Z" level=debug msg="info: # Server"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Server"
time="2022-01-10T14:02:05Z" level=debug msg="info: redis_version:6.0.5"
time="2022-01-10T14:02:05Z" level=debug msg="info: redis_git_sha1:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: redis_git_dirty:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: redis_build_id:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: redis_mode:cluster"
time="2022-01-10T14:02:05Z" level=debug msg="info: os:Amazon ElastiCache"
time="2022-01-10T14:02:05Z" level=debug msg="info: arch_bits:64"
time="2022-01-10T14:02:05Z" level=debug msg="info: multiplexing_api:epoll"
time="2022-01-10T14:02:05Z" level=debug msg="info: atomicvar_api:atomic-builtin"
time="2022-01-10T14:02:05Z" level=debug msg="info: gcc_version:0.0.0"
time="2022-01-10T14:02:05Z" level=debug msg="info: process_id:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: run_id:a20787f1c26718fd8c3b27a0c7dc33fce98ed847"
time="2022-01-10T14:02:05Z" level=debug msg="info: tcp_port:6379"
time="2022-01-10T14:02:05Z" level=debug msg="info: uptime_in_seconds:10641682"
time="2022-01-10T14:02:05Z" level=debug msg="info: uptime_in_days:123"
time="2022-01-10T14:02:05Z" level=debug msg="info: hz:10"
time="2022-01-10T14:02:05Z" level=debug msg="info: configured_hz:10"
time="2022-01-10T14:02:05Z" level=debug msg="info: lru_clock:14433373"
time="2022-01-10T14:02:05Z" level=debug msg="info: executable:-"
time="2022-01-10T14:02:05Z" level=debug msg="info: config_file:-"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Clients"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Clients"
time="2022-01-10T14:02:05Z" level=debug msg="info: connected_clients:43"
time="2022-01-10T14:02:05Z" level=debug msg="info: client_recent_max_input_buffer:2"
time="2022-01-10T14:02:05Z" level=debug msg="info: client_recent_max_output_buffer:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: blocked_clients:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: tracking_clients:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: clients_in_timeout_table:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Memory"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Memory"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory:7332408"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_human:6.99M"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_rss:12472320"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_rss_human:11.89M"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_peak:44823168"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_peak_human:42.75M"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_peak_perc:16.36%"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_overhead:6872326"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_startup:5075656"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_dataset:460082"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_dataset_perc:20.39%"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_allocated:15722448"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_active:17088512"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_resident:22155264"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_lua:37888"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_lua_human:37.00K"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_scripts:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_memory_scripts_human:0B"
time="2022-01-10T14:02:05Z" level=debug msg="info: number_of_cached_scripts:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: maxmemory:436469760"
time="2022-01-10T14:02:05Z" level=debug msg="info: maxmemory_human:416.25M"
time="2022-01-10T14:02:05Z" level=debug msg="info: maxmemory_policy:volatile-lru"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_frag_ratio:1.09"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_frag_bytes:1366064"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_rss_ratio:1.30"
time="2022-01-10T14:02:05Z" level=debug msg="info: allocator_rss_bytes:5066752"
time="2022-01-10T14:02:05Z" level=debug msg="info: rss_overhead_ratio:0.56"
time="2022-01-10T14:02:05Z" level=debug msg="info: rss_overhead_bytes:-9682944"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_fragmentation_ratio:1.72"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_fragmentation_bytes:5203664"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_not_counted_for_evict:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_replication_backlog:1048576"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_clients_slaves:17242"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_clients_normal:724164"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_aof_buffer:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: mem_allocator:jemalloc-5.1.0"
time="2022-01-10T14:02:05Z" level=debug msg="info: active_defrag_running:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: lazyfree_pending_objects:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Persistence"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Persistence"
time="2022-01-10T14:02:05Z" level=debug msg="info: loading:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_changes_since_last_save:14698"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_bgsave_in_progress:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_last_save_time:1631181643"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_last_bgsave_status:ok"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_last_bgsave_time_sec:-1"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_current_bgsave_time_sec:-1"
time="2022-01-10T14:02:05Z" level=debug msg="info: rdb_last_cow_size:380928"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_enabled:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_rewrite_in_progress:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_rewrite_scheduled:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_last_rewrite_time_sec:-1"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_current_rewrite_time_sec:-1"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_last_bgrewrite_status:ok"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_last_write_status:ok"
time="2022-01-10T14:02:05Z" level=debug msg="info: aof_last_cow_size:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: module_fork_in_progress:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: module_fork_last_cow_size:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # SSL"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: SSL"
time="2022-01-10T14:02:05Z" level=debug msg="info: ssl_enabled:yes"
time="2022-01-10T14:02:05Z" level=debug msg="info: ssl_current_certificate_not_before_date:Sep 9 00:00:00 2021 GMT"
time="2022-01-10T14:02:05Z" level=debug msg="info: ssl_current_certificate_not_after_date:Oct 8 23:59:59 2022 GMT"
time="2022-01-10T14:02:05Z" level=debug msg="info: ssl_current_certificate_serial:03F73A2DF7378D026B732E33CBE46BB0"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Stats"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Stats"
time="2022-01-10T14:02:05Z" level=debug msg="info: total_connections_received:13901512"
time="2022-01-10T14:02:05Z" level=debug msg="info: total_commands_processed:27745440"
time="2022-01-10T14:02:05Z" level=debug msg="info: instantaneous_ops_per_sec:25"
time="2022-01-10T14:02:05Z" level=debug msg="info: total_net_input_bytes:5126801182"
time="2022-01-10T14:02:05Z" level=debug msg="info: total_net_output_bytes:231168148282"
time="2022-01-10T14:02:05Z" level=debug msg="info: instantaneous_input_kbps:0.85"
time="2022-01-10T14:02:05Z" level=debug msg="info: instantaneous_output_kbps:21.87"
time="2022-01-10T14:02:05Z" level=debug msg="info: rejected_connections:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: sync_full:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: sync_partial_ok:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: sync_partial_err:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: expired_keys:2186"
time="2022-01-10T14:02:05Z" level=debug msg="info: expired_stale_perc:0.00"
time="2022-01-10T14:02:05Z" level=debug msg="info: expired_time_cap_reached_count:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: expire_cycle_cpu_milliseconds:156513"
time="2022-01-10T14:02:05Z" level=debug msg="info: evicted_keys:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: keyspace_hits:9047"
time="2022-01-10T14:02:05Z" level=debug msg="info: keyspace_misses:17280"
time="2022-01-10T14:02:05Z" level=debug msg="info: pubsub_channels:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: pubsub_patterns:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: latest_fork_usec:265"
time="2022-01-10T14:02:05Z" level=debug msg="info: migrate_cached_sockets:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: slave_expires_tracked_keys:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: active_defrag_hits:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: active_defrag_misses:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: active_defrag_key_hits:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: active_defrag_key_misses:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: tracking_total_keys:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: tracking_total_items:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: tracking_total_prefixes:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: unexpected_error_replies:0"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Replication"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Replication"
time="2022-01-10T14:02:05Z" level=debug msg="info: role:master"
time="2022-01-10T14:02:05Z" level=debug msg="info: connected_slaves:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: slave0:ip=10.17.1.161,port=6379,state=online,offset=606359638,lag=0"
time="2022-01-10T14:02:05Z" level=debug msg="info: master_replid:f940006536f8a5a1baf08c889b69c7ec1370af55"
time="2022-01-10T14:02:05Z" level=debug msg="info: master_replid2:0000000000000000000000000000000000000000"
time="2022-01-10T14:02:05Z" level=debug msg="info: master_repl_offset:606359638"
time="2022-01-10T14:02:05Z" level=debug msg="info: second_repl_offset:-1"
time="2022-01-10T14:02:05Z" level=debug msg="info: repl_backlog_active:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: repl_backlog_size:1048576"
time="2022-01-10T14:02:05Z" level=debug msg="info: repl_backlog_first_byte_offset:605311063"
time="2022-01-10T14:02:05Z" level=debug msg="info: repl_backlog_histlen:1048576"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # CPU"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: CPU"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_cpu_sys:14378.554467"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_cpu_user:55102.588953"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_cpu_sys_children:0.000000"
time="2022-01-10T14:02:05Z" level=debug msg="info: used_cpu_user_children:0.001069"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Modules"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Modules"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Commandstats"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Commandstats"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_slowlog:calls=162,usec=2447,usec_per_call=15.10"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_config:calls=9832551,usec=201004759,usec_per_call=20.44"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_exists:calls=1770,usec=7886,usec_per_call=4.46"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_command:calls=1,usec=1215,usec_per_call=1215.00"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_info:calls=13183935,usec=5099344640,usec_per_call=386.78"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_latency:calls=82,usec=1253,usec_per_call=15.28"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_get:calls=24557,usec=93931,usec_per_call=3.83"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_cluster:calls=4380,usec=443042,usec_per_call=101.15"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_replconf:calls=10603846,usec=16838538,usec_per_call=1.59"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_set:calls=14698,usec=89843,usec_per_call=6.11"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_auth:calls=14256921,usec=104291627,usec_per_call=7.32"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_psync:calls=1,usec=589,usec_per_call=589.00"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_ping:calls=39779161,usec=35132849,usec_per_call=0.88"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_client:calls=354811,usec=11947663,usec_per_call=33.67"
time="2022-01-10T14:02:05Z" level=debug msg="info: cmdstat_hello:calls=22,usec=306,usec_per_call=13.91"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Cluster"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Cluster"
time="2022-01-10T14:02:05Z" level=debug msg="info: cluster_enabled:1"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="info: # Keyspace"
time="2022-01-10T14:02:05Z" level=debug msg="set fieldClass: Keyspace"
time="2022-01-10T14:02:05Z" level=debug msg="info: db0:keys=64,expires=0,avg_ttl=0"
time="2022-01-10T14:02:05Z" level=debug msg="parseDBKeyspaceString inputKey: [db0] inputVal: [keys=64,expires=0,avg_ttl=0]"
time="2022-01-10T14:02:05Z" level=debug msg="info: "
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - running command: LATENCY [LATEST]"
time="2022-01-10T14:02:05Z" level=debug msg="c.Do() - done"
time="2022-01-10T14:02:05Z" level=debug msg="Creating cluster object"
time="2022-01-10T14:02:05Z" level=debug msg="Running refresh on cluster object"
### curl returns :
; curl localhost:8080/metrics -o /dev/null
; % Total % Received % Xferd Average Speed Time Time Time Current
; Dload Upload Total Spent Left Speed
; 100 26430 0 26430 0 0 414 0 --:--:-- 0:01:03 --:--:-- 5916
SD shouldn't in the exporter but be handled by the Prometheus server (plus additional plumbing if necessary)
Imo that's true only in context of getting initial endpoint addresses, especially on application startup. Please remember that SD is refreshed periodically, usually tens of seconds while application itself should contact cluster almost every. That's why application (cluster client) is first to know when things go bad before end user get's his bad experience. Please consider existence of legacy systems without SD and yet , they are resilient.
BR
time="2022-01-10T14:01:35Z" level=debug msg="Trying DialURL(): rediss://clustercfg.xxxxxx.cache.amazonaws.com:6379"
...
time="2022-01-10T14:01:35Z" level=debug msg="Running refresh on cluster object"
time="2022-01-10T14:01:50Z" level=error msg="Cluster refresh failed: redisc: all nodes failed\nread tcp 10.0.37.233:52814->10.0.53.93:6379: i/o timeout"
...
time="2022-01-10T14:01:50Z" level=debug msg="scrapeRedisHost() done"
Looks reasonable - 15s for the scrape which is entirely due to the timeout.
I hear you about the SD but this can (and should) all be done outside the exporter.
I hear you about the SD but this can (and should) all be done outside the exporter.
I respect your opinion yet i cannot fully agree.
My perspective is following:
Looks reasonable - 15s for the scrape which is entirely due to the timeout.
I't would be great to hear any suggestions that may be helpful with this bug but now i am not sure if you consider this as a bug or not.
My perspective is following:
...
I hear you but I don't know what Elasticache does under the hood. One thing you can observe is that the run_id is different from scrape to scrape so looks like you're getting a different node? also, keep in mind that the exporter opens a new connection for every scrape, your app might not do that and keep a persistent connection.
I't would be great to hear any suggestions that may be helpful with this bug but now i am not sure if you consider this as a bug or not.
Still not sure if this is a bug or not but you can run with REDIS_EXPORTER_IS_CLUSTER=false
and see if that helps. Esp if you don't use check-keys feature.
Hi,
I see that you closed the issue. Well, i have just managed to install redis with REDIS_EXPORTER_IS_CLUSTER=false
. Running against the same cluster i see one big improvement - exporter is responding instantly. Looking into response i still see data being returned in round-robin manner. Each time i do curl i get data from different Redis cluster node.
What do you think about it @oliver006 ?
Regards,
Cluster monitoring questions and possible issues
Hi. I am running Redis 6 clusters on AWS with tls/ssl enabled. First of all i'd like to make sure that my assumptions are right. As for now i am trying to configure a single redis_exporter process per redis cluster, therefore i expect that exporter should be able to auto detect the cluster in a way as regular client does providing metrics with appropriate cardinality as well. That also makes sense if one concerns features that allow key count to be reported.
What version of redis_exporter are you running? v1.33.0-alpine
Running the exporter K8s deployment, default command line from vanilla dockerfile with following envs:
What i see is that exporter is somehow iterating all over 3 cluster shards (note db0 key count):
Some node metrics are also reported in log file f.e.:
There are some errors however:
Are those errors cricital? I am sure that this connection timeout is not caused by real network problems. There is a micorservice running on the same cluster and node that uses very same redis cluster. Logs and tracing shows that Redis is in use and i can cause new keys to be created. Connection and credentials should be fine since we get some metrics.
There is also another question. Redis_exporter is exceptionally slow.
curl localhost:8080/metrics
takes more or less 60 seconds to complete. Once again, looks like cluster was correctly auto detected:But on the other side i can't get metrics for all shards at once. Each time i do curl i get values for a single shard:
Expected behavior It would be nice if anyone can answer my usage pattern questions first. If this exporter is considered as cluster-wide exporter then maybe this issue can be considered as a bug. In such case exporter should be a source of coherent full-cluster-state information each time we call GET /metrics.
Kind Regards Bart