search

oliver006 / redis_exporter

Prometheus Exporter for ValKey & Redis Metrics. Supports ValKey and Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x
https://github.com/oliver006/redis_exporter
MIT License
3.04k stars 860 forks source link

Wrong hit/miss value on Elasticache 7.0.7 #816

Closed gillg closed 11 months ago

gillg commented 12 months ago

Describe the problem When I compare INFO Stats keyspace_* with the exporter value I see a problem. The exporter always displays "0" where INFO is displaying:

keyspace_hits:232692
keyspace_misses:137713

What version of redis_exporter are you running? Please run redis_exporter --version if you're not sure what version you're running.

redis_instance_info{maxmemory_policy="volatile-lru",os="Amazon ElastiCache",process_id="1",redis_build_id="0",redis_mode="standalone",redis_version="7.0.7",role="slave",run_id="467828b573b92ae61bac60356d0548c9a66c4d3a",tcp_port="6379"} 1

Running the exporter REDIS_ADDR=redis://xxxxxxxxxx-ro.xxxxxx.ng.0001.xxxxx.cache.amazonaws.com:6379

Screenshots

# HELP redis_keyspace_hits_total keyspace_hits_total metric
# TYPE redis_keyspace_hits_total counter
redis_keyspace_hits_total 0
# HELP redis_keyspace_misses_total keyspace_misses_total metric
# TYPE redis_keyspace_misses_total counter
redis_keyspace_misses_total 0
INFO Stats
# Stats
total_connections_received:32
total_commands_processed:560844
instantaneous_ops_per_sec:0
total_net_input_bytes:68835819
total_net_output_bytes:46645299
total_net_repl_input_bytes:7443712
total_net_repl_output_bytes:7702410
instantaneous_input_kbps:0.08
instantaneous_output_kbps:11.17
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.03
rejected_connections:0
sync_full:3
sync_partial_ok:0
sync_partial_err:3
expired_keys:6
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:21
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:232692
keyspace_misses:137713
pubsub_channels:1
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:624
total_forks:1
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:52
dump_payload_sanitizations:0
total_reads_processed:260421
total_writes_processed:279234
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:505
reply_buffer_expands:941
gillg commented 12 months ago

By wrinting that issue... I'm wondering if it's not because I point the exporter to a readonly endpoint... It's still strange because the other metrics vary when my application hit the RW endpoint, so it's not really consistent in that case.

oliver006 commented 12 months ago

Can you run with the debug flag enabled to see what the difference might be?

gillg commented 12 months ago

Ok, I added the debug flag and the data get by "info" on the readonly endpoints is different. I manualy made some checks, and in fact the RO endpoint in pointing to the slave replica. It's not pointing to the master as read only... It's a little bit confusing but that explains why hit & misses are wrong.

But... if redis expose shared data like the number of commands and their type on the replica, I don't understand why it's not consistent with the hit/miss values. Do you think the correct approach in that case is to monitor the RW endpoint or the two endpoints ?

oliver006 commented 12 months ago

Yeah, monitor the RW instance. Or all of them, but separately (e.g. don't scrape a round-robin load-balanced endpoint)