search

Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.02k stars 564 forks source link

[CRASH] ckeysExpired == db->expireSize() #743

Open rmd4 opened 7 months ago

rmd4 commented 7 months ago

Crash report

407496:397491:C 19 Nov 2023 11:01:19.885 * RDB: 4 MB of memory used by copy-on-write
397462:397491:M 19 Nov 2023 11:01:19.992 * Background saving terminated with success
397462:397491:M 19 Nov 2023 11:11:59.701 * 10 changes in 300 seconds. Saving...
397462:397491:M 19 Nov 2023 11:11:59.773 * Background saving started by pid 407927
397462:397491:M 19 Nov 2023 11:11:59.773 * Background saving started
407927:397491:C 19 Nov 2023 11:12:13.422 * DB saved on disk
407927:397491:C 19 Nov 2023 11:12:13.476 * RDB: 6 MB of memory used by copy-on-write
397462:397491:M 19 Nov 2023 11:12:13.626 * Background saving terminated with success
397462:397491:M 19 Nov 2023 11:17:14.080 * 10 changes in 300 seconds. Saving...
397462:397491:M 19 Nov 2023 11:17:14.166 * Background saving started by pid 408145
397462:397491:M 19 Nov 2023 11:17:14.166 * Background saving started

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
408145:397491:C 19 Nov 2023 11:17:28.270 # === ASSERTION FAILED ===
408145:397491:C 19 Nov 2023 11:17:28.270 # ==> rdb.cpp:1372 'ckeysExpired == db->expireSize()' is not true

------ STACK TRACE ------

Backtrace:
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x58e) [0x5569ce3f740e]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x115) [0x5569ce3f85d5]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x73) [0x5569ce3f8963]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackgroundFork(rdbSaveInfo*)+0x122) [0x5569ce3f91d2]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackground(rdbSaveInfo*)+0xd8) [0x5569ce3f9888]
keydb-rdb-bgsave 192.168.254.52:6379(serverCron(aeEventLoop*, long long, void*)+0xe14) [0x5569ce39c854]
keydb-rdb-bgsave 192.168.254.52:6379(aeProcessEvents+0x224) [0x5569ce3934a4]
keydb-rdb-bgsave 192.168.254.52:6379(aeMain+0x3e) [0x5569ce393a6e]
keydb-rdb-bgsave 192.168.254.52:6379(workerThreadMain(void*)+0xfe) [0x5569ce3abe9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f24b3125ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a40) [0x7f24b31b7a40]

------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b532cd0401cb0da4
redis_mode:standalone
os:Linux 5.15.0-88-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:408145
process_supervised:no
run_id:2ee93088a878f55d0bdc926eb51adf235d1d7439
tcp_port:6379
server_time_usec:1700392634080800
uptime_in_seconds:16315
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5893818
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb-6379.conf
availability_zone:
features:cluster_mget

# Clients
connected_clients:55
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:20
thread_1_clients:21
thread_2_clients:14
thread_3_clients:0

# Memory
used_memory:2211691160
used_memory_human:2.06G
used_memory_rss:2272120832
used_memory_rss_human:2.12G
used_memory_peak:14828083264
used_memory_peak_human:13.81G
used_memory_peak_perc:14.92%
used_memory_overhead:28263272
used_memory_startup:3287000
used_memory_dataset:2183427888
used_memory_dataset_perc:98.87%
allocator_allocated:2213289064
allocator_active:2225258496
allocator_resident:2321199104
total_system_memory:134980608000
total_system_memory_human:125.71G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:51539607552
maxmemory_human:48.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.01
allocator_frag_bytes:11969432
allocator_rss_ratio:1.04
allocator_rss_bytes:95940608
rss_overhead_ratio:0.98
rss_overhead_bytes:-49078272
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:60431456
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1127776
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:9193
rdb_bgsave_in_progress:0
rdb_last_save_time:1700392333
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:14
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:6316032
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:88
total_commands_processed:9319761
instantaneous_ops_per_sec:0
total_net_input_bytes:106185379481
total_net_output_bytes:798164883
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:752200
expired_stale_perc:0.00
expired_time_cap_reached_count:5
expire_cycle_cpu_milliseconds:88799
evicted_keys:0
keyspace_hits:11211
keyspace_misses:2444
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:72330
total_forks:21
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:18951592
total_writes_processed:9319802
instantaneous_lock_contention:1
avg_lock_contention:0.031250
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:bfd56f64a0d362477baa6f6689c00df36bad0667
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:1.409685
used_cpu_user:12.014361
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:108
used_cpu_sys_main_thread:1.409685
used_cpu_user_main_thread:12.014361

# Modules

# Commandstats
cmdstat_get:calls=13655,usec=193595,usec_per_call=14.18,rejected_calls=0,failed_calls=0
cmdstat_del:calls=2880411,usec=13713807,usec_per_call=4.76,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=6,usec=3,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=2886,usec=31633,usec_per_call=10.96,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=88,usec=281,usec_per_call=3.19,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=2,usec=3,usec_per_call=1.50,rejected_calls=0,failed_calls=0
cmdstat_set:calls=3211387,usec=8627542,usec_per_call=2.69,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3211326,usec=5992111,usec_per_call=1.87,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=322081,expires=2343,avg_ttl=2386193452,cached_keys=322081

# KeyDB
mvcc_depth:0

------ CLIENT LIST OUTPUT ------
id=48 addr=192.168.254.53:33122 laddr=192.168.254.52:6379 fd=37 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=74 addr=192.168.254.52:48314 laddr=192.168.254.52:6379 fd=63 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=75 addr=192.168.254.53:33248 laddr=192.168.254.52:6379 fd=64 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=76 addr=192.168.254.52:48390 laddr=192.168.254.52:6379 fd=69 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=49 addr=192.168.254.53:33134 laddr=192.168.254.52:6379 fd=38 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=50 addr=192.168.254.52:48262 laddr=192.168.254.52:6379 fd=40 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=51 addr=192.168.254.52:48260 laddr=192.168.254.52:6379 fd=39 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=72 addr=192.168.254.53:33246 laddr=192.168.254.52:6379 fd=61 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=73 addr=192.168.254.53:33234 laddr=192.168.254.52:6379 fd=62 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=52 addr=192.168.254.52:48266 laddr=192.168.254.52:6379 fd=41 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=53 addr=192.168.254.53:33144 laddr=192.168.254.52:6379 fd=43 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=54 addr=192.168.254.53:33150 laddr=192.168.254.52:6379 fd=42 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=55 addr=192.168.254.52:48274 laddr=192.168.254.52:6379 fd=44 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=71 addr=192.168.254.53:33228 laddr=192.168.254.52:6379 fd=60 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=56 addr=192.168.254.52:48278 laddr=192.168.254.52:6379 fd=45 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=57 addr=192.168.254.52:48276 laddr=192.168.254.52:6379 fd=46 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=58 addr=192.168.254.53:33158 laddr=192.168.254.52:6379 fd=47 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=59 addr=192.168.254.53:33166 laddr=192.168.254.52:6379 fd=48 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=92 addr=192.168.254.53:60818 laddr=192.168.254.52:6379 fd=81 name= age=5082 idle=315 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=60 addr=192.168.254.52:48280 laddr=192.168.254.52:6379 fd=49 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=61 addr=192.168.254.52:48286 laddr=192.168.254.52:6379 fd=50 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=62 addr=192.168.254.52:48298 laddr=192.168.254.52:6379 fd=51 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=63 addr=192.168.254.53:33172 laddr=192.168.254.52:6379 fd=53 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=64 addr=192.168.254.52:48306 laddr=192.168.254.52:6379 fd=54 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=65 addr=192.168.254.53:33174 laddr=192.168.254.52:6379 fd=52 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=79 addr=192.168.254.52:48406 laddr=192.168.254.52:6379 fd=70 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=80 addr=192.168.254.52:48356 laddr=192.168.254.52:6379 fd=71 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=81 addr=192.168.254.52:48368 laddr=192.168.254.52:6379 fd=72 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=82 addr=192.168.254.52:48412 laddr=192.168.254.52:6379 fd=73 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=83 addr=192.168.254.53:33294 laddr=192.168.254.52:6379 fd=74 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=84 addr=192.168.254.52:48378 laddr=192.168.254.52:6379 fd=75 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=85 addr=192.168.254.53:33256 laddr=192.168.254.52:6379 fd=76 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=86 addr=192.168.254.53:33272 laddr=192.168.254.52:6379 fd=77 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=87 addr=192.168.254.53:33284 laddr=192.168.254.52:6379 fd=78 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=88 addr=192.168.254.52:48336 laddr=192.168.254.52:6379 fd=65 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=89 addr=192.168.254.52:48320 laddr=192.168.254.52:6379 fd=66 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=90 addr=192.168.254.52:48366 laddr=192.168.254.52:6379 fd=67 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=91 addr=192.168.254.52:48346 laddr=192.168.254.52:6379 fd=68 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=66 addr=192.168.254.53:33186 laddr=192.168.254.52:6379 fd=55 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=67 addr=192.168.254.53:33196 laddr=192.168.254.52:6379 fd=56 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=68 addr=192.168.254.53:33206 laddr=192.168.254.52:6379 fd=57 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=77 addr=192.168.254.52:48404 laddr=192.168.254.52:6379 fd=79 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=78 addr=192.168.254.53:33298 laddr=192.168.254.52:6379 fd=80 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=69 addr=192.168.254.53:33212 laddr=192.168.254.52:6379 fd=58 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=70 addr=192.168.254.53:33214 laddr=192.168.254.52:6379 fd=59 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=39 addr=192.168.254.53:38410 laddr=192.168.254.52:6379 fd=28 name= age=9493 idle=968 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=40 addr=192.168.254.53:38424 laddr=192.168.254.52:6379 fd=29 name= age=9493 idle=223 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=41 addr=192.168.254.53:38430 laddr=192.168.254.52:6379 fd=32 name= age=9492 idle=142 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=42 addr=192.168.254.53:47514 laddr=192.168.254.52:6379 fd=30 name= age=8569 idle=66 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=93 addr=192.168.254.53:60450 laddr=192.168.254.52:6379 fd=82 name= age=848 idle=848 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=43 addr=192.168.254.52:55578 laddr=192.168.254.52:6379 fd=34 name= age=6259 idle=6259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=44 addr=192.168.254.53:33090 laddr=192.168.254.52:6379 fd=31 name= age=6173 idle=141 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=45 addr=192.168.254.52:48246 laddr=192.168.254.52:6379 fd=33 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=46 addr=192.168.254.53:33106 laddr=192.168.254.52:6379 fd=35 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=47 addr=192.168.254.52:48254 laddr=192.168.254.52:6379 fd=36 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
408145:397491:C 19 Nov 2023 11:17:28.272 # Bio thread for job type #0 terminated
408145:397491:C 19 Nov 2023 11:17:28.272 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
408145:397491:C 19 Nov 2023 11:17:28.272 # Accessing address: 0x7f24b0e95910
408145:397491:C 19 Nov 2023 11:17:28.272 # Crashed running the instruction at: 0x7f24b3123bdd

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(pthread_cancel+0x1d) [0x7f24b3123bdd]

Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f24b30d3520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cancel+0x1d) [0x7f24b3123bdd]
keydb-rdb-bgsave 192.168.254.52:6379(bioKillThreads+0x65) [0x5569ce45c825]
keydb-rdb-bgsave 192.168.254.52:6379(doFastMemoryTest()+0x3f) [0x5569ce43860f]
keydb-rdb-bgsave 192.168.254.52:6379(_serverAssert+0xc6) [0x5569ce438746]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x58e) [0x5569ce3f740e]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x115) [0x5569ce3f85d5]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x73) [0x5569ce3f8963]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackgroundFork(rdbSaveInfo*)+0x122) [0x5569ce3f91d2]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackground(rdbSaveInfo*)+0xd8) [0x5569ce3f9888]
keydb-rdb-bgsave 192.168.254.52:6379(serverCron(aeEventLoop*, long long, void*)+0xe14) [0x5569ce39c854]
keydb-rdb-bgsave 192.168.254.52:6379(aeProcessEvents+0x224) [0x5569ce3934a4]
keydb-rdb-bgsave 192.168.254.52:6379(aeMain+0x3e) [0x5569ce393a6e]
keydb-rdb-bgsave 192.168.254.52:6379(workerThreadMain(void*)+0xfe) [0x5569ce3abe9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f24b3125ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a40) [0x7f24b31b7a40]

------ REGISTERS ------
408145:397491:C 19 Nov 2023 11:17:28.273 #
RAX:b4973fcc4d1b9400 RBX:0000000000000001
RCX:0000000000000002 RDX:ffffffffffffe308
RDI:00007f24b0e95640 RSI:00007f242c9fcb10
RBP:00007f242c9fe640 RSP:00007f242c9fa3b0
R8 :00005569cec93c50 R9 :0000000000000200
R10:00007f24b2200900 R11:0000000000000010
R12:00005569cea88a30 R13:00005569ce921bb0
R14:00005569ce921be8 R15:000000000004ea21
RIP:00007f24b3123bdd EFL:0000000000010202
CSGSFS:002b000000000033
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bf) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3be) -> 00007f24ad9ffc60
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bd) -> b4973fcc4d1b9400
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bc) -> 00007f242a20508b
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bb) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3ba) -> 0000000600000007
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b9) -> 00007f24b1696c68
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b8) -> 00007f24b31274d0
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b7) -> 00007f242c9fa3f0
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b6) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b5) -> 00007f24b312759c
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b4) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b3) -> 00007f24b1696640
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b2) -> 00007f24b3123b30
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b1) -> 00000000000001c0
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b0) -> 00005569cec93c50

------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b532cd0401cb0da4
redis_mode:standalone
os:Linux 5.15.0-88-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:408145
process_supervised:no
run_id:2ee93088a878f55d0bdc926eb51adf235d1d7439
tcp_port:6379
server_time_usec:1700392634080800
uptime_in_seconds:16315
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5893818
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb-6379.conf
availability_zone:
features:cluster_mget

# Clients
connected_clients:55
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:20
thread_1_clients:21
thread_2_clients:14
thread_3_clients:0

# Memory
used_memory:2211693800
used_memory_human:2.06G
used_memory_rss:2272120832
used_memory_rss_human:2.12G
used_memory_peak:14828083264
used_memory_peak_human:13.81G
used_memory_peak_perc:14.92%
used_memory_overhead:28263272
used_memory_startup:3287000
used_memory_dataset:2183430528
used_memory_dataset_perc:98.87%
allocator_allocated:2213289064
allocator_active:2225258496
allocator_resident:2321199104
total_system_memory:134980608000
total_system_memory_human:125.71G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:51539607552
maxmemory_human:48.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.01
allocator_frag_bytes:11969432
allocator_rss_ratio:1.04
allocator_rss_bytes:95940608
rss_overhead_ratio:0.98
rss_overhead_bytes:-49078272
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:60431456
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1127776
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:9193
rdb_bgsave_in_progress:0
rdb_last_save_time:1700392333
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:14
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:6316032
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:88
total_commands_processed:9319761
instantaneous_ops_per_sec:0
total_net_input_bytes:106185379481
total_net_output_bytes:798164883
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:752200
expired_stale_perc:0.00
expired_time_cap_reached_count:5
expire_cycle_cpu_milliseconds:88799
evicted_keys:0
keyspace_hits:11211
keyspace_misses:2444
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:72330
total_forks:21
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:18951592
total_writes_processed:9319802
instantaneous_lock_contention:1
avg_lock_contention:0.031250
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:bfd56f64a0d362477baa6f6689c00df36bad0667
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:1.409841
used_cpu_user:12.015691
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:108
used_cpu_sys_main_thread:1.409841
used_cpu_user_main_thread:12.015691

# Modules

# Commandstats
cmdstat_get:calls=13655,usec=193595,usec_per_call=14.18,rejected_calls=0,failed_calls=0
cmdstat_del:calls=2880411,usec=13713807,usec_per_call=4.76,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=6,usec=3,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=2886,usec=31633,usec_per_call=10.96,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=88,usec=281,usec_per_call=3.19,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=2,usec=3,usec_per_call=1.50,rejected_calls=0,failed_calls=0
cmdstat_set:calls=3211387,usec=8627542,usec_per_call=2.69,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3211326,usec=5992111,usec_per_call=1.87,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=322081,expires=2343,avg_ttl=2386193452,cached_keys=322081

# KeyDB
mvcc_depth:0

------ CLIENT LIST OUTPUT ------
id=48 addr=192.168.254.53:33122 laddr=192.168.254.52:6379 fd=37 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=74 addr=192.168.254.52:48314 laddr=192.168.254.52:6379 fd=63 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=75 addr=192.168.254.53:33248 laddr=192.168.254.52:6379 fd=64 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=76 addr=192.168.254.52:48390 laddr=192.168.254.52:6379 fd=69 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=49 addr=192.168.254.53:33134 laddr=192.168.254.52:6379 fd=38 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=50 addr=192.168.254.52:48262 laddr=192.168.254.52:6379 fd=40 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=51 addr=192.168.254.52:48260 laddr=192.168.254.52:6379 fd=39 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=72 addr=192.168.254.53:33246 laddr=192.168.254.52:6379 fd=61 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=73 addr=192.168.254.53:33234 laddr=192.168.254.52:6379 fd=62 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=52 addr=192.168.254.52:48266 laddr=192.168.254.52:6379 fd=41 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=53 addr=192.168.254.53:33144 laddr=192.168.254.52:6379 fd=43 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=54 addr=192.168.254.53:33150 laddr=192.168.254.52:6379 fd=42 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=55 addr=192.168.254.52:48274 laddr=192.168.254.52:6379 fd=44 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=71 addr=192.168.254.53:33228 laddr=192.168.254.52:6379 fd=60 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=56 addr=192.168.254.52:48278 laddr=192.168.254.52:6379 fd=45 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=57 addr=192.168.254.52:48276 laddr=192.168.254.52:6379 fd=46 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=58 addr=192.168.254.53:33158 laddr=192.168.254.52:6379 fd=47 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=59 addr=192.168.254.53:33166 laddr=192.168.254.52:6379 fd=48 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=92 addr=192.168.254.53:60818 laddr=192.168.254.52:6379 fd=81 name= age=5082 idle=315 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=60 addr=192.168.254.52:48280 laddr=192.168.254.52:6379 fd=49 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=61 addr=192.168.254.52:48286 laddr=192.168.254.52:6379 fd=50 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=62 addr=192.168.254.52:48298 laddr=192.168.254.52:6379 fd=51 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=63 addr=192.168.254.53:33172 laddr=192.168.254.52:6379 fd=53 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=64 addr=192.168.254.52:48306 laddr=192.168.254.52:6379 fd=54 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=65 addr=192.168.254.53:33174 laddr=192.168.254.52:6379 fd=52 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=79 addr=192.168.254.52:48406 laddr=192.168.254.52:6379 fd=70 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=80 addr=192.168.254.52:48356 laddr=192.168.254.52:6379 fd=71 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=81 addr=192.168.254.52:48368 laddr=192.168.254.52:6379 fd=72 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=82 addr=192.168.254.52:48412 laddr=192.168.254.52:6379 fd=73 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=83 addr=192.168.254.53:33294 laddr=192.168.254.52:6379 fd=74 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=84 addr=192.168.254.52:48378 laddr=192.168.254.52:6379 fd=75 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=85 addr=192.168.254.53:33256 laddr=192.168.254.52:6379 fd=76 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=86 addr=192.168.254.53:33272 laddr=192.168.254.52:6379 fd=77 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=87 addr=192.168.254.53:33284 laddr=192.168.254.52:6379 fd=78 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=88 addr=192.168.254.52:48336 laddr=192.168.254.52:6379 fd=65 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=89 addr=192.168.254.52:48320 laddr=192.168.254.52:6379 fd=66 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=90 addr=192.168.254.52:48366 laddr=192.168.254.52:6379 fd=67 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=91 addr=192.168.254.52:48346 laddr=192.168.254.52:6379 fd=68 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=66 addr=192.168.254.53:33186 laddr=192.168.254.52:6379 fd=55 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=67 addr=192.168.254.53:33196 laddr=192.168.254.52:6379 fd=56 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=68 addr=192.168.254.53:33206 laddr=192.168.254.52:6379 fd=57 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=77 addr=192.168.254.52:48404 laddr=192.168.254.52:6379 fd=79 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=78 addr=192.168.254.53:33298 laddr=192.168.254.52:6379 fd=80 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=69 addr=192.168.254.53:33212 laddr=192.168.254.52:6379 fd=58 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=70 addr=192.168.254.53:33214 laddr=192.168.254.52:6379 fd=59 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=39 addr=192.168.254.53:38410 laddr=192.168.254.52:6379 fd=28 name= age=9493 idle=968 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=40 addr=192.168.254.53:38424 laddr=192.168.254.52:6379 fd=29 name= age=9493 idle=223 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=41 addr=192.168.254.53:38430 laddr=192.168.254.52:6379 fd=32 name= age=9492 idle=142 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=42 addr=192.168.254.53:47514 laddr=192.168.254.52:6379 fd=30 name= age=8569 idle=66 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=93 addr=192.168.254.53:60450 laddr=192.168.254.52:6379 fd=82 name= age=848 idle=848 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=43 addr=192.168.254.52:55578 laddr=192.168.254.52:6379 fd=34 name= age=6259 idle=6259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=44 addr=192.168.254.53:33090 laddr=192.168.254.52:6379 fd=31 name= age=6173 idle=141 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=45 addr=192.168.254.52:48246 laddr=192.168.254.52:6379 fd=33 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=46 addr=192.168.254.53:33106 laddr=192.168.254.52:6379 fd=35 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=47 addr=192.168.254.52:48254 laddr=192.168.254.52:6379 fd=36 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
397462:397491:M 19 Nov 2023 11:17:28.507 # Background saving cancelled
397462:397491:M 19 Nov 2023 11:17:28.607 * 10 changes in 300 seconds. Saving...
397462:397491:M 19 Nov 2023 11:17:28.696 * Background saving started by pid 408160
397462:397491:M 19 Nov 2023 11:17:28.696 * Background saving started

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
408160:397491:C 19 Nov 2023 11:17:42.750 # === ASSERTION FAILED ===
408160:397491:C 19 Nov 2023 11:17:42.750 # ==> rdb.cpp:1372 'ckeysExpired == db->expireSize()' is not true

I'm having the prolem on my production server. I'm not using replication.

From what I can tell, the crashes are triggered by a client connection making a change to the DB and triggering a write of the RDB file. The crash appears to happen during the write of the DB file. This then happens every time a new change is made to the DB.

I can stop the crashes by flushing the DB.

Unfortunately, this is a production server and there's a limit to what I can do.

I've flushed the DB and then put the server under significant load with the following script. I basically create a large number of keys with large values and random expire times, and then deletes them again. Some will naturally expire before I get round to deleting. This appears to work fine.

#!/usr/bin/env python

import random
import redis
import secrets
import threading
import time

REDIS_SERVER = dict(host='192.168.254.52', port=6379, db=0, password="*****")

def set_keys():
    client = redis.Redis(**REDIS_SERVER)

    v = secrets.token_hex(16384)

    keys = []
    for i in range(65536):
        k = secrets.token_hex(32)
        keys += [k]
        client.set(k, v)
        client.expire(k, random.randint(1, 32))
        if len(keys) > 8192:
            for k in keys:
                client.delete(k)
                keys = []

threads = []

for i in range(32):
    thread = threading.Thread(target=set_keys)
    threads.append(thread)
    thread.start()
    time.sleep(1)

for thread in threads:
    thread.join()

With the system crashing, I've set up a second instance on an identical machine and used a python script to copy all keys, values and associated TTLs to the new instance. The second system works fine, dumps to the RDB when required.

If I point the production clients at this new instance after about 24 hours, the issue returns....so it's not hardware related.

I have tried

enable-async-commands no

which I still have in place, but this does not resolve the issue.

It must be something to do with the contents of the DB or the way my clients (Python) are using it but I'm unable to isolate the problem.

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
keithchew commented 7 months ago

hi @rmd4

I have been testing keydb, and have experienced similar crashes. Adding more logging, I noticed when keydb is not properly shutdown, it loses the TTLs in they flash storage:

1023:1127:M 21 Nov 2023 02:04:41.188 * ckeysExpired [92601] db->expireSize() [0]

After this point, keydb will keep crashing on shutdown, and intermittently at runtime (this check is also done in other parts of the code at runtime).

Will post here if I find anything else useful.

In the meantime, the workaround is to delete the flash folder and reload from RDB, not very nice. Ugh, sorry, even with this workaround, another runtime error occurs when keydb is restarted:

9:578:M 21 Nov 2023 02:31:54.496 # === ASSERTION FAILED ===
9:578:M 21 Nov 2023 02:31:54.496 # ==> db.cpp:1833 'm_numexpires > 0' is not true
keithchew commented 7 months ago

hi @rmd4

It looks like KeyDB has some issues with the m_numexpires field when flash is enabled. My temporary solution is to patch it and rebuild m_numexpires on startup. At least it is not crashing on shutdown and startup anymore, so I can keep testing.

keithchew commented 7 months ago

After looking into this a bit more, the m_numexpires was introduced in:

https://github.com/Snapchat/KeyDB/pull/720

@msotheeswaran-sc it looks like this was pushed in v6.3.4 to fix the OOM issues by evicting items in flash. Briefly looking at the PR, m_numexpires in managed at runtime, but if the server restarts, it will be 0 and will no longer be in sync with flash?

rmd4 commented 7 months ago

I did wonder if this was related to flash storge as I have also seen the 

db.cpp:1833 'm_numexpires > 0'

issue. However, I turned off flash storage and that one was resolved, athough it was effectively replaced with the db->expireSize() issue. Both would occur at roughly the same time of day (0405am even though my housekeeping starts at 0200 and is complete by 0300 - no idea why 0405).

My current config does not make use of flash:

# Enable FLASH support (Experimental Feature)
#storage-provider flash /mnt/keydb/keydb

I even removed the /mnt/keydb mount point, just to make sure ;) However, I'm still getting the 

ckeysExpired == db->expireSize()

issue. I can 'resolve' it by turning off database dumping completely:

save ""

#save 900 1
#save 300 10
#save 60 10000

I've been stable now for 48+ hours.

I don't believe it's releated to an unclean shutdown as I can flush the DB and restart (DB save enabled, no flash) and the issue will return within 24 hours of production use.

It must be linked to something I'm managing to place in the DB, but as per the Python script above, it's not something I have managed to reproduce other than with my production code (a website where load is usage driven).

keithchew commented 7 months ago

hi @rmd4

As per my previous comment, I tracked it down to the eviction changes for flash introduced in v6.3.4. My temporary workaround for this is to set m_numexpires on startup. So far, haven't had the crash, ie with both flash and data dumping enabled.

For unclean shutdown, use a KILL -9 and also make sure you have entries with TTL in flash. And this may not be the only way to trigger the bug, as you have mentioned, it can also happen at runtime. I did find that once the variable is out of sync, then the crash can happen anytime (dump at shutdown or at runtime).

My current goal is to stress test the flash feature in the most challenging way I can, ie maxmemory smaller than dataset (so keys get evicted from memory), 5000+ read/writes per second, module scanning, RDB background saving, etc. I have had countless crashes, and just hacking the code to workaround them. So far, I am very impressed with KeyDB.

splitice commented 7 months ago

Technically a duplicate of https://github.com/Snapchat/KeyDB/issues/739 but there is more discussion here.

I don't use flash storage and yet have this very issue.

@keithchew can you please share your patch?

KirschenStas commented 7 months ago

Hello! I've received the same issue on keydb.

Finding the root case of issue, i've notes, that if keyspace looks like this:

keydb@keydb-0:/data$ keydb-cli info keyspace
# Keyspace
db0:keys=3305,expires=3305,avg_ttl=44022308,cached_keys=3305
db1:keys=11204,expires=11204,avg_ttl=75608578,cached_keys=11204
db2:keys=6136,expires=6136,avg_ttl=851903,cached_keys=6136
db3:keys=64,expires=70,avg_ttl=826725,cached_keys=64
db4:keys=26541,expires=26543,avg_ttl=4932380,cached_keys=26541

keys in db3 and db4 are not equal to expires.

keydb-cli bgsave reports in log STACK TRACE (see attached keydb-0.log file)

if run commands: keydb-cli -n 3 flushdb keydb-cli -n 4 flushdb

then command keydb-cli bgsave runs successfully.

I've also notes, that this behavior occurs when application, that uses keydb service call DEL-command on keys with TTL.

From redis-exporter, if make substitution of redis_db_keys and redis_db_keys_expiring, it'll be negative indicators: image

keithchew commented 7 months ago

Hi @splitice

In db.cpp, you can update m_numexpires when KeyDB iterates the keys on startup, eg:

  size_t offset = 0;
  std::unique_ptr<expireEntry> spexpire = deserializeExpire((const char*)data, cbData, &offset);
  if (spexpire != nullptr) {
          m_numexpires += 1;
  }
splitice commented 6 months ago

@keithchew I couldnt find that line in db.cpp, it appears that you mean snapshot.cpp?

But that can't be right either, that code path is called on many others (such as the KEYS command) and will end up corrupting m_numexpires when it is.

keithchew commented 6 months ago

Hi @splitice

On my local testing copy, I have created a new method that is only called on startup, to prevent the multiple flows. The snippet above is what I added, do a search for "iterate" which will have the code that does the iteration.

splitice commented 6 months ago

@keithchew could you share an actual patch? Thats far beyond just adding a line or two and requires an actual understanding of the control flow of the relatively undocumented KeyDB source code.

splitice commented 6 months ago
  1. Heres the commit that introduces the issue https://github.com/Snapchat/KeyDB/commit/ffac55273a322d53c32372536cd3043d0b413989
  2. I'm willing to be proven wrong (and I havent yet tested it) but in reviewing the code I think the actual problem is in restoreSnapshot.

m_numexpires = psnapshot->m_numexpires;

needs to be added at the bottom.

This won't fix anyone whose RDB file is now corrupt as of 6.3.4, so restore from a 6.3.3 backup. I'll test this in the coming days.

diff --git a/src/snapshot.cpp b/src/snapshot.cpp
index 94956ebc2..fe6b98983 100644
--- a/src/snapshot.cpp
+++ b/src/snapshot.cpp
@@ -168,6 +168,7 @@ void redisDbPersistentData::restoreSnapshot(const redisDbPersistentDataSnapshot
     dictEmpty(m_pdictTombstone, nullptr);
     endSnapshot(psnapshot);
     serverAssert(size() == expectedSize);
+    m_numexpires = psnapshot->m_numexpires;
 }

 // This function is all about minimizing the amount of work done under global lock
keithchew commented 6 months ago

Hi @splitice

Sorry I don't have a patch as yet, as I am just copying and pasting snippets here and there as quick fixes to prevent crashes, as my current goal is to evaluate KeyDB for a project. Once I have completed the evaluation, hopefully I will have time to post PRs back here. In the meantime, all I can provide are guidelines for developers who are going through the same issues.

ilyavaiser commented 5 months ago

Will this error be corrected? It is surprising to catch such problems in releases. I thought that this project was much more stable

splitice commented 5 months ago

BTW while I did have a go at a fixing this I felt the current release was too much of a bad idea and ended up doing a downgrade.

We are currently considering if KeyDB is something we can consider production grade and what would be involved in migration to redis cluster.

keithchew commented 5 months ago

Given the relatively large number of tweaks I made to get my copy working (and this is only for my use-cases), I agree with @splitice that you should not use v6.3.4 if you encounter this issue in your setup. Would be better to wait for the next official release from the developers and check if it is fixed then. For me, I need v6.3.4 because I have to get quite a number of dependent things working for my evaluation, eg Flash, modules, etc, so I pushed forward instead.

nguyenduybinh commented 4 months ago

i have the same issue after i upgraded my keydb server from 6.3.2 to 6.3.4. My keydb server now can't save rdb file to persistent. Are there any plans to fix this issue?