search

Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.02k stars 564 forks source link

[CRASH] Random Full KeyDB Crash #745

Open SwordOfSouls opened 7 months ago

SwordOfSouls commented 7 months ago

Crash report

Was simply using Redisson Remote Services with 10 workers and receivers.

2023-11-21 00:43:01 === KEYDB BUG REPORT START: Cut & paste starting from here ===
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.944 # ------------------------------------------------
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.944 # !!! Software Failure. Press left mouse button to continue
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.945 # Guru Meditation: Unknown btype in replyToBlockedClientTimedOut(). #blocked.cpp:246
2023-11-21 00:43:01 
2023-11-21 00:43:01 ------ STACK TRACE ------
2023-11-21 00:43:01 
2023-11-21 00:43:01 Backtrace:
2023-11-21 00:43:01 keydb-server *:6379(replyToBlockedClientTimedOut(client*)+0x72) [0x558c27676a12]
2023-11-21 00:43:01 keydb-server *:6379(checkBlockedClientTimeout(client*, long long)+0x46) [0x558c27635ac6]
2023-11-21 00:43:01 keydb-server *:6379(handleBlockedClientsTimeout()+0x110) [0x558c27635bf0]
2023-11-21 00:43:01 keydb-server *:6379(beforeSleep(aeEventLoop*)+0x171) [0x558c277272a1]
2023-11-21 00:43:01 keydb-server *:6379(aeProcessEvents+0x3d0) [0x558c27732750]
2023-11-21 00:43:01 keydb-server *:6379(aeMain+0x3a) [0x558c27736cda]
2023-11-21 00:43:01 keydb-server *:6379(workerThreadMain(void*)+0x7e) [0x558c2771eade]
2023-11-21 00:43:01 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3781e6e609]
2023-11-21 00:43:01 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f3781d93133]
2023-11-21 00:43:01 
2023-11-21 00:43:01 ------ INFO OUTPUT ------
2023-11-21 00:43:01 # Server
2023-11-21 00:43:01 redis_version:6.3.4
2023-11-21 00:43:01 redis_git_sha1:7e7e5e57
2023-11-21 00:43:01 redis_git_dirty:1
2023-11-21 00:43:01 redis_build_id:bb8fc59400781b64
2023-11-21 00:43:01 redis_mode:standalone
2023-11-21 00:43:01 os:Linux 5.15.133.1-microsoft-standard-WSL2 x86_64
2023-11-21 00:43:01 arch_bits:64
2023-11-21 00:43:01 multiplexing_api:epoll
2023-11-21 00:43:01 atomicvar_api:atomic-builtin
2023-11-21 00:43:01 gcc_version:9.4.0
2023-11-21 00:43:01 process_id:1
2023-11-21 00:43:01 process_supervised:no
2023-11-21 00:43:01 run_id:7791b84100fe4f72e62848542ad490160f687a6d
2023-11-21 00:43:01 tcp_port:6379
2023-11-21 00:43:01 server_time_usec:1700556181953646
2023-11-21 00:43:01 uptime_in_seconds:10080
2023-11-21 00:43:01 uptime_in_days:0
2023-11-21 00:43:01 hz:10
2023-11-21 00:43:01 configured_hz:10
2023-11-21 00:43:01 lru_clock:6057365
2023-11-21 00:43:01 executable:/data/keydb-server
2023-11-21 00:43:01 config_file:/etc/keydb/keydb.conf
2023-11-21 00:43:01 availability_zone:
2023-11-21 00:43:01 features:cluster_mget
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Clients
2023-11-21 00:43:01 connected_clients:0
2023-11-21 00:43:01 cluster_connections:0
2023-11-21 00:43:01 maxclients:10000
2023-11-21 00:43:01 client_recent_max_input_buffer:0
2023-11-21 00:43:01 client_recent_max_output_buffer:0
2023-11-21 00:43:01 blocked_clients:0
2023-11-21 00:43:01 tracking_clients:0
2023-11-21 00:43:01 clients_in_timeout_table:204
2023-11-21 00:43:01 current_client_thread:1
2023-11-21 00:43:01 thread_0_clients:0
2023-11-21 00:43:01 thread_1_clients:0
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Memory
2023-11-21 00:43:01 used_memory:2672336
2023-11-21 00:43:01 used_memory_human:2.55M
2023-11-21 00:43:01 used_memory_rss:30027776
2023-11-21 00:43:01 used_memory_rss_human:28.64M
2023-11-21 00:43:01 used_memory_peak:17400288
2023-11-21 00:43:01 used_memory_peak_human:16.59M
2023-11-21 00:43:01 used_memory_peak_perc:15.36%
2023-11-21 00:43:01 used_memory_overhead:2172256
2023-11-21 00:43:01 used_memory_startup:2170872
2023-11-21 00:43:01 used_memory_dataset:500080
2023-11-21 00:43:01 used_memory_dataset_perc:99.72%
2023-11-21 00:43:01 allocator_allocated:4346712
2023-11-21 00:43:01 allocator_active:5636096
2023-11-21 00:43:01 allocator_resident:12832768
2023-11-21 00:43:01 total_system_memory:29430140928
2023-11-21 00:43:01 total_system_memory_human:27.41G
2023-11-21 00:43:01 used_memory_lua:46080
2023-11-21 00:43:01 used_memory_lua_human:45.00K
2023-11-21 00:43:01 used_memory_scripts:1272
2023-11-21 00:43:01 used_memory_scripts_human:1.24K
2023-11-21 00:43:01 number_of_cached_scripts:5
2023-11-21 00:43:01 maxmemory:0
2023-11-21 00:43:01 maxmemory_human:0B
2023-11-21 00:43:01 maxmemory_policy:noeviction
2023-11-21 00:43:01 allocator_frag_ratio:1.30
2023-11-21 00:43:01 allocator_frag_bytes:1289384
2023-11-21 00:43:01 allocator_rss_ratio:2.28
2023-11-21 00:43:01 allocator_rss_bytes:7196672
2023-11-21 00:43:01 rss_overhead_ratio:2.34
2023-11-21 00:43:01 rss_overhead_bytes:17195008
2023-11-21 00:43:01 mem_fragmentation_ratio:11.24
2023-11-21 00:43:01 mem_fragmentation_bytes:27356616
2023-11-21 00:43:01 mem_not_counted_for_evict:0
2023-11-21 00:43:01 mem_replication_backlog:0
2023-11-21 00:43:01 mem_clients_slaves:0
2023-11-21 00:43:01 mem_clients_normal:0
2023-11-21 00:43:01 mem_aof_buffer:0
2023-11-21 00:43:01 mem_allocator:jemalloc-5.2.1
2023-11-21 00:43:01 active_defrag_running:0
2023-11-21 00:43:01 lazyfree_pending_objects:0
2023-11-21 00:43:01 lazyfreed_objects:0
2023-11-21 00:43:01 storage_provider:none
2023-11-21 00:43:01 available_system_memory:unavailable
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Persistence
2023-11-21 00:43:01 loading:0
2023-11-21 00:43:01 current_cow_size:0
2023-11-21 00:43:01 current_cow_size_age:0
2023-11-21 00:43:01 current_fork_perc:0.00
2023-11-21 00:43:01 current_save_keys_processed:0
2023-11-21 00:43:01 current_save_keys_total:0
2023-11-21 00:43:01 rdb_changes_since_last_save:374
2023-11-21 00:43:01 rdb_bgsave_in_progress:0
2023-11-21 00:43:01 rdb_last_save_time:1700556125
2023-11-21 00:43:01 rdb_last_bgsave_status:ok
2023-11-21 00:43:01 rdb_last_bgsave_time_sec:0
2023-11-21 00:43:01 rdb_current_bgsave_time_sec:-1
2023-11-21 00:43:01 rdb_last_cow_size:802816
2023-11-21 00:43:01 aof_enabled:0
2023-11-21 00:43:01 aof_rewrite_in_progress:0
2023-11-21 00:43:01 aof_rewrite_scheduled:0
2023-11-21 00:43:01 aof_last_rewrite_time_sec:-1
2023-11-21 00:43:01 aof_current_rewrite_time_sec:-1
2023-11-21 00:43:01 aof_last_bgrewrite_status:ok
2023-11-21 00:43:01 aof_last_write_status:ok
2023-11-21 00:43:01 aof_last_cow_size:0
2023-11-21 00:43:01 module_fork_in_progress:0
2023-11-21 00:43:01 module_fork_last_cow_size:0
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Stats
2023-11-21 00:43:01 total_connections_received:1189
2023-11-21 00:43:01 total_commands_processed:20717
2023-11-21 00:43:01 instantaneous_ops_per_sec:0
2023-11-21 00:43:01 total_net_input_bytes:1754898
2023-11-21 00:43:01 total_net_output_bytes:742431
2023-11-21 00:43:01 instantaneous_input_kbps:0.00
2023-11-21 00:43:01 instantaneous_output_kbps:0.00
2023-11-21 00:43:01 rejected_connections:0
2023-11-21 00:43:01 sync_full:0
2023-11-21 00:43:01 sync_partial_ok:0
2023-11-21 00:43:01 sync_partial_err:0
2023-11-21 00:43:01 expired_keys:15
2023-11-21 00:43:01 expired_stale_perc:0.00
2023-11-21 00:43:01 expired_time_cap_reached_count:0
2023-11-21 00:43:01 expire_cycle_cpu_milliseconds:68
2023-11-21 00:43:01 evicted_keys:0
2023-11-21 00:43:01 keyspace_hits:284
2023-11-21 00:43:01 keyspace_misses:0
2023-11-21 00:43:01 pubsub_channels:0
2023-11-21 00:43:01 pubsub_patterns:0
2023-11-21 00:43:01 latest_fork_usec:899
2023-11-21 00:43:01 total_forks:26
2023-11-21 00:43:01 migrate_cached_sockets:0
2023-11-21 00:43:01 slave_expires_tracked_keys:0
2023-11-21 00:43:01 active_defrag_hits:0
2023-11-21 00:43:01 active_defrag_misses:0
2023-11-21 00:43:01 active_defrag_key_hits:0
2023-11-21 00:43:01 active_defrag_key_misses:0
2023-11-21 00:43:01 tracking_total_keys:0
2023-11-21 00:43:01 tracking_total_items:0
2023-11-21 00:43:01 tracking_total_prefixes:0
2023-11-21 00:43:01 unexpected_error_replies:0
2023-11-21 00:43:01 total_error_replies:0
2023-11-21 00:43:01 dump_payload_sanitizations:0
2023-11-21 00:43:01 total_reads_processed:19974
2023-11-21 00:43:01 total_writes_processed:19658
2023-11-21 00:43:01 instantaneous_lock_contention:2
2023-11-21 00:43:01 avg_lock_contention:0.015625
2023-11-21 00:43:01 storage_provider_read_hits:0
2023-11-21 00:43:01 storage_provider_read_misses:0
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Replication
2023-11-21 00:43:01 role:master
2023-11-21 00:43:01 connected_slaves:0
2023-11-21 00:43:01 master_failover_state:no-failover
2023-11-21 00:43:01 master_replid:1c545b66dabe8a5eba68f88072ad111e9dbf0d1e
2023-11-21 00:43:01 master_replid2:0000000000000000000000000000000000000000
2023-11-21 00:43:01 master_repl_offset:0
2023-11-21 00:43:01 second_repl_offset:-1
2023-11-21 00:43:01 repl_backlog_active:0
2023-11-21 00:43:01 repl_backlog_size:1048576
2023-11-21 00:43:01 repl_backlog_first_byte_offset:0
2023-11-21 00:43:01 repl_backlog_histlen:0
2023-11-21 00:43:01 
2023-11-21 00:43:01 # CPU
2023-11-21 00:43:01 used_cpu_sys:11.131091
2023-11-21 00:43:01 used_cpu_user:12.458180
2023-11-21 00:43:01 used_cpu_sys_children:0.058104
2023-11-21 00:43:01 used_cpu_user_children:0.062015
2023-11-21 00:43:01 server_threads:2
2023-11-21 00:43:01 long_lock_waits:0
2023-11-21 00:43:01 used_cpu_sys_main_thread:2.281713
2023-11-21 00:43:01 used_cpu_user_main_thread:2.556364
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Modules
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Commandstats
2023-11-21 00:43:01 cmdstat_hdel:calls=284,usec=759,usec_per_call=2.67,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_setnx:calls=284,usec=1280,usec_per_call=4.51,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_del:calls=269,usec=751,usec_per_call=2.79,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_eval:calls=1121,usec=27284,usec_per_call=24.34,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_rpush:calls=1032,usec=8697,usec_per_call=8.43,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_blpop:calls=950,usec=28076,usec_per_call=29.55,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_ping:calls=15656,usec=19357,usec_per_call=1.24,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_hget:calls=284,usec=973,usec_per_call=3.43,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_hset:calls=284,usec=2023,usec_per_call=7.12,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 cmdstat_pexpire:calls=553,usec=707,usec_per_call=1.28,rejected_calls=0,failed_calls=0
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Errorstats
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Cluster
2023-11-21 00:43:01 cluster_enabled:0
2023-11-21 00:43:01 
2023-11-21 00:43:01 # Keyspace
2023-11-21 00:43:01 db0:keys=1,expires=0,avg_ttl=0,cached_keys=1
2023-11-21 00:43:01 
2023-11-21 00:43:01 # KeyDB
2023-11-21 00:43:01 mvcc_depth:0
2023-11-21 00:43:01 
2023-11-21 00:43:01 ------ CLIENT LIST OUTPUT ------
2023-11-21 00:43:01 
2023-11-21 00:43:01 ------ MODULES INFO OUTPUT ------
2023-11-21 00:43:01 
2023-11-21 00:43:01 ------ FAST MEMORY TEST ------
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.956 # main thread terminated
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.957 # Bio thread for job type #0 terminated
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.959 # Bio thread for job type #1 terminated
2023-11-21 00:43:01 1:26:M 21 Nov 2023 08:43:01.960 # Bio thread for job type #2 terminated
2023-11-21 00:43:01 
2023-11-21 00:43:01 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

Aditional information

  1. Docker running on WSL 2 (ubuntu-20.04)
  2. N/A