351001:351012:M 09 Jan 2023 21:50:38.434 Background saving terminated with success
351001:351012:M 09 Jan 2023 21:51:39.056 10000 changes in 60 seconds. Saving...
351001:351012:M 09 Jan 2023 21:51:39.057 Background saving started
351001:1793291:M 09 Jan 2023 21:51:43.450 DB saved on disk
351001:1793291:M 09 Jan 2023 21:51:43.475 RDB: 393 MB of memory used by copy-on-write
351001:351012:M 09 Jan 2023 21:51:43.565 Background saving terminated with success
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
351001:351012:M 09 Jan 2023 21:51:49.892 # ------------------------------------------------
351001:351012:M 09 Jan 2023 21:51:49.892 # !!! Software Failure. Press left mouse button to continue
351001:351012:M 09 Jan 2023 21:51:49.892 # Guru Meditation: Unknown hash encoding #defrag.cpp:918
------ FAST MEMORY TEST ------
351001:351012:M 09 Jan 2023 21:51:49.906 # main thread terminated
351001:351012:M 09 Jan 2023 21:51:49.907 # Bio thread for job type #0 terminated
351001:351012:M 09 Jan 2023 21:51:49.907 # Bio thread for job type #1 terminated
351001:351012:M 09 Jan 2023 21:51:49.907 # Bio thread for job type #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== KEYDB BUG REPORT END. Make sure to include from START to END. ===
Aditional information
Ubuntu 22.04.1 LTS
Situation: Multiple workers do parallel inserts using Watch + Multi/exec. The multi contains a bunch of HSETNX, 1 ZADD, 1 ZREMRANGEBYSCORE, and 1 SET. Not sure if this matters at all.
2 times out of 2 this has happened when running on a large load. Not at the exact same time (or datapoint afaik) but around the 10-15m mark
Crash report
351001:351012:M 09 Jan 2023 21:50:38.434 Background saving terminated with success 351001:351012:M 09 Jan 2023 21:51:39.056 10000 changes in 60 seconds. Saving... 351001:351012:M 09 Jan 2023 21:51:39.057 Background saving started 351001:1793291:M 09 Jan 2023 21:51:43.450 DB saved on disk 351001:1793291:M 09 Jan 2023 21:51:43.475 RDB: 393 MB of memory used by copy-on-write 351001:351012:M 09 Jan 2023 21:51:43.565 Background saving terminated with success
=== KEYDB BUG REPORT START: Cut & paste starting from here === 351001:351012:M 09 Jan 2023 21:51:49.892 # ------------------------------------------------ 351001:351012:M 09 Jan 2023 21:51:49.892 # !!! Software Failure. Press left mouse button to continue 351001:351012:M 09 Jan 2023 21:51:49.892 # Guru Meditation: Unknown hash encoding #defrag.cpp:918
------ STACK TRACE ------
Backtrace: ./keydb-server 0.0.0.0:6379(defragKey(redisDb, dictEntry)+0x3af) [0x56357a73654f] ./keydb-server 0.0.0.0:6379(defragScanCallback(void, dictEntry const)+0xd) [0x56357a73658d] ./keydb-server 0.0.0.0:6379(+0x15ccfd) [0x56357a630cfd] ./keydb-server 0.0.0.0:6379(activeDefragCycle()+0x3d3) [0x56357a732b83] ./keydb-server 0.0.0.0:6379(databasesCron(bool)+0x75) [0x56357a633df5] ./keydb-server 0.0.0.0:6379(serverCron(aeEventLoop, long long, void)+0x164) [0x56357a634a74] ./keydb-server 0.0.0.0:6379(aeProcessEvents+0x26c) [0x56357a62c6bc] ./keydb-server 0.0.0.0:6379(aeMain+0x3e) [0x56357a62cdae] ./keydb-server 0.0.0.0:6379(workerThreadMain(void*)+0x1b0) [0x56357a6440f0] /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f9d8d19ab43] /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9d8d22ca00]
------ INFO OUTPUT ------
Server
redis_version:255.255.255 redis_git_sha1:b50f0cc2 redis_git_dirty:0 redis_build_id:c77f60a8db92ffa9 redis_mode:standalone os:Linux 5.15.0-43-generic x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:11.2.0 process_id:351001 process_supervised:no run_id:0db4fa99060c869e0ae65f0f24c5669d78245d99 tcp_port:6379 server_time_usec:1673297509904718 uptime_in_seconds:834 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:12353125 executable:/root/KeyDB/src/./keydb-server config_file:/etc/redis/keydb.conf
Clients
connected_clients:60 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:10503 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0 current_client_thread:0 thread_0_clients:32 thread_1_clients:28
Memory
used_memory:259400712 used_memory_human:247.38M used_memory_rss:386244608 used_memory_rss_human:368.35M used_memory_peak:384517552 used_memory_peak_human:366.70M used_memory_peak_perc:67.46% used_memory_overhead:50489308 used_memory_startup:2121080 used_memory_dataset:208911404 used_memory_dataset_perc:81.20% allocator_allocated:260202800 allocator_active:357376000 allocator_resident:390918144 total_system_memory:134982266880 total_system_memory_human:125.71G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.37 allocator_frag_bytes:97173200 allocator_rss_ratio:1.09 allocator_rss_bytes:33542144 rss_overhead_ratio:0.99 rss_overhead_bytes:-4673536 mem_fragmentation_ratio:1.49 mem_fragmentation_bytes:126722912 mem_not_counted_for_evict:1048576 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:1169116 mem_aof_buffer:0 mem_allocator:jemalloc-5.2.1 active_defrag_running:10 lazyfree_pending_objects:0 lazyfreed_objects:0 storage_provider:none
Persistence
loading:0 current_cow_size:412393472 current_cow_size_age:7 current_fork_perc:0.00 current_save_keys_processed:866305 current_save_keys_total:0 rdb_changes_since_last_save:182562 rdb_bgsave_in_progress:0 rdb_last_save_time:1673297503 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:4 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:412938240 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0
Stats
total_connections_received:138831 total_commands_processed:23315897 instantaneous_ops_per_sec:44540 total_net_input_bytes:1803921916 total_net_output_bytes:345670690 instantaneous_input_kbps:3136.62 instantaneous_output_kbps:668.45 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 keyspace_hits:16297114 keyspace_misses:2545301 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 total_forks:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:465789 active_defrag_misses:5111858 active_defrag_key_hits:276711 active_defrag_key_misses:431777 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:0 dump_payload_sanitizations:0 total_reads_processed:1787073 total_writes_processed:1645320 instantaneous_lock_contention:2 avg_lock_contention:1.156250 storage_provider_read_hits:0 storage_provider_read_misses:0
Replication
role:master connected_slaves:0 master_failover_state:no-failover master_replid:7a9c45fe5c7254ec0e5fcccf6a274fed45af893c master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0
CPU
used_cpu_sys:74.926231 used_cpu_user:255.116738 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000 server_threads:2 long_lock_waits:11743 used_cpu_sys_main_thread:29.902559 used_cpu_user_main_thread:106.542323
Modules
Commandstats
cmdstat_zadd:calls=176695,usec=20346977,usec_per_call=115.15,rejected_calls=0,failed_calls=0 cmdstat_zremrangebyscore:calls=176695,usec=1908120,usec_per_call=10.80,rejected_calls=0,failed_calls=0 cmdstat_exec:calls=315467,usec=37499031,usec_per_call=118.87,rejected_calls=0,failed_calls=0 cmdstat_info:calls=1,usec=704,usec_per_call=704.00,rejected_calls=0,failed_calls=0 cmdstat_hmget:calls=18526947,usec=41379146,usec_per_call=2.23,rejected_calls=0,failed_calls=0 cmdstat_hsetnx:calls=2858161,usec=11025610,usec_per_call=3.86,rejected_calls=0,failed_calls=0 cmdstat_command:calls=1,usec=2192,usec_per_call=2192.00,rejected_calls=0,failed_calls=0 cmdstat_set:calls=176695,usec=555285,usec_per_call=3.14,rejected_calls=0,failed_calls=0 cmdstat_multi:calls=315467,usec=201176,usec_per_call=0.64,rejected_calls=0,failed_calls=0 cmdstat_auth:calls=138831,usec=650579,usec_per_call=4.69,rejected_calls=0,failed_calls=0 cmdstat_get:calls=315468,usec=712394,usec_per_call=2.26,rejected_calls=0,failed_calls=0 cmdstat_scan:calls=1,usec=38,usec_per_call=38.00,rejected_calls=0,failed_calls=0 cmdstat_watch:calls=315468,usec=606625,usec_per_call=1.92,rejected_calls=0,failed_calls=0
Errorstats
Cluster
cluster_enabled:0
Keyspace
db0:keys=970160,expires=0,avg_ttl=0,cached_keys=970160
KeyDB
mvcc_depth:0
------ CLIENT LIST OUTPUT ------ id=28 addr=127.0.0.1:48556 laddr=127.0.0.1:6379 fd=40 name= age=782 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=8 addr=127.0.0.1:48510 laddr=127.0.0.1:6379 fd=20 name= age=782 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=hmget user=default redir=-1 id=9 addr=127.0.0.1:48512 laddr=127.0.0.1:6379 fd=21 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=138832 addr=127.0.0.1:34532 laddr=127.0.0.1:6379 fd=84 name= age=0 idle=0 flags=d db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1 id=138833 addr=127.0.0.1:34536 laddr=127.0.0.1:6379 fd=103 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=auth user=default redir=-1 id=138834 addr=127.0.0.1:34538 laddr=127.0.0.1:6379 fd=104 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1 id=10 addr=127.0.0.1:48514 laddr=127.0.0.1:6379 fd=22 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=11 addr=127.0.0.1:48516 laddr=127.0.0.1:6379 fd=23 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=12 addr=127.0.0.1:48518 laddr=127.0.0.1:6379 fd=24 name= age=782 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=4 addr=127.0.0.1:48502 laddr=127.0.0.1:6379 fd=16 name= age=783 idle=783 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=scan user=default redir=-1 id=138824 addr=127.0.0.1:34514 laddr=127.0.0.1:6379 fd=90 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1 id=138825 addr=127.0.0.1:34516 laddr=127.0.0.1:6379 fd=70 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1 id=5 addr=127.0.0.1:48504 laddr=127.0.0.1:6379 fd=17 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=6 addr=127.0.0.1:48506 laddr=127.0.0.1:6379 fd=18 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=7 addr=127.0.0.1:48508 laddr=127.0.0.1:6379 fd=19 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=13 addr=127.0.0.1:48520 laddr=127.0.0.1:6379 fd=25 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=138831 addr=127.0.0.1:34528 laddr=127.0.0.1:6379 fd=68 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1 id=14 addr=127.0.0.1:48524 laddr=127.0.0.1:6379 fd=26 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=15 addr=127.0.0.1:48526 laddr=127.0.0.1:6379 fd=27 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=16 addr=127.0.0.1:48528 laddr=127.0.0.1:6379 fd=28 name= age=782 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=17 addr=127.0.0.1:48530 laddr=127.0.0.1:6379 fd=29 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=18 addr=127.0.0.1:48532 laddr=127.0.0.1:6379 fd=30 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=19 addr=127.0.0.1:48534 laddr=127.0.0.1:6379 fd=31 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=20 addr=127.0.0.1:48538 laddr=127.0.0.1:6379 fd=32 name= age=782 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=21 addr=127.0.0.1:48540 laddr=127.0.0.1:6379 fd=33 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=138829 addr=127.0.0.1:34524 laddr=127.0.0.1:6379 fd=71 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1 id=22 addr=127.0.0.1:48542 laddr=127.0.0.1:6379 fd=34 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=23 addr=127.0.0.1:48546 laddr=127.0.0.1:6379 fd=35 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=29 addr=127.0.0.1:48558 laddr=127.0.0.1:6379 fd=41 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=30 addr=127.0.0.1:48560 laddr=127.0.0.1:6379 fd=42 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=31 addr=127.0.0.1:48562 laddr=127.0.0.1:6379 fd=43 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=138827 addr=127.0.0.1:34520 laddr=127.0.0.1:6379 fd=74 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1 id=32 addr=127.0.0.1:48564 laddr=127.0.0.1:6379 fd=44 name= age=781 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=33 addr=127.0.0.1:48566 laddr=127.0.0.1:6379 fd=45 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=34 addr=127.0.0.1:48568 laddr=127.0.0.1:6379 fd=46 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=137477 addr=127.0.0.1:59876 laddr=127.0.0.1:6379 fd=99 name= age=83 idle=75 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=info user=default redir=-1 id=35 addr=127.0.0.1:48570 laddr=127.0.0.1:6379 fd=47 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=36 addr=127.0.0.1:48572 laddr=127.0.0.1:6379 fd=48 name= age=781 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=37 addr=127.0.0.1:48576 laddr=127.0.0.1:6379 fd=49 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=38 addr=127.0.0.1:48578 laddr=127.0.0.1:6379 fd=50 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=39 addr=127.0.0.1:48580 laddr=127.0.0.1:6379 fd=51 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=40 addr=127.0.0.1:48582 laddr=127.0.0.1:6379 fd=52 name= age=781 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=41 addr=127.0.0.1:48584 laddr=127.0.0.1:6379 fd=53 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=52 addr=127.0.0.1:48608 laddr=127.0.0.1:6379 fd=64 name= age=781 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=42 addr=127.0.0.1:48586 laddr=127.0.0.1:6379 fd=54 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=43 addr=127.0.0.1:48588 laddr=127.0.0.1:6379 fd=55 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=138808 addr=127.0.0.1:34482 laddr=127.0.0.1:6379 fd=89 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1 id=138809 addr=127.0.0.1:34484 laddr=127.0.0.1:6379 fd=88 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=29 obl=0 oll=0 omem=0 tot-mem=61477 events=r cmd=exec user=default redir=-1 id=44 addr=127.0.0.1:48590 laddr=127.0.0.1:6379 fd=56 name= age=781 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=45 addr=127.0.0.1:48592 laddr=127.0.0.1:6379 fd=57 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=46 addr=127.0.0.1:48594 laddr=127.0.0.1:6379 fd=58 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=47 addr=127.0.0.1:48598 laddr=127.0.0.1:6379 fd=59 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=48 addr=127.0.0.1:48600 laddr=127.0.0.1:6379 fd=60 name= age=781 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=49 addr=127.0.0.1:48602 laddr=127.0.0.1:6379 fd=61 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=50 addr=127.0.0.1:48604 laddr=127.0.0.1:6379 fd=62 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=51 addr=127.0.0.1:48606 laddr=127.0.0.1:6379 fd=63 name= age=781 idle=781 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=24 addr=127.0.0.1:48548 laddr=127.0.0.1:6379 fd=36 name= age=782 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=hmget user=default redir=-1 id=25 addr=127.0.0.1:48550 laddr=127.0.0.1:6379 fd=37 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=26 addr=127.0.0.1:48552 laddr=127.0.0.1:6379 fd=38 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1 id=27 addr=127.0.0.1:48554 laddr=127.0.0.1:6379 fd=39 name= age=782 idle=782 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=auth user=default redir=-1
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------ 351001:351012:M 09 Jan 2023 21:51:49.906 # main thread terminated 351001:351012:M 09 Jan 2023 21:51:49.907 # Bio thread for job type #0 terminated 351001:351012:M 09 Jan 2023 21:51:49.907 # Bio thread for job type #1 terminated 351001:351012:M 09 Jan 2023 21:51:49.907 # Bio thread for job type #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== KEYDB BUG REPORT END. Make sure to include from START to END. ===
Aditional information
Situation: Multiple workers do parallel inserts using Watch + Multi/exec. The multi contains a bunch of HSETNX, 1 ZADD, 1 ZREMRANGEBYSCORE, and 1 SET. Not sure if this matters at all.
2 times out of 2 this has happened when running on a large load. Not at the exact same time (or datapoint afaik) but around the 10-15m mark