Skoucail
commented
2 years ago We are having the same issue.
Open danialmmllc opened 3 years ago
Skoucail
commented
2 years ago We are having the same issue.
pklimczu
commented
2 years ago Unfortunately, it also happens at our side:
1:M 29 Sep 2021 15:42:03.295 # Redis 6.2.4 crashed by signal: 11, si_code: 1
1:M 29 Sep 2021 15:42:03.295 # Accessing address: 0xe0000000e
1:M 29 Sep 2021 15:42:03.295 # Crashed running the instruction at: 0x7f9663976f1b
------ STACK TRACE ------
EIP:
/usr/lib/redis/modules/module-oss.so(+0xd0f1b)[0x7f9663976f1b]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f9664a22730]
/usr/lib/redis/modules/module-oss.so(+0xd0f1b)[0x7f9663976f1b]
/usr/lib/redis/modules/module-oss.so(+0xdcb76)[0x7f9663982b76]
/usr/lib/redis/modules/module-oss.so(+0xdd53a)[0x7f966398353a]
/usr/lib/redis/modules/module-oss.so(+0x184780)[0x7f9663a2a780]
/usr/lib/redis/modules/module-oss.so(+0x18482e)[0x7f9663a2a82e]
/usr/lib/redis/modules/module-oss.so(uv__io_poll+0x3d8)[0x7f9663a33ec8]
/usr/lib/redis/modules/module-oss.so(uv_run+0x105)[0x7f9663a2b095]
/usr/lib/redis/modules/module-oss.so(+0xdc83e)[0x7f966398283e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f9664a17fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f96649464cf]
------ REGISTERS ------
1:M 29 Sep 2021 15:42:03.296 #
RAX:0000000000001554 RBX:0000000000000000
RCX:0000000000004000 RDX:0000000000001554
RDI:00007f9663c07a70 RSI:0000000000000001
RBP:0000000e0000000e RSP:00007f966271a080
R8 :0000000000000064 R9 :000000770000007c
R10:00007f966271df28 R11:000000000000000f
R12:00007f9663c07a70 R13:00007f9663c06e70
R14:00007f9663c06ca0 R15:00007f9663c06ef0
RIP:00007f9663976f1b EFL:0000000000010202
CSGSFS:002b000000000033
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a08f) -> 00007f966271a110
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a08e) -> 00007f9663c06e88
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a08d) -> 0000000000000008
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a08c) -> 0000000000000000
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a08b) -> 00007f966271a0d0
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a08a) -> 00007f966271a0d0
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a089) -> 00007f9663a2a780
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a088) -> 00007f966271a0d0
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a087) -> 0000000000000000
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a086) -> 00007f9663c06ca0
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a085) -> 00007f966398353a
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a084) -> 000055731bb4b020
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a083) -> 00007f9663982b76
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a082) -> 000055731bb4b040
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a081) -> 00007f9644000df0
1:M 29 Sep 2021 15:42:03.296 # (00007f966271a080) -> 00007f9644000dc0
------ INFO OUTPUT ------
# Server
redis_version:6.2.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:49ac482504d79440
redis_mode:cluster
os:Linux 5.11.0-37-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.3.0
process_id:1
process_supervised:no
run_id:c890fe090286943e11655492564bfda4cd426143
tcp_port:7005
server_time_usec:1632930123292800
uptime_in_seconds:1
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5540171
executable:/data/redis-server
config_file:/usr/local/etc/redis/redis.conf
io_threads_active:0
# Clients
connected_clients:0
cluster_connections:10
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:8656792
used_memory_human:8.26M
used_memory_rss:25247744
used_memory_rss_human:24.08M
used_memory_peak:8717592
used_memory_peak_human:8.31M
used_memory_peak_perc:99.30%
used_memory_overhead:8581608
used_memory_startup:8581600
used_memory_dataset:75184
used_memory_dataset_perc:99.99%
allocator_allocated:9214520
allocator_active:9654272
allocator_resident:12881920
total_system_memory:50455261184
total_system_memory_human:46.99G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.05
allocator_frag_bytes:439752
allocator_rss_ratio:1.33
allocator_rss_bytes:3227648
rss_overhead_ratio:1.96
rss_overhead_bytes:12365824
mem_fragmentation_ratio:2.92
mem_fragmentation_bytes:16593264
mem_not_counted_for_evict:4
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:8
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1632930122
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:0
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:0
total_commands_processed:9
instantaneous_ops_per_sec:5
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:00a6a2df87069a322dbca240e4d45bc41d75b781
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.033866
used_cpu_user:0.071495
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.004111
used_cpu_user_main_thread:0.000000
# Modules
module:name=rg,ver=999999,api=1,filters=1,usedby=[rg],using=[rg],options=[]
module:name=search,ver=999999,api=1,filters=0,usedby=[],using=[ReJSON],options=[]
module:name=ReJSON,ver=999999,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
# Commandstats
cmdstat_cluster:calls=2,usec=116,usec_per_call=58.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=4,usec=35,usec_per_call=8.75,rejected_calls=0,failed_calls=0
cmdstat_search.CLUSTERREFRESH:calls=1,usec=354,usec_per_call=354.00,rejected_calls=0,failed_calls=0
cmdstat_config:calls=2,usec=37,usec_per_call=18.50,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:1
# Keyspace
------ CLIENT LIST OUTPUT ------
------ MODULES INFO OUTPUT ------
# rg
rg_nexecutions:0
rg_nregistrations:0
# rg_regisrations
# rg_plugins
rg_GearsPythonPlugin:version=10000
# rg_python_stats
rg_TotalAllocated:16234196
rg_PeakAllocated:6026602
rg_CurrAllocated:5957471
# rg_python_requirements
------ FAST MEMORY TEST ------
1:M 29 Sep 2021 15:42:03.298 # main thread terminated
1:M 29 Sep 2021 15:42:03.298 # Bio thread for job type #0 terminated
1:M 29 Sep 2021 15:42:03.298 # Bio thread for job type #1 terminated
1:M 29 Sep 2021 15:42:03.298 # Bio thread for job type #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/lib/redis/modules/module-oss.so (base 0x7f96638a6000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Redis crashes as soon as when trying to make a cluster with more than one server. There is no redisearch index or data in redis. Same issue happens with version 2.0.8.
To reproduce basically build
module-oss.sofor linux (in my case CentOS Linux 7). Create 3 master 3 slave cluster setup for redis and start redis server with the module (followed Creating and using a Redis Cluster from https://redis.io/topics/cluster-tutorial)redis-server ./redis.conf --loadmodule /usr/local/lib/module-oss.sothe first few will startup correctly but then some will crash and keep crashing on restart with the following error.Seems like if you create the redis cluster first without loading RSCoordinator module and then restart them and load module it will work, but after that adding any new machine to the cluster will crash with the same error.