search

RediSearch / RediSearch

A query and indexing engine for Redis, providing secondary indexing, full-text search, vector similarity search and aggregations.
https://redis.io/docs/stack/search/
Other
5.49k stars 521 forks source link

RediSearch crash on FT.DROP for (nonexisting?) index #2152

Open StrathCole opened 3 years ago

StrathCole commented 3 years ago

When sending an FT.DROP to a master node that might not have this index yes, the node crashes. The log was done with module-oss.so (RSCoordinator) but I tried the redisearch.so, too. The messages remain the same, no matter if using redis 6.0 or 6.2.5.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
14146:M 31 Jul 2021 16:00:59.235 # === ASSERTION FAILED ===
14146:M 31 Jul 2021 16:00:59.235 # ==> /tmp/RSCoordinator/src/dep/RediSearch/src/redis_index.c:472 'RedisModule_CallReplyType(rep) == REDISMODULE_REPLY_INTEGER' is not true

------ STACK TRACE ------

Backtrace:
/opt/redis-server/module-oss.so(+0x119ef4)[0x7fb64a23fef4]
/opt/redis-server/module-oss.so(+0x119f5d)[0x7fb64a23ff5d]
/opt/redis-server/module-oss.so(+0x10de7e)[0x7fb64a233e7e]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](RedisModuleCommandDispatcher+0x54)[0x561672d73894]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](call+0x9d)[0x561672cec9fd]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](processCommand+0x5b3)[0x561672ced863]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](processCommandAndResetClient+0x1c)[0x561672d01dec]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](processInputBuffer+0xea)[0x561672d044da]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](+0xf2e28)[0x561672d93e28]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](aeProcessEvents+0x2a1)[0x561672ce5a61]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](aeMain+0x1d)[0x561672ce5cbd]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](main+0x321)[0x561672ce2431]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7fb64b42809b]
/usr/bin/redis-server 10.0.0.3:7002 [cluster](_start+0x2a)[0x561672ce295a]

------ INFO OUTPUT ------
# Server
redis_version:6.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:bbd201c875e4b653
redis_mode:cluster
os:Linux 4.19.0-17-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.3.0
process_id:14146
process_supervised:systemd
run_id:b44a0adde871ed86ecc16165a9fd57b200b61637
tcp_port:7002
server_time_usec:1627740059234792
uptime_in_seconds:272
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:350107
executable:/usr/bin/redis-server
config_file:/etc/redis/redis-master.conf
io_threads_active:0

# Clients
connected_clients:48
cluster_connections:10
maxclients:10000
client_recent_max_input_buffer:104
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:508208328
used_memory_human:484.67M
used_memory_rss:531210240
used_memory_rss_human:506.60M
used_memory_peak:508264680
used_memory_peak_human:484.72M
used_memory_peak_perc:99.99%
used_memory_overhead:19448216
used_memory_startup:1561600
used_memory_dataset:488760112
used_memory_dataset_perc:96.47%
allocator_allocated:508492992
allocator_active:509739008
allocator_resident:524283904
total_system_memory:32938229760
total_system_memory_human:30.68G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:1246016
allocator_rss_ratio:1.03
allocator_rss_bytes:14544896
rss_overhead_ratio:1.01
rss_overhead_bytes:6926336
mem_fragmentation_ratio:1.05
mem_fragmentation_bytes:23127128
mem_not_counted_for_evict:890
mem_replication_backlog:1048576
mem_clients_slaves:20512
mem_clients_normal:963552
mem_aof_buffer:896
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:1519616
current_cow_size_age:2
current_fork_perc:31.27
current_save_keys_processed:91137
current_save_keys_total:291409
rdb_changes_since_last_save:16533
rdb_bgsave_in_progress:1
rdb_last_save_time:1627739922
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:3
rdb_current_bgsave_time_sec:3
rdb_last_cow_size:4263936
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:3
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:3649536
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:195158460
aof_base_size:194476104
aof_pending_rewrite:0
aof_buffer_length:225
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:148
total_commands_processed:4488
instantaneous_ops_per_sec:579
total_net_input_bytes:195171164
total_net_output_bytes:195493102
instantaneous_input_kbps:114.73
instantaneous_output_kbps:122.70
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_keys:11
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:22
evicted_keys:0
keyspace_hits:48383
keyspace_misses:9
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:16843
total_forks:3
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:4
dump_payload_sanitizations:0
total_reads_processed:3791
total_writes_processed:6932
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.4,port=7005,state=online,offset=142840835,lag=1
master_failover_state:no-failover
master_replid:a4ed8d8bdad59b3bc3f792353f0ce577638481ff
master_replid2:d9bfd8eac71486c47f928e0b5552a070fe4e67e4
master_repl_offset:142879932
second_repl_offset:142198348
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:142197147
repl_backlog_histlen:682786

# CPU
used_cpu_sys:2.579550
used_cpu_user:5.188567
used_cpu_sys_children:0.765847
used_cpu_user_children:5.345197
used_cpu_sys_main_thread:2.477669
used_cpu_user_main_thread:5.108679

# Modules
module:name=search,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat__FT.SEARCH:calls=1,usec=680,usec_per_call=680.00,rejected_calls=0,failed_calls=0
cmdstat_readonly:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_search.CLUSTERREFRESH:calls=267,usec=39976,usec_per_call=149.72,rejected_calls=0,failed_calls=0
cmdstat_get:calls=104,usec=1189,usec_per_call=11.43,rejected_calls=0,failed_calls=0
cmdstat_select:calls=1,usec=2,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_config:calls=1,usec=8,usec_per_call=8.00,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=22,usec=34,usec_per_call=1.55,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=12,usec=484,usec_per_call=40.33,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=1,usec=214,usec_per_call=214.00,rejected_calls=0,failed_calls=0
cmdstat_cluster:calls=629,usec=24056,usec_per_call=38.24,rejected_calls=3,failed_calls=0
cmdstat_info:calls=3,usec=26,usec_per_call=8.67,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=128,usec=310,usec_per_call=2.42,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=8,usec=5,usec_per_call=0.62,rejected_calls=0,failed_calls=0
cmdstat_del:calls=4,usec=141,usec_per_call=35.25,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=1,usec=17414,usec_per_call=17414.00,rejected_calls=0,failed_calls=0
cmdstat_FT.DROP:calls=1,usec=121,usec_per_call=121.00,rejected_calls=0,failed_calls=0
cmdstat_hmset:calls=3304,usec=297104,usec_per_call=89.92,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_LOADING:count=3
errorstat_index-livesearch-brands_:count=1

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=291406,expires=63,avg_ttl=5756589

------ CLIENT LIST OUTPUT ------
id=712 addr=10.0.0.4:36922 laddr=10.0.0.3:7002 fd=40 name= age=78 idle=78 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=720 addr=10.0.0.2:45576 laddr=10.0.0.3:7002 fd=42 name= age=76 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=721 addr=10.0.0.2:45578 laddr=10.0.0.3:7002 fd=71 name= age=76 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=722 addr=10.0.0.2:45582 laddr=10.0.0.3:7002 fd=72 name= age=76 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=723 addr=10.0.0.2:45584 laddr=10.0.0.3:7002 fd=73 name= age=76 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=324 addr=10.0.0.4:36386 laddr=10.0.0.3:7002 fd=27 name= age=188 idle=188 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=307 addr=10.0.0.2:44180 laddr=10.0.0.3:7002 fd=22 name= age=193 idle=193 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=461 addr=10.0.0.2:44946 laddr=10.0.0.3:7002 fd=23 name= age=148 idle=148 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=33 addr=10.0.0.3:46016 laddr=10.0.0.3:7002 fd=75 name= age=270 idle=270 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=34 addr=10.0.0.3:46018 laddr=10.0.0.3:7002 fd=76 name= age=270 idle=270 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=944 addr=10.0.0.2:46004 laddr=10.0.0.3:7002 fd=98 name= age=11 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=945 addr=10.0.0.2:46006 laddr=10.0.0.3:7002 fd=99 name= age=11 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=111 addr=10.0.0.2:43232 laddr=10.0.0.3:7002 fd=9 name= age=254 idle=254 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=35 addr=10.0.0.3:46020 laddr=10.0.0.3:7002 fd=77 name= age=270 idle=270 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=584 addr=10.0.0.2:45290 laddr=10.0.0.3:7002 fd=85 name= age=117 idle=117 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=585 addr=10.0.0.2:45292 laddr=10.0.0.3:7002 fd=87 name= age=117 idle=117 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=488 addr=10.0.0.4:37635 laddr=10.0.0.3:7002 fd=35 name= age=140 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=225 oll=0 omem=0 tot-mem=20512 events=r cmd=replconf user=default redir=-1
id=634 addr=10.0.0.2:45356 laddr=10.0.0.3:7002 fd=89 name= age=102 idle=102 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=489 addr=10.0.0.4:36520 laddr=10.0.0.3:7002 fd=37 name= age=140 idle=140 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=527 addr=10.0.0.4:36560 laddr=10.0.0.3:7002 fd=69 name= age=131 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=528 addr=10.0.0.4:36562 laddr=10.0.0.3:7002 fd=70 name= age=131 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=529 addr=10.0.0.4:36564 laddr=10.0.0.3:7002 fd=79 name= age=131 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=682 addr=10.0.0.2:45518 laddr=10.0.0.3:7002 fd=26 name= age=87 idle=87 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=683 addr=10.0.0.2:45520 laddr=10.0.0.3:7002 fd=20 name= age=87 idle=87 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=530 addr=10.0.0.4:36566 laddr=10.0.0.3:7002 fd=80 name= age=131 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=490 addr=10.0.0.4:36522 laddr=10.0.0.3:7002 fd=38 name= age=140 idle=140 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=521 addr=10.0.0.2:45108 laddr=10.0.0.3:7002 fd=46 name= age=133 idle=132 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=770 addr=10.0.0.4:36960 laddr=10.0.0.3:7002 fd=94 name= age=63 idle=63 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=976 addr=10.0.0.4:37120 laddr=10.0.0.3:7002 fd=103 name= age=2 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=771 addr=10.0.0.4:36962 laddr=10.0.0.3:7002 fd=95 name= age=63 idle=63 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=491 addr=10.0.0.4:36524 laddr=10.0.0.3:7002 fd=43 name= age=140 idle=140 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=492 addr=10.0.0.4:36526 laddr=10.0.0.3:7002 fd=44 name= age=140 idle=140 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=36 addr=10.0.0.3:46022 laddr=10.0.0.3:7002 fd=78 name= age=270 idle=270 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=849 addr=10.0.0.2:45856 laddr=10.0.0.3:7002 fd=97 name= age=42 idle=41 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=602 addr=10.0.0.4:36676 laddr=10.0.0.3:7002 fd=88 name= age=112 idle=112 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=654 addr=10.0.0.4:36706 laddr=10.0.0.3:7002 fd=90 name= age=97 idle=93 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=252 addr=10.0.0.2:43490 laddr=10.0.0.3:7002 fd=30 name= age=208 idle=208 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=29 addr=10.0.0.3:45992 laddr=10.0.0.3:7002 fd=61 name= age=271 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=_FT.SEARCH user=default redir=-1
id=798 addr=10.0.0.2:45798 laddr=10.0.0.3:7002 fd=96 name= age=57 idle=57 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=30 addr=10.0.0.3:45994 laddr=10.0.0.3:7002 fd=62 name= age=271 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=62 qbuf-free=40892 argv-mem=39 obl=0 oll=0 omem=0 tot-mem=61503 events=r cmd=_FT.DROP user=default redir=-1
id=31 addr=10.0.0.3:45996 laddr=10.0.0.3:7002 fd=64 name= age=271 idle=271 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=32 addr=10.0.0.3:45998 laddr=10.0.0.3:7002 fd=66 name= age=271 idle=271 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=10 addr=10.0.0.2:43034 laddr=10.0.0.3:7002 fd=31 name= age=272 idle=272 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=378 addr=10.0.0.4:36450 laddr=10.0.0.3:7002 fd=28 name= age=173 idle=173 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=11 addr=10.0.0.2:43038 laddr=10.0.0.3:7002 fd=32 name= age=272 idle=272 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=546 addr=10.0.0.4:36596 laddr=10.0.0.3:7002 fd=84 name= age=127 idle=127 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=setex user=default redir=-1
id=12 addr=10.0.0.2:43036 laddr=10.0.0.3:7002 fd=33 name= age=272 idle=272 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=13 addr=10.0.0.2:43040 laddr=10.0.0.3:7002 fd=34 name= age=272 idle=272 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1
id=984 addr=10.0.0.2:46054 laddr=10.0.0.3:7002 fd=100 name= age=0 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=38 obl=0 oll=0 omem=0 tot-mem=61502 events=r cmd=FT.DROP user=default redir=-1

------ CURRENT CLIENT INFO ------
id=30 addr=10.0.0.3:45994 laddr=10.0.0.3:7002 fd=62 name= age=271 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=62 qbuf-free=40892 argv-mem=39 obl=0 oll=0 omem=0 tot-mem=61503 events=r cmd=_FT.DROP user=default redir=-1
argv[0]: '_FT.DROP'
argv[1]: 'index-livesearch-countries-de'
argv[2]: 'DD'

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
14146:M 31 Jul 2021 16:00:59.238 # Bio thread for job type #0 terminated
14146:M 31 Jul 2021 16:00:59.239 # Bio thread for job type #1 terminated
14146:M 31 Jul 2021 16:00:59.239 # Bio thread for job type #2 terminated
*** Preparing to test memory region 561672e9e000 (2281472 bytes)
*** Preparing to test memory region 561675040000 (2441216 bytes)
*** Preparing to test memory region 7fb6081ff000 (8388608 bytes)
*** Preparing to test memory region 7fb6089ff000 (2097152 bytes)
*** Preparing to test memory region 7fb608c00000 (8388608 bytes)
*** Preparing to test memory region 7fb609400000 (583143424 bytes)
*** Preparing to test memory region 7fb630000000 (135168 bytes)
*** Preparing to test memory region 7fb634000000 (135168 bytes)
*** Preparing to test memory region 7fb638000000 (4194304 bytes)
*** Preparing to test memory region 7fb6385ff000 (13631488 bytes)
*** Preparing to test memory region 7fb639300000 (8388608 bytes)
*** Preparing to test memory region 7fb639b00000 (7340032 bytes)
*** Preparing to test memory region 7fb63a400000 (8388608 bytes)
*** Preparing to test memory region 7fb63ac00000 (4194304 bytes)
*** Preparing to test memory region 7fb63b200000 (8388608 bytes)
*** Preparing to test memory region 7fb63ba00000 (10485760 bytes)
*** Preparing to test memory region 7fb63c407000 (3670016 bytes)
*** Preparing to test memory region 7fb63c788000 (8388608 bytes)
*** Preparing to test memory region 7fb63cf89000 (8388608 bytes)
*** Preparing to test memory region 7fb63d78a000 (8388608 bytes)
*** Preparing to test memory region 7fb63df8b000 (8388608 bytes)
*** Preparing to test memory region 7fb63e78c000 (8388608 bytes)
*** Preparing to test memory region 7fb63ef8d000 (8388608 bytes)
*** Preparing to test memory region 7fb63f78e000 (8388608 bytes)
*** Preparing to test memory region 7fb63ff8f000 (8388608 bytes)
*** Preparing to test memory region 7fb640790000 (8388608 bytes)
*** Preparing to test memory region 7fb640f91000 (8388608 bytes)
*** Preparing to test memory region 7fb641792000 (8388608 bytes)
*** Preparing to test memory region 7fb641f93000 (8388608 bytes)
*** Preparing to test memory region 7fb642794000 (8388608 bytes)
*** Preparing to test memory region 7fb642f95000 (8388608 bytes)
*** Preparing to test memory region 7fb643796000 (8388608 bytes)
*** Preparing to test memory region 7fb643f97000 (8388608 bytes)
*** Preparing to test memory region 7fb644798000 (8388608 bytes)
*** Preparing to test memory region 7fb644f99000 (8388608 bytes)
*** Preparing to test memory region 7fb64579a000 (8388608 bytes)
*** Preparing to test memory region 7fb645f9b000 (8388608 bytes)
*** Preparing to test memory region 7fb64679c000 (8388608 bytes)
*** Preparing to test memory region 7fb646f9d000 (8388608 bytes)
*** Preparing to test memory region 7fb64779e000 (8388608 bytes)
*** Preparing to test memory region 7fb647f9f000 (8388608 bytes)
*** Preparing to test memory region 7fb6487a0000 (8388608 bytes)
*** Preparing to test memory region 7fb648fa1000 (8388608 bytes)
*** Preparing to test memory region 7fb6497a2000 (8388608 bytes)
*** Preparing to test memory region 7fb64a122000 (16384 bytes)
*** Preparing to test memory region 7fb64a485000 (12288 bytes)
*** Preparing to test memory region 7fb64a488000 (3145728 bytes)
*** Preparing to test memory region 7fb64aa00000 (8388608 bytes)
*** Preparing to test memory region 7fb64b274000 (24576 bytes)
*** Preparing to test memory region 7fb64b402000 (8192 bytes)
*** Preparing to test memory region 7fb64b5c1000 (16384 bytes)
*** Preparing to test memory region 7fb64b5e2000 (16384 bytes)
*** Preparing to test memory region 7fb64b686000 (4096 bytes)
*** Preparing to test memory region 7fb64b819000 (8192 bytes)
*** Preparing to test memory region 7fb64b84d000 (4096 bytes)
.O.O.O.O.O.15815:C 31 Jul 2021 16:01:00.110 * DB saved on disk
15815:C 31 Jul 2021 16:01:00.119 * RDB: 81 MB of memory used by copy-on-write
O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
ashtul commented 3 years ago

@StrathCole thank you for the issue. You are the master branch, correct? Can you please look at this branch where I've made a fix and confirm it eliminates the issue? Can you share more details about your topology? Can you send me the steps to reproduce?

ashtul commented 3 years ago

@StrathCole Do you have expiration or eviction policies for the indexes?

StrathCole commented 3 years ago

Hello, thanks for looking into this. I don't think there are special policies set, it is a "default" redis cluster.

I will have to set up a new "dummy cluster" for re-testing this because I cannot risk crashing the live-system currently. Sorry for the delay because of that.

StrathCole commented 3 years ago

Okay, I have tried to reproduce the behaviour. To send the commands I have used the RedisCluster class from phpredis with "rawCommand" method.

First I made a test with module-oss.so from the RSCoordinator github repository. It worked without problems. Then I tried using redisearch.so from the RediSearch github repository.

The 6 nodes all additionally listen on a unix socket (because we had some problems with tcp states on the live server lately, but as said it works correcly with the RSCoordinator branch).

$redis = new RedisCluster(null, ['127.0.0.1:7001', '127.0.0.1:7000', '127.0.0.1:7002', '127.0.0.1:7003', '127.0.0.1:7004', '127.0.0.1:7005']);
$redis->setOption(RedisCluster::OPT_SLAVE_FAILOVER, RedisCluster::FAILOVER_ERROR);

var_dump($redis->rawCommand('unix:///opt/redis/redis-7000.sock', 'FT.CREATE', 'testindex', 'ON', 'HASH', 'PREFIX', '1', 'xa_', 'STOPWORDS', '0', 'SCHEMA', 'n_id', 'TEXT', 'SORTABLE', 'n_name', 'TEXT', 'WEIGHT', '5.0', 'SORTABLE'));

This results in "true" so the command succeeded. But now there comes the strange part. Although the script connected to the node 7000, the index was only(!) created on node 7002 and its replication node:

root@redistest:/opt/redis# redis-cli -c -p 7000 FT._LIST
(empty array)
root@redistest:/opt/redis# redis-cli -c -p 7001 FT._LIST
(empty array)
root@redistest:/opt/redis# redis-cli -c -p 7002 FT._LIST
1) "testindex"
root@redistest:/opt/redis# redis-cli -c -p 7003 FT._LIST
1) "testindex"
root@redistest:/opt/redis# redis-cli -c -p 7004 FT._LIST
(empty array)
root@redistest:/opt/redis# redis-cli -c -p 7005 FT._LIST
(empty array)

When now using module-oss.so instead, the FT.CREATE creates the index on all nodes.

But in the test I could not reproduce the dying redis instance, the FT. functions just only worked on some targets and not on all. I then switched to module-oss.so and none* of the nodes showed the created index. But again, I could no longer make the instances crash.

My guess now is that there was a very uncommon situation on the live system that we first used redisearch.so and created indexes that did not work. When we switched to module-oss.so after it, the nodes crashed on FT.DROP. There was some notes in the log saying "missing index but key information exists" or similar. So I can only guess that I missed a step while trying to reproduce it on a fresh server now. :cry: Sorry.

svaraborut commented 2 years ago

While using https://github.com/redis/redis-py it turned out that .dropindex() uses FT.DROP which is not present in the documentation. Why does this command still work if there is FT.DROPINDEX?

Furthermore, the two commands have different behaviors, in fact, FT.DROP still crashes the server but FT.DROPINDEX does not. Is there a motivation behind this behavior?

Redis: redis_version:6.2.6 RediSearch module: 20406