search

RediSearch / RSCoordinator

RediSearch coordinator
Other
23 stars 10 forks source link

redis crash when aggregate query include limit 0 0 #255

Open geekyijun opened 3 years ago

geekyijun commented 3 years ago

1.The commands that caused the crash

2.The crash env includes the following:

  1. Full crash log

redis-server: /opt/cachecloud/module/RSCoordinator/src/dist_aggregate.c:308: RSExecDistAggregate: Assertion `QueryError_HasError(&status)' failed. spurious cond_wait wakeup spurious cond_wait wakeup spurious cond_wait wakeup spurious cond_wait wakeup spurious cond_wait wakeup spurious cond_wait wakeup spurious cond_wait wakeup spurious cond_wait wakeup

=== REDIS BUG REPORT START: Cut & paste starting from here === 135493:M 08 Apr 2021 09:47:56.531 # Redis 6.2.0 crashed by signal: 6, si_code: -6 135493:M 08 Apr 2021 09:47:56.531 # Killed by PID: 135493, UID: 400 135493:M 08 Apr 2021 09:47:56.531 # Crashed running the instruction at: 0x7fa894445277

------ STACK TRACE ------ EIP: /lib64/libc.so.6(gsignal+0x37)[0x7fa894445277]

Backtrace: /lib64/libpthread.so.0(+0xf6d0)[0x7fa8947eb6d0] /lib64/libc.so.6(gsignal+0x37)[0x7fa894445277] /lib64/libc.so.6(abort+0x148)[0x7fa894446968] /lib64/libc.so.6(+0x2f096)[0x7fa89443e096] /lib64/libc.so.6(+0x2f142)[0x7fa89443e142] /opt/cachecloud/module/module-oss.so(+0xb864a)[0x7fa89349564a] /opt/cachecloud/module/module-oss.so(+0xe1370)[0x7fa8934be370] /opt/cachecloud/module/module-oss.so(+0xe616e)[0x7fa8934c316e] /lib64/libpthread.so.0(+0x7e25)[0x7fa8947e3e25] /lib64/libc.so.6(clone+0x6d)[0x7fa89450dbad]

------ REGISTERS ------ 135493:M 08 Apr 2021 09:47:56.531 # RAX:0000000000000000 RBX:00007fa895324000 RCX:00007fa894445277 RDX:0000000000000006 RDI:0000000000021145 RSI:0000000000021153 RBP:00007fa894599580 RSP:00007fa88d6b2298 R8 :0000000000000001 R9 :0000000000020580 R10:0000000000000008 R11:0000000000000202 R12:00007fa8935590bb R13:00007fa893559150 R14:00007fa894017018 R15:0000000000000007 RIP:00007fa894445277 EFL:0000000000000202 CSGSFS:002b000000000033 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a7) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a6) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a5) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a4) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a3) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a2) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a1) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b22a0) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b229f) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b229e) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b229d) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b229c) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b229b) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b229a) -> 0000000000000000 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b2299) -> 0000000000000020 135493:M 08 Apr 2021 09:47:56.531 # (00007fa88d6b2298) -> 00007fa894446968

------ INFO OUTPUT ------

Server

redis_version:6.2.0 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:ab46c119497089db redis_mode:cluster os:Linux 4.9.99-31.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:7.3.0 process_id:135493 process_supervised:no run_id:ae5e81a117fbc82eaaf3fdcf7150eda55190c691 tcp_port:6460 server_time_usec:1617846476531186 uptime_in_seconds:52982 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:7233740 executable:/opt/cachecloud/redis-6.2.0/src/redis-server config_file:/opt/cachecloud/conf/xxx.207/redis-cluster-6460.conf io_threads_active:0

Clients

connected_clients:25 cluster_connections:10 maxclients:10000 client_recent_max_input_buffer:48 client_recent_max_output_buffer:0 blocked_clients:1 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:14397328 used_memory_human:13.73M used_memory_rss:12877824 used_memory_rss_human:12.28M used_memory_peak:14767576 used_memory_peak_human:14.08M used_memory_peak_perc:97.49% used_memory_overhead:12725512 used_memory_startup:1540816 used_memory_dataset:1671816 used_memory_dataset_perc:13.00% allocator_allocated:15522688 allocator_active:16887808 allocator_resident:26529792 total_system_memory:270126182400 total_system_memory_human:251.57G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:536870912 maxmemory_human:512.00M maxmemory_policy:volatile-lfu allocator_frag_ratio:1.09 allocator_frag_bytes:1365120 allocator_rss_ratio:1.57 allocator_rss_bytes:9641984 rss_overhead_ratio:0.49 rss_overhead_bytes:-13651968 mem_fragmentation_ratio:0.90 mem_fragmentation_bytes:-1433200 mem_not_counted_for_evict:4 mem_replication_backlog:10485760 mem_clients_slaves:20512 mem_clients_normal:512648 mem_aof_buffer:8 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0

Persistence

loading:0 current_cow_size:0 current_fork_perc:0.00% current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1617793495 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:671744 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0 aof_current_size:404441 aof_base_size:404441 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0

Stats

total_connections_received:4106 total_commands_processed:224066 instantaneous_ops_per_sec:4 total_net_input_bytes:2554069 total_net_output_bytes:6290221 instantaneous_input_kbps:0.05 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:1 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:310 evicted_keys:0 keyspace_hits:3856 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:783 total_forks:2 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:18 dump_payload_sanitizations:0 total_reads_processed:69238 total_writes_processed:17557 io_threaded_reads_processed:0 io_threaded_writes_processed:0

Replication

role:master connected_slaves:1 slave0:ip=xxx.207,port=7460,state=online,offset=74074,lag=1 master_failover_state:no-failover master_replid:f8e56b215301ef6dcd3e168340ca58f62317e2e0 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:74074 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:10000000 repl_backlog_first_byte_offset:1 repl_backlog_histlen:74074

CPU

used_cpu_sys:79.306066 used_cpu_user:85.649662 used_cpu_sys_children:0.003993 used_cpu_user_children:0.004216 used_cpu_sys_main_thread:0.000000 used_cpu_user_main_thread:0.000863

Modules

module:name=search,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]

Commandstats

cmdstat_cluster:calls=106848,usec=4392740,usec_per_call=41.11,rejected_calls=0,failed_calls=0 cmdstathost:calls=0,usec=0,usec_per_call=0.00,rejected_calls=1,failed_calls=0 cmdstat_info:calls=1125,usec=86313,usec_per_call=76.72,rejected_calls=0,failed_calls=0 cmdstat_latency:calls=882,usec=950,usec_per_call=1.08,rejected_calls=0,failed_calls=0 cmdstat_client:calls=2,usec=234,usec_per_call=117.00,rejected_calls=0,failed_calls=0 cmdstat_ping:calls=238,usec=79,usec_per_call=0.33,rejected_calls=1,failed_calls=0 cmdstat_search.CLUSTERREFRESH:calls=52972,usec=5656106,usec_per_call=106.78,rejected_calls=0,failed_calls=0 cmdstat_slowlog:calls=44,usec=81,usec_per_call=1.84,rejected_calls=0,failed_calls=0 cmdstat_FT.CURSOR:calls=3,usec=33,usec_per_call=11.00,rejected_calls=0,failed_calls=3 cmdstat_psync:calls=1,usec=705,usec_per_call=705.00,rejected_calls=0,failed_calls=0 cmdstat_get:calls=0,usec=0,usec_per_call=0.00,rejected_calls=1,failed_calls=0 cmdstat_auth:calls=8165,usec=18445,usec_per_call=2.26,rejected_calls=0,failed_calls=1 cmdstat__FT.AGGREGATE:calls=0,usec=0,usec_per_call=0.00,rejected_calls=8,failed_calls=0 cmdstat_config:calls=883,usec=8962,usec_per_call=10.15,rejected_calls=0,failed_calls=0 cmdstat_FT.AGGREGATE:calls=9,usec=376,usec_per_call=41.78,rejected_calls=0,failed_calls=0 cmdstat_command:calls=1,usec=765,usec_per_call=765.00,rejected_calls=0,failed_calls=0 cmdstat_replconf:calls=52891,usec=39936,usec_per_call=0.76,rejected_calls=0,failed_calls=0

Errorstats

errorstat_Cursor:count=3 errorstat_ERR:count=4 errorstat_NOAUTH:count=10 errorstat_WRONGPASS:count=1

Cluster

cluster_enabled:1

Keyspace

db0:keys=3325,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=34 addr=xxx.207:45020 laddr=xxx.207:6460 fd=39 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=19 addr=xxx.207:44966 laddr=xxx.207:6460 fd=18 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=20 addr=xxx.207:44968 laddr=xxx.207:6460 fd=19 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=21 addr=xxx.207:44970 laddr=xxx.207:6460 fd=20 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=22 addr=xxx.207:44972 laddr=xxx.207:6460 fd=21 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=23 addr=xxx.207:44982 laddr=xxx.207:6460 fd=28 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=24 addr=xxx.207:44984 laddr=xxx.207:6460 fd=29 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=41 addr=xxx.207:45142 laddr=xxx.207:6460 fd=50 name= age=52981 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=_FT.AGGREGATE user=default redir=-1 id=42 addr=xxx.207:45144 laddr=xxx.207:6460 fd=52 name= age=52981 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=_FT.AGGREGATE user=default redir=-1 id=43 addr=xxx.207:45146 laddr=xxx.207:6460 fd=54 name= age=52981 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=_FT.AGGREGATE user=default redir=-1 id=44 addr=xxx.207:45148 laddr=xxx.207:6460 fd=56 name= age=52981 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=_FT.AGGREGATE user=default redir=-1 id=382 addr=xxx.207:56996 laddr=xxx.207:6460 fd=71 name= age=52906 idle=52906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1 id=383 addr=xxx.207:56998 laddr=xxx.207:6460 fd=72 name= age=52906 idle=52906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1 id=384 addr=xxx.207:57000 laddr=xxx.207:6460 fd=73 name= age=52906 idle=52906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1 id=385 addr=xxx.207:57002 laddr=xxx.207:6460 fd=74 name= age=52906 idle=52906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default redir=-1 id=36 addr=xxx.207:41775 laddr=xxx.207:6460 fd=40 name= age=52981 idle=1 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=replconf user=default redir=-1 id=25 addr=xxx.207:44986 laddr=xxx.207:6460 fd=30 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=26 addr=xxx.207:44988 laddr=xxx.207:6460 fd=31 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=27 addr=xxx.207:44998 laddr=xxx.207:6460 fd=32 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=28 addr=xxx.207:45000 laddr=xxx.207:6460 fd=33 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=29 addr=xxx.207:45002 laddr=xxx.207:6460 fd=34 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=30 addr=xxx.207:45004 laddr=xxx.207:6460 fd=35 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=215740 addr=10.7.36.148:53180 laddr=xxx.207:6460 fd=75 name= age=67 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=37 obl=0 oll=0 omem=0 tot-mem=61533 events=r cmd=FT.AGGREGATE user=default redir=-1 id=31 addr=xxx.207:45014 laddr=xxx.207:6460 fd=36 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=32 addr=xxx.207:45016 laddr=xxx.207:6460 fd=37 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1 id=33 addr=xxx.207:45018 laddr=xxx.207:6460 fd=38 name= age=52982 idle=52982 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------ 135493:M 08 Apr 2021 09:47:56.532 # main thread terminated 135493:M 08 Apr 2021 09:47:56.532 # Bio thread for job type #0 terminated 135493:M 08 Apr 2021 09:47:56.532 # Bio thread for job type #1 terminated 135493:M 08 Apr 2021 09:47:56.532 # Bio thread for job type #2 terminated 135493:M 08 Apr 2021 09:47:56.532 # IO thread(tid:140361777960704) terminated 135493:M 08 Apr 2021 09:47:56.532 # IO thread(tid:140361769568000) terminated 135493:M 08 Apr 2021 09:47:56.532 # IO thread(tid:140361761175296) terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: gsignal (base: 0x7fa894445240) Module: /lib64/libc.so.6 (base 0x7fa89440f000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x7fa894445240 -D -b binary -m i386:x86-64 /tmp/dump.bin

135493:M 08 Apr 2021 09:47:56.532 # dump of function (hexdump of 183 bytes): 648b0c25d4020000648b3425d002000085f67534b8ba0000000f0589c164890425d002000089c64863d74863f64863f9b8ea0000000f05483d00f0ffff771ef3c30f1f800000000085c97fdb89c8f7d881e1ffffff7f0f44c689c1ebca488b15ccfb3800f7d86489024883c8ffc3669085ff780cf7dfe9850200000f1f440000488b05a9fb380064c70016000000b8ffffffffc3662e0f1f8400000000006690900f1f440000662e0f1f84000000000048c7c00f000000

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/redis/redis/issues

Suspect RAM error? Use redis-server --test-memory to verify it.



What is the reason for this, thanks.
ashtul commented 3 years ago

@geekyijun Thank you for the issue. It seems your RediSearch and RSCoordinator versions are not aligned which should not happen. How are you running it? Did you build it from the repo or got a docker?

geekyijun commented 3 years ago

@ashtul thank you for your reply I switched rediSearch and RSCoordinator to version 2.0.6 and run them once.

build from the repo :

git clone https://github.com/RediSearch/RSCoordinator.git
cd RSCoordinator && git checkout v2.0.6
git clone https://github.com/RediSearch/RediSearch.git
cd RediSearch && git checkout v2.0.6
cp -rf RediSearch RSCoordinator/src/dep/
cd RSCoordinator
mkdir build
python configure.py
cd build
make 
// get module-oss.so

but , The crash problem still reappears。

the log:

198050:M 08 Apr 2021 15:32:08.846 # Server initialized
RSValue size: 16
198050:M 08 Apr 2021 15:32:08.848 * <search> redis version observed by redisearch : 6.2.0
198050:M 08 Apr 2021 15:32:08.848 * <search> Redis version found by RedisSearch : 6.2.0 - oss
198050:M 08 Apr 2021 15:32:08.848 * <search> RediSearch version 2.0.6 (Git=v2.0.4-55-gb0db8b2)
198050:M 08 Apr 2021 15:32:08.848 * <search> Low level api version 1 initialized successfully
198050:M 08 Apr 2021 15:32:08.848 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  1000000, search pool size: 20, index pool size: 8,
198050:M 08 Apr 2021 15:32:08.848 * <search> Initialized thread pool!
198050:M 08 Apr 2021 15:32:08.848 * <search> Cluster configuration: 0 partitions, type: 0, coordinator timeout: 0ms
Creating thread...
Thread created
198050:M 08 Apr 2021 15:32:08.849 * <search> Register write commands
198050:M 08 Apr 2021 15:32:08.849 * Module 'search' loaded from /opt/cachecloud/module/module-oss-2.0.6.so
...

//calling FT.AGGREGATE myIndex * limit 0 0

redis-server: /opt/cachecloud/soft/RSCoordinator/src/dist_aggregate.c:308: RSExecDistAggregate: Assertion `QueryError_HasError(&status)' failed.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
198050:S 08 Apr 2021 15:33:11.748 # Redis 6.2.0 crashed by signal: 6, si_code: -6
198050:S 08 Apr 2021 15:33:11.748 # Killed by PID: 198050, UID: 400
198050:S 08 Apr 2021 15:33:11.748 # Crashed running the instruction at: 0x7febbedca277

------ STACK TRACE ------
EIP:
/lib64/libc.so.6(gsignal+0x37)[0x7febbedca277]

Backtrace:
/lib64/libpthread.so.0(+0xf6d0)[0x7febbf1706d0]
/lib64/libc.so.6(gsignal+0x37)[0x7febbedca277]
/lib64/libc.so.6(abort+0x148)[0x7febbedcb968]
/lib64/libc.so.6(+0x2f096)[0x7febbedc3096]
/lib64/libc.so.6(+0x2f142)[0x7febbedc3142]
/opt/cachecloud/module/module-oss-2.0.6.so(+0xb82ca)[0x7febbdc992ca]
/opt/cachecloud/module/module-oss-2.0.6.so(+0xe07e0)[0x7febbdcc17e0]
/opt/cachecloud/module/module-oss-2.0.6.so(+0xe51ce)[0x7febbdcc61ce]
/lib64/libpthread.so.0(+0x7e25)[0x7febbf168e25]
/lib64/libc.so.6(clone+0x6d)[0x7febbee92bad]

------ REGISTERS ------
198050:S 08 Apr 2021 15:33:11.748 #
RAX:0000000000000000 RBX:00007febbfca9000
RCX:00007febbedca277 RDX:0000000000000006
RDI:00000000000305a2 RSI:00000000000305a8
RBP:00007febbef1e580 RSP:00007febbb6bd298
R8 :0000000000000001 R9 :00000000000205a0
R10:0000000000000008 R11:0000000000000202
R12:00007febbdd5a0db R13:00007febbdd5a180
R14:00007febbe8168c0 R15:0000000000000006
RIP:00007febbedca277 EFL:0000000000000202
CSGSFS:002b000000000033
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a7) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a6) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a5) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a4) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a3) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a2) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a1) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd2a0) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd29f) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd29e) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd29d) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd29c) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd29b) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd29a) -> 0000000000000000
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd299) -> 0000000000000020
198050:S 08 Apr 2021 15:33:11.748 # (00007febbb6bd298) -> 00007febbedcb968

------ INFO OUTPUT ------
# Server
redis_version:6.2.0
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:ab46c119497089db
redis_mode:cluster
os:Linux 4.9.99-31.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:7.3.0
process_id:198050
process_supervised:no
run_id:a0b8da5f84b5066a5f6725baddc88ad34abec769
tcp_port:6460
server_time_usec:1617867191748280
uptime_in_seconds:63
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7254455
executable:/opt/cachecloud/redis-6.2.0/src/redis-server
config_file:/opt/cachecloud/conf/10.19.123.207/redis-cluster-6460.conf
io_threads_active:0
...
ashtul commented 3 years ago

@geekyijun Can you please send a set of commands which will reproduce the crush?

MeirShpilraien commented 3 years ago

Fix: https://github.com/RediSearch/RediSearch/pull/2350