Closed ecamaj closed 1 year ago
bing openai gives weird explanations about the size of vector: _The problem here is that Redis crashed with a segmentation fault while executing the FP32_InnerProductSIMD16Ext_SSEimpl function, which is part of the RediSearch module. This function is used to compute the inner product of two vectors using SSE instructions. According to this GitHub issue1, this is a known bug that affects RediSearch when the vector size is not equal to 128, which is the default vector size for RediSearch. The bug causes Redis to access invalid memory addresses and crash.
here is also starting logs:
9:C 11 Jul 2023 17:21:22.951 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9:C 11 Jul 2023 17:21:22.952 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=9, just started
9:C 11 Jul 2023 17:21:22.952 # Configuration loaded
9:M 11 Jul 2023 17:21:22.955 monotonic clock: POSIX clock_gettime
9:M 11 Jul 2023 17:21:22.957 Running mode=standalone, port=6379.
9:M 11 Jul 2023 17:21:22.957 # Server initialized
9:M 11 Jul 2023 17:21:22.958 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
9:M 11 Jul 2023 17:21:22.968
This is a Redis issue. Please file it to http://github.com/redis/redis/issues as per their guidance.
` ------ STACK TRACE ------ EIP: /opt/redis-stack/lib/redisearch.so(_Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m+0xd)[0x7f977b03ec5d]
Backtrace: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f977bccf420] /opt/redis-stack/bin/redis-server :6379(invalidFunctionWasCalled+0x0)[0x55a76501b490] /opt/redis-stack/lib/redisearch.so(_Z30FP32_InnerProductSIMD16Ext_SSEPKvS0_m+0xd)[0x7f977b03eced] /opt/redis-stack/lib/redisearch.so(_ZNK9HNSWIndexIffE11searchLayerEjPKvmm+0x13a)[0x7f977b02fb2a] /opt/redis-stack/lib/redisearch.so(_ZN9HNSWIndexIffE12appendVectorEPKvm+0x471)[0x7f977b0314d1] /opt/redis-stack/lib/redisearch.so(VecSimIndex_AddVector+0x32)[0x7f977b007302] /opt/redis-stack/lib/redisearch.so(IndexerBulkAdd+0x137)[0x7f977af71ab7] /opt/redis-stack/lib/redisearch.so(+0xdf262)[0x7f977af89262] /opt/redis-stack/lib/redisearch.so(+0xdf683)[0x7f977af89683] /opt/redis-stack/lib/redisearch.so(Indexer_Add+0x5d)[0x7f977af8a24d] /opt/redis-stack/lib/redisearch.so(Document_AddToIndexes+0xf1)[0x7f977af71df1] /opt/redis-stack/lib/redisearch.so(IndexSpec_UpdateDoc+0x168)[0x7f977afbb3a8] /opt/redis-stack/lib/redisearch.so(Indexes_UpdateMatchingWithSchemaRules+0xd6)[0x7f977afbbb06] /opt/redis-stack/lib/redisearch.so(HashNotificationCallback+0x138)[0x7f977af96438] /opt/redis-stack/bin/redis-server :6379(moduleNotifyKeyspaceEvent+0xfc)[0x55a7650587ec] /opt/redis-stack/bin/redis-server :6379(notifyKeyspaceEvent+0x35)[0x55a765044775] /opt/redis-stack/bin/redis-server :6379(hsetCommand+0xfd)[0x55a76500e6ad] /opt/redis-stack/bin/redis-server :6379(call+0xf0)[0x55a764fc7940] /opt/redis-stack/bin/redis-server :6379(processCommand+0x643)[0x55a764fc9643] /opt/redis-stack/bin/redis-server :6379(processCommandAndResetClient+0x20)[0x55a764fdcd20] /opt/redis-stack/bin/redis-server :6379(processInputBuffer+0xea)[0x55a764fdf67a] /opt/redis-stack/bin/redis-server :6379(+0x1018ac)[0x55a7650788ac] /opt/redis-stack/bin/redis-server :6379(aeProcessEvents+0x2ca)[0x55a764fbfe6a] /opt/redis-stack/bin/redis-server :6379(aeMain+0x1d)[0x55a764fc00fd] /opt/redis-stack/bin/redis-server :6379(main+0x33a)[0x55a764fbc50a] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f977baed083] /opt/redis-stack/bin/redis-server *:6379(_start+0x2e)[0x55a764fbca0e]
------ REGISTERS ------ 9:M 11 Jul 2023 17:02:29.431 # RAX:00007ffea41b6810 RBX:00007f977b790388 RCX:000055a76734a1c8 RDX:0000000000000600 RDI:00007ffea41b5010 RSI:00007f97709ff354 RBP:00007ffea41b4f78 RSP:00007ffea41b4ed8 R8 :0000000000000000 R9 :00000000000000c8 R10:000000000000000c R11:00007f9770a00be8 R12:00007ffea41b69a0 R13:0000000000000000 R14:00007ffea41b69b8 R15:0000000000000001 RIP:00007f977b03ec5d EFL:0000000000000287 CSGSFS:002b000000000033 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee7) -> 00007f977b761f00 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee6) -> 000055a764fd183f 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee5) -> 00007f977b784920 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee4) -> 0000000000000000 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee3) -> 0000000000000010 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee2) -> 00007ffea41b4f60 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee1) -> 00007ffea41b4f50 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ee0) -> 0000000000000000 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4edf) -> 00007ffea41b5010 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ede) -> 00000000000000c8 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4edd) -> 0000000000000000 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4edc) -> 0000000000000000 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4edb) -> 0000000000000000 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4eda) -> 00007f977b02fb2a 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ed9) -> 00007ffea41b4f78 9:M 11 Jul 2023 17:02:29.431 # (00007ffea41b4ed8) -> 00007f977b03eced
------ INFO OUTPUT ------
Server
redis_version:6.2.12 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:4dbc2487343b0024 redis_mode:standalone os:Linux 5.15.82-0-virt x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:9.4.0 process_id:9 process_supervised:no run_id:4c42b8b19c205682654832a2034fe0bc0188e6d9 tcp_port:6379 server_time_usec:1689094949388361 uptime_in_seconds:320 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:11373349 executable:/opt/redis-stack/bin/redis-server config_file: io_threads_active:0
Clients
connected_clients:5 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:264 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0
Memory
used_memory:8923792 used_memory_human:8.51M used_memory_rss:22589440 used_memory_rss_human:21.54M used_memory_peak:8923792 used_memory_peak_human:8.51M used_memory_peak_perc:100.34% used_memory_overhead:1117384 used_memory_startup:1014368 used_memory_dataset:7806408 used_memory_dataset_perc:98.70% allocator_allocated:2110880 allocator_active:2453504 allocator_resident:4857856 total_system_memory:8347402240 total_system_memory_human:7.77G used_memory_lua:30720 used_memory_lua_human:30.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.16 allocator_frag_bytes:342624 allocator_rss_ratio:1.98 allocator_rss_bytes:2404352 rss_overhead_ratio:4.65 rss_overhead_bytes:17731584 mem_fragmentation_ratio:16.62 mem_fragmentation_bytes:21230432 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:102904 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0
Persistence
loading:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:5 rdb_bgsave_in_progress:0 rdb_last_save_time:1689094629 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0
Stats
total_connections_received:9 total_commands_processed:54 instantaneous_ops_per_sec:0 total_net_input_bytes:39345 total_net_output_bytes:150589 instantaneous_input_kbps:0.01 instantaneous_output_kbps:2.85 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:111 evicted_keys:0 keyspace_hits:2 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 total_forks:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:4 dump_payload_sanitizations:0 total_reads_processed:57 total_writes_processed:52 io_threaded_reads_processed:0 io_threaded_writes_processed:0
Replication
role:master connected_slaves:0 master_failover_state:no-failover master_replid:847f2c39490bae91fa454fab13e9c18864c1835c master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0
CPU
used_cpu_sys:8.133421 used_cpu_user:5.184259 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000 used_cpu_sys_main_thread:7.848980 used_cpu_user_main_thread:5.137514
Modules
module:name=search,ver=20609,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors] module:name=ReJSON,ver=20407,api=1,filters=0,usedby=[search|graph],using=[],options=[handle-io-errors] module:name=timeseries,ver=10810,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors] module:name=graph,ver=21010,api=1,filters=0,usedby=[],using=[ReJSON],options=[] module:name=bf,ver=20405,api=1,filters=0,usedby=[],using=[],options=[]
Commandstats
cmdstat_FT.CREATE:calls=3,usec=17133,usec_per_call=5711.00,rejected_calls=0,failed_calls=1 cmdstat_ping:calls=3,usec=230,usec_per_call=76.67,rejected_calls=0,failed_calls=0 cmdstat_module:calls=6,usec=1584,usec_per_call=264.00,rejected_calls=0,failed_calls=0 cmdstat_hset:calls=1,usec=29289,usec_per_call=29289.00,rejected_calls=0,failed_calls=0 cmdstat_info:calls=29,usec=80891,usec_per_call=2789.34,rejected_calls=0,failed_calls=0 cmdstat_FT.INFO:calls=11,usec=10437,usec_per_call=948.82,rejected_calls=0,failed_calls=3 cmdstat_FT.SEARCH:calls=1,usec=11811,usec_per_call=11811.00,rejected_calls=0,failed_calls=0
Errorstats
errorstat_Index:count=1 errorstat_Unknown:count=3
Cluster
cluster_enabled:0
Keyspace
db0:keys=2,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------ id=17 addr=172.17.0.1:34970 laddr=172.17.0.2:6379 fd=13 name= age=300 idle=300 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=FT.INFO user=default redir=-1 id=19 addr=172.17.0.1:34982 laddr=172.17.0.2:6379 fd=14 name= age=295 idle=294 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20648 events=r cmd=FT.SEARCH user=default redir=-1 id=22 addr=172.17.0.1:37402 laddr=172.17.0.2:6379 fd=15 name= age=270 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=22334 qbuf-free=18620 argv-mem=7426 obl=4 oll=0 omem=0 tot-mem=68930 events=r cmd=hset user=default redir=-1 id=33 addr=172.17.0.1:39332 laddr=172.17.0.2:6379 fd=16 name= age=121 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=info user=default redir=-1 id=11 addr=172.17.0.1:39886 laddr=172.17.0.2:6379 fd=12 name= age=312 idle=312 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20744 events=r cmd=FT.CREATE user=default redir=-1
------ CURRENT CLIENT INFO ------ id=22 addr=172.17.0.1:37402 laddr=172.17.0.2:6379 fd=15 name= age=270 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=22334 qbuf-free=18620 argv-mem=7426 obl=4 oll=0 omem=0 tot-mem=68930 events=r cmd=hset user=default redir=-1 argv[0]: 'HSET' argv[1]: 'doc:embeddings:6f3acf2ae6f80fa3e9828bab1a234d5d9f8515b1' argv[2]: 'content' argv[3]: '...some text...' argv[4]: 'content_vector' argv[5]: '�q�<:'����V�*ּȍ������W����:�����iؼ�5w<;?�;�y6�FC̺%�!�O�E<�=5<̗�<�i����ѼW/<����� �!b麊��<A�ڻH�����3�.u$���l<.u$�v֬;�$�����<�5�;�H�;��L< �:� :�o�����S��=<ͯ <���<9��:��[<�f���?���eF<�%�<v@Y��'m��0�<ul' =.��<`m5�2�»t���[ argv[6]: 'metadata' argv[7]: '{"source": "https://mydeployment.blob.core.windows.net/documents/keks2.txt", "chunk": 1, "key": "doc:embeddings:6f3acf2ae6f80fa3e9828bab1a234d5d9f8515b1", "filename": "keks2.txt"}' 9:M 11 Jul 2023 17:02:29.433 # key 'doc:embeddings:6f3acf2ae6f80fa3e9828bab1a234d5d9f8515b1' found in DB containing the following object: 9:M 11 Jul 2023 17:02:29.433 # Object type: 4 9:M 11 Jul 2023 17:02:29.434 # Object encoding: 2 9:M 11 Jul 2023 17:02:29.434 # Object refcount: 1
------ MODULES INFO OUTPUT ------
search_version
search_version:2.6.9 search_redis_version:6.2.12 - oss
search_index
search_number_of_indexes:2
search_fields_statistics
search_fields_text:Text=5 search_fields_vector:Vector=1,HSNW=1
search_dialect_statistics
search_dialect_1:0 search_dialect_2:1 search_dialect_3:0
search_runtime_configurations
search_concurrent_mode:OFF search_enableGC:ON search_minimal_term_prefix:2 search_maximal_prefix_expansions:200 search_query_timeout_ms:500 search_timeout_policy:return search_cursor_read_size:1000 search_cursor_max_idle_time:300000 search_max_doc_table_size:1000000 search_max_search_results:10000 search_max_aggregate_results:10000 search_search_pool_size:20 search_index_pool_size:8 search_gc_scan_size:100 search_min_phonetic_term_length:3
ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func 1: rejson::info_func 2: modulesCollectInfo at /w/redis-stack/redis-stack/redis/src/module.c:7100:9 3: logModulesInfo at /w/redis-stack/redis-stack/redis/src/debug.c:1624:22 4: printCrashReport at /w/redis-stack/redis-stack/redis/src/debug.c:1896:5 5: sigsegvHandler at /w/redis-stack/redis-stack/redis/src/debug.c:1878:5 6:
7: _Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m
8: _Z30FP32_InnerProductSIMD16Ext_SSEPKvS0_m
9: _ZNK9HNSWIndexIffE11searchLayerEjPKvmm
10: _ZN9HNSWIndexIffE12appendVectorEPKvm
11: VecSimIndex_AddVector
12: IndexerBulkAdd
13: indexBulkFields
14: Indexer_Process
15: Indexer_Add
16: Document_AddToIndexes
17: IndexSpec_UpdateDoc
18: Indexes_UpdateMatchingWithSchemaRules
19: HashNotificationCallback
20: moduleNotifyKeyspaceEvent
at / w/redis-stack/redis-stack/redis/src/module.c:5983:13
21: notifyKeyspaceEvent
at /w/redis-stack/redis-stack/redis/src/notify.c:111:6
22: hsetCommand
at /w/redis-stack/redis-stack/redis/src/t_hash.c:684:5
23: call
at /w/redis-stack/redis-stack/redis/src/server.c:3750:5
24: processCommand
at /w/redis-stack/redis-stack/redis/src/server.c:4297:9
25: processCommandAndResetClient
at /w/redis-stack/redis-stack/redis/src/networking.c:2105:9
26: processInputBuffer
at /w/redis-stack/redis-stack/redis/src/networking.c:2206:17
27: callHandler
at /w/redis-stack/redis-stack/redis/src/connhelpers.h:79:18
connSocketEventHandler
at /w/redis-stack/redis-stack/redis/src/connection.c:295:14
28: aeProcessEvents
at /w/redis-stack/redis-stack/redis/src/ae.c:427:17
29: aeMain
at /w/redis-stack/redis-stack/redis/src/ae.c:487:9
30: main
at /w/redis-stack/redis-stack/redis/src/server.c:6474:5
31: libc_start_main
32: _start
graph_executing commands
------ FAST MEMORY TEST ------ 9:M 11 Jul 2023 17:02:29.607 # Bio thread for job type #0 terminated 9:M 11 Jul 2023 17:02:29.609 # Bio thread for job type #1 terminated 9:M 11 Jul 2023 17:02:29.610 # Bio thread for job type #2 terminated Preparing to test memory region 55a765190000 (2281472 bytes) Preparing to test memory region 55a767305000 (405504 bytes) Preparing to test memory region 7f976a9ae000 (10485760 bytes) Preparing to test memory region 7f976c000000 (135168 bytes) Preparing to test memory region 7f97709ff000 (8388608 bytes) Preparing to test memory region 7f9771200000 (8388608 bytes) Preparing to test memory region 7f9771a00000 (4194304 bytes) Preparing to test memory region 7f9771fbc000 (2621440 bytes) Preparing to test memory region 7f977223d000 (8388608 bytes) Preparing to test memory region 7f9772a3e000 (8388608 bytes) Preparing to test memory region 7f977323f000 (8388608 bytes) Preparing to test memory region 7f9773a40000 (8388608 bytes) Preparing to test memory region 7f97744fc000 (8388608 bytes) Preparing to test memory region 7f9774cfd000 (8388608 bytes) Preparing to test memory region 7f977558c000 (4096 bytes) Preparing to test memory region 7f977558e000 (8388608 bytes) Preparing to test memory region 7f9775d8f000 (8388608 bytes) Preparing to test memory region 7f9776590000 (8388608 bytes) Preparing to test memory region 7f9776d91000 (8388608 bytes) Preparing to test memory region 7f9777592000 (8388608 bytes) Preparing to test memory region 7f9777d93000 (8388608 bytes) Preparing to test memory region 7f9779cc3000 (12288 bytes) Preparing to test memory region 7f9779cc7000 (8388608 bytes) Preparing to test memory region 7f977a4c8000 (8388608 bytes) Preparing to test memory region 7f977aea7000 (12288 bytes) Preparing to test memory region 7f977b1fd000 (12288 bytes) Preparing to test memory region 7f977b200000 (8388608 bytes) Preparing to test memory region 7f977bac3000 (24576 bytes) Preparing to test memory region 7f977bcb7000 (16384 bytes) Preparing to test memory region 7f977bcda000 (16384 bytes) Preparing to test memory region 7f977bfb1000 (16384 bytes) Preparing to test memory region 7f977c19d000 (8192 bytes) *** Preparing to test memory region 7f977c1d0000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------ Symbol: _Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m (base: 0x7f977b03ec50) Module: /opt/redis-stack/lib/redisearch.so (base 0x7f977aeaa000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x7f977b03ec50 -D -b binary -m i386:x86-64 /tmp/dump.bin
9:M 11 Jul 2023 17:02:30.886 # dump of function (hexdump of 141 bytes): f30f1efa488d04974839c77373c5f057c90f1f8000000000c5f8101fc5e059064883c7404883c640c5f81067d0c5f8106fe0c5f81056f0c5f858c1c5d8594ed0c5f858c1c5d0594ee0c5f858c1c5f8104ff0c5f059cac5f058c84839f877b9c5f0c6c155c5fa58c1c5f015d1c5fa58d2c5f0c6c1ffc5ea58c0c3660f1f440000c5f857c0c366662e0f1f840000
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
If a Redis module was involved, please open in the module's repo instead.
Suspect RAM error? Use redis-server --test-memory to verify it.
Some other issues could be detected by redis-server --check-system Illegal instruction (core dumped) `