search

redis / redis-om-python

Object mapping, and more, for Redis and Python
MIT License
1.06k stars 108 forks source link

Redis 6.2.12 crashed by signal: 6 #547

Closed jcq15 closed 11 months ago

jcq15 commented 11 months ago

I'm using redis-om-python with redis-stack. Here is the report. Thanks for any help!

=== REDIS BUG REPORT START: Cut & paste starting from here === 9:M 28 Jul 2023 01:15:23.374 # Redis 6.2.12 crashed by signal: 6, si_code: -6 9:M 28 Jul 2023 01:15:23.374 # Crashed running the instruction at: 0x7f801bdf600b

------ STACK TRACE ------ EIP: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f801bdf600b]

Backtrace: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f801bfb9420] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f801bdf600b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f801bdd5859] /opt/redis-stack/lib/rejson.so(+0x1c85f7)[0x7f80146985f7] /opt/redis-stack/lib/rejson.so(+0x1c6af2)[0x7f8014696af2] /opt/redis-stack/lib/rejson.so(+0x1c697a)[0x7f801469697a] /opt/redis-stack/lib/rejson.so(+0x1c6672)[0x7f8014696672] /opt/redis-stack/lib/rejson.so(+0x1c5726)[0x7f8014695726] /opt/redis-stack/lib/rejson.so(+0x1c63c2)[0x7f80146963c2] /opt/redis-stack/lib/rejson.so(+0x3f203)[0x7f801450f203] /opt/redis-stack/lib/rejson.so(+0x3f29d)[0x7f801450f29d] /opt/redis-stack/lib/rejson.so(+0x930c3)[0x7f80145630c3] /opt/redis-stack/lib/rejson.so(JSONAPI_getInt+0x6b)[0x7f80145bbbab] /opt/redis-stack/lib/redisearch.so(JSON_StoreInDocField+0x113)[0x7f801b1915d3] /opt/redis-stack/lib/redisearch.so(JSON_LoadDocumentField+0xd1)[0x7f801b191841] /opt/redis-stack/lib/redisearch.so(Document_LoadSchemaFieldJson+0x11d)[0x7f801b17417d] /opt/redis-stack/lib/redisearch.so(IndexSpec_UpdateDoc+0x1a0)[0x7f801b1bb3e0] /opt/redis-stack/lib/redisearch.so(Indexes_UpdateMatchingWithSchemaRules+0xd6)[0x7f801b1bbb06] /opt/redis-stack/lib/redisearch.so(HashNotificationCallback+0x1fd)[0x7f801b1964fd] /opt/redis-stack/bin/redis-server :6379(moduleNotifyKeyspaceEvent+0xfc)[0x55bac94697ec] /opt/redis-stack/bin/redis-server :6379(notifyKeyspaceEvent+0x35)[0x55bac9455775] /opt/redis-stack/bin/redis-server :6379(RM_NotifyKeyspaceEvent+0x33)[0x55bac94604b3] /opt/redis-stack/lib/rejson.so(+0x10bf62)[0x7f80145dbf62] /opt/redis-stack/lib/rejson.so(+0x4ad51)[0x7f801451ad51] /opt/redis-stack/lib/rejson.so(+0xee697)[0x7f80145be697] /opt/redis-stack/bin/redis-server :6379(RedisModuleCommandDispatcher+0x67)[0x55bac9467417] /opt/redis-stack/bin/redis-server :6379(call+0xf0)[0x55bac93d8940] /opt/redis-stack/bin/redis-server :6379(processCommand+0x643)[0x55bac93da643] /opt/redis-stack/bin/redis-server :6379(processCommandAndResetClient+0x20)[0x55bac93edd20] /opt/redis-stack/bin/redis-server :6379(processInputBuffer+0xea)[0x55bac93f067a] /opt/redis-stack/bin/redis-server :6379(+0x1018ac)[0x55bac94898ac] /opt/redis-stack/bin/redis-server :6379(aeProcessEvents+0x2ca)[0x55bac93d0e6a] /opt/redis-stack/bin/redis-server :6379(aeMain+0x1d)[0x55bac93d10fd] /opt/redis-stack/bin/redis-server :6379(main+0x33a)[0x55bac93cd50a] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f801bdd7083] /opt/redis-stack/bin/redis-server *:6379(_start+0x2e)[0x55bac93cda0e]

------ REGISTERS ------ 9:M 28 Jul 2023 01:15:23.375 # RAX:0000000000000000 RBX:00007f801bdaef80 RCX:00007f801bdf600b RDX:0000000000000000 RDI:0000000000000002 RSI:00007ffee932c5e0 RBP:00007ffee932ca58 RSP:00007ffee932c5e0 R8 :0000000000000000 R9 :00007ffee932c5e0 R10:0000000000000008 R11:0000000000000246 R12:0000000000000001 R13:0000000000000001 R14:00007f8014786528 R15:00007ffee932c990 RIP:00007f801bdf600b EFL:0000000000000246 CSGSFS:002b000000000033 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5ef) -> 00007ffee932c720 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5ee) -> 0000000000000000 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5ed) -> 0000000000000000 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5ec) -> 0000000000000010 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5eb) -> 00007f80147865c0 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5ea) -> 00007ffee932c7e0 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e9) -> 00007f80146931ed 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e8) -> 00007ffee932c697 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e7) -> 0000000000000001 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e6) -> 00007ffee932c697 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e5) -> 0000000000000001 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e4) -> 0000000000000001 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e3) -> 00007f801bfb8344 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e2) -> 0000000000000000 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e1) -> 00007ffee932c697 9:M 28 Jul 2023 01:15:23.375 # (00007ffee932c5e0) -> 0000000000000000

------ INFO OUTPUT ------

Server

redis_version:6.2.12 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:4dbc2487343b0024 redis_mode:standalone os:Linux 5.19.0-1025-aws x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:9.4.0 process_id:9 process_supervised:no run_id:1f7c0d1b5078c36ec6fdd4e84b6807690e574d81 tcp_port:6379 server_time_usec:1690506923358641 uptime_in_seconds:45628 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:12785323 executable:/opt/redis-stack/bin/redis-server config_file:/redis-stack.conf io_threads_active:0

Clients

connected_clients:1 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:168 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:35747072 used_memory_human:34.09M used_memory_rss:52899840 used_memory_rss_human:50.45M used_memory_peak:35747072 used_memory_peak_human:34.09M used_memory_peak_perc:100.01% used_memory_overhead:2267336 used_memory_startup:1014472 used_memory_dataset:33479736 used_memory_dataset_perc:96.39% allocator_allocated:35827568 allocator_active:36835328 allocator_resident:39825408 total_system_memory:4110970880 total_system_memory_human:3.83G used_memory_lua:30720 used_memory_lua_human:30.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.03 allocator_frag_bytes:1007760 allocator_rss_ratio:1.08 allocator_rss_bytes:2990080 rss_overhead_ratio:1.33 rss_overhead_bytes:13074432 mem_fragmentation_ratio:1.48 mem_fragmentation_bytes:17197296 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:20648 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0

Persistence

loading:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:60 rdb_bgsave_in_progress:0 rdb_last_save_time:1690506920 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:1265664 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0

Stats

total_connections_received:2 total_commands_processed:212 instantaneous_ops_per_sec:6 total_net_input_bytes:74545 total_net_output_bytes:83358 instantaneous_input_kbps:2.63 instantaneous_output_kbps:0.85 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:941 evicted_keys:0 keyspace_hits:282995 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:1347 total_forks:21 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:0 dump_payload_sanitizations:0 total_reads_processed:212 total_writes_processed:210 io_threaded_reads_processed:0 io_threaded_writes_processed:0

Replication

role:master connected_slaves:0 master_failover_state:no-failover master_replid:8296e900be401189944f8ea16c1f6142d9585112 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:6.724217 used_cpu_user:64.395662 used_cpu_sys_children:0.550217 used_cpu_user_children:4.194333 used_cpu_sys_main_thread:6.578226 used_cpu_user_main_thread:62.951034

Modules

module:name=ReJSON,ver=20407,api=1,filters=0,usedby=[search|graph],using=[],options=[handle-io-errors] module:name=search,ver=20609,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors] module:name=graph,ver=21010,api=1,filters=0,usedby=[],using=[ReJSON],options=[] module:name=timeseries,ver=10810,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors] module:name=bf,ver=20405,api=1,filters=0,usedby=[],using=[],options=[]

Commandstats

cmdstat_command:calls=2,usec=20,usec_per_call=10.00,rejected_calls=0,failed_calls=0 cmdstat_hset:calls=132,usec=15934,usec_per_call=120.71,rejected_calls=0,failed_calls=0 cmdstat_auth:calls=2,usec=11,usec_per_call=5.50,rejected_calls=0,failed_calls=0 cmdstat_FT.SEARCH:calls=51,usec=392995,usec_per_call=7705.78,rejected_calls=0,failed_calls=0 cmdstat_info:calls=2,usec=38,usec_per_call=19.00,rejected_calls=0,failed_calls=0 cmdstat_json.set:calls=11,usec=1603,usec_per_call=145.73,rejected_calls=0,failed_calls=0 cmdstat_FT.INFO:calls=6,usec=288,usec_per_call=48.00,rejected_calls=0,failed_calls=0 cmdstat_get:calls=6,usec=23,usec_per_call=3.83,rejected_calls=0,failed_calls=0

Errorstats

Cluster

cluster_enabled:0

Keyspace

db0:keys=24251,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=25 addr=172.17.0.1:45744 laddr=172.17.0.3:6379 fd=12 name= age=45543 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=512 qbuf-free=40442 argv-mem=481 obl=0 oll=0 omem=0 tot-mem=61953 events=r cmd=json.set user=default redir=-1

------ CURRENT CLIENT INFO ------ id=25 addr=172.17.0.1:45744 laddr=172.17.0.3:6379 fd=12 name= age=45543 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=512 qbuf-free=40442 argv-mem=481 obl=0 oll=0 omem=0 tot-mem=61953 events=r cmd=json.set user=default redir=-1 argv[0]: 'JSON.SET' argv[1]: 'karma:bots.karma.UserTable:01H698K7M2BPAFF93T2GNXAQ39' argv[2]: '.' argv[3]: '{"pk": "01H698K7M2BPAFF93T2GNXAQ39", "user_id": "98acba66", "name": "\u5c3c\u83ab\u514b", "word": "4", "bought": [21, 21, 24, 22, 21, 101], "items": [25616506980096071, 59826576399327, 113507698638, 165643800, 169791, 101], "prices": [12800, 256000, 10240000, 102400000, 1536000000, 1374389534720000000], "eff": [1, 1, 1, 1, 1, 1], "level": 5, "karma": 2368245463503591660, "total": 9240193141634191760, "base_eff": 10}' 9:M 28 Jul 2023 01:15:23.375 # key 'karma:bots.karma.UserTable:01H698K7M2BPAFF93T2GNXAQ39' found in DB containing the following object: 9:M 28 Jul 2023 01:15:23.375 # Object type: 5 9:M 28 Jul 2023 01:15:23.375 # Object encoding: 0 9:M 28 Jul 2023 01:15:23.375 # Object refcount: 1

------ MODULES INFO OUTPUT ------

ReJSON_trace

ReJSON_trace: 0: redis_module::base_info_func 1: rejson::info_func 2: modulesCollectInfo at /w/redis-stack/redis-stack/redis/src/module.c:7100:9 3: logModulesInfo at /w/redis-stack/redis-stack/redis/src/debug.c:1624:22 4: printCrashReport at /w/redis-stack/redis-stack/redis/src/debug.c:1896:5 5: sigsegvHandler at /w/redis-stack/redis-stack/redis/src/debug.c:1878:5 6: 7: gsignal 8: abort 9: std::sys::unix::abort_internal 10: rust_panic 11: std::panicking::rust_panic_with_hook 12: std::panicking::begin_panic_handler::{{closure}} 13: std::sys_common::backtrace::__rust_end_short_backtrace 14: rust_begin_unwind 15: core::panicking::panic_fmt 16: core::panicking::panic 17: rejson::jsonpath::json_node::::get_long 18: JSONAPI_getInt 19: JSON_StoreInDocField 20: JSON_LoadDocumentField 21: Document_LoadSchemaFieldJson 22: IndexSpec_UpdateDoc 23: Indexes_UpdateMatchingWithSchemaRules 24: HashNotificationCallback 25: moduleNotifyKeyspaceEvent at /w/redis-stack/redis-stack/redis/src/module.c:5983:13 26: notifyKeyspaceEvent at /w/redis-stack/redis-stack/redis/src/notify.c:111:6 27: RM_NotifyKeyspaceEvent at /__w/redis-stack/redis-stack/redis/src/module.c:5951:5 28: redis_module::raw::notify_keyspace_event 29: rejson::commands::json_set 30: rejson::RedisModule_OnLoad::do_command 31: RedisModuleCommandDispatcher at /w/redis-stack/redis-stack/redis/src/module.c:695:5 32: call at /w/redis-stack/redis-stack/redis/src/server.c:3750:5 33: processCommand at /w/redis-stack/redis-stack/redis/src/server.c:4297:9 34: processCommandAndResetClient at /w/redis-stack/redis-stack/redis/src/networking.c:2105:9 35: processInputBuffer at /w/redis-stack/redis-stack/redis/src/networking.c:2206:17 36: callHandler at /w/redis-stack/redis-stack/redis/src/connhelpers.h:79:18 connSocketEventHandler at /w/redis-stack/redis-stack/redis/src/connection.c:295:14 37: aeProcessEvents at /w/redis-stack/redis-stack/redis/src/ae.c:427:17 38: aeMain at /w/redis-stack/redis-stack/redis/src/ae.c:487:9 39: main at /w/redis-stack/redis-stack/redis/src/server.c:6474:5 40: __libc_start_main 41: _start

search_version

search_version:2.6.9 search_redis_version:6.2.12 - oss

search_index

search_number_of_indexes:3

search_fields_statistics

search_fields_numeric:Numeric=7 search_fields_tag:Tag=9

search_dialect_statistics

search_dialect_1:1 search_dialect_2:0 search_dialect_3:0

search_runtime_configurations

search_concurrent_mode:OFF search_enableGC:ON search_minimal_term_prefix:2 search_maximal_prefix_expansions:200 search_query_timeout_ms:500 search_timeout_policy:return search_cursor_read_size:1000 search_cursor_max_idle_time:300000 search_max_doc_table_size:1000000 search_max_search_results:10000 search_max_aggregate_results:10000 search_search_pool_size:20 search_index_pool_size:8 search_gc_scan_size:100 search_min_phonetic_term_length:3

graph_executing commands

------ FAST MEMORY TEST ------ 9:M 28 Jul 2023 01:15:23.401 # Bio thread for job type #0 terminated 9:M 28 Jul 2023 01:15:23.401 # Bio thread for job type #1 terminated 9:M 28 Jul 2023 01:15:23.401 # Bio thread for job type #2 terminated Preparing to test memory region 55bac95a1000 (2281472 bytes) Preparing to test memory region 55bacabc0000 (405504 bytes) Preparing to test memory region 7f8008000000 (135168 bytes) Preparing to test memory region 7f800f000000 (8388608 bytes) Preparing to test memory region 7f800f800000 (4194304 bytes) Preparing to test memory region 7f800fdcc000 (40894464 bytes) Preparing to test memory region 7f80124cd000 (8388608 bytes) Preparing to test memory region 7f8012cce000 (8388608 bytes) Preparing to test memory region 7f80134cf000 (8388608 bytes) Preparing to test memory region 7f8013cd0000 (8388608 bytes) Preparing to test memory region 7f801478c000 (8388608 bytes) Preparing to test memory region 7f8014f8d000 (8388608 bytes) Preparing to test memory region 7f801578e000 (8388608 bytes) Preparing to test memory region 7f8015f8f000 (8388608 bytes) Preparing to test memory region 7f8016790000 (8388608 bytes) Preparing to test memory region 7f8016f91000 (8388608 bytes) Preparing to test memory region 7f8017792000 (8388608 bytes) Preparing to test memory region 7f8017f93000 (8388608 bytes) Preparing to test memory region 7f8019ec3000 (12288 bytes) Preparing to test memory region 7f8019ec7000 (8388608 bytes) Preparing to test memory region 7f801a6c8000 (8388608 bytes) Preparing to test memory region 7f801b0a7000 (12288 bytes) Preparing to test memory region 7f801b3fd000 (12288 bytes) Preparing to test memory region 7f801b400000 (8388608 bytes) Preparing to test memory region 7f801bd4f000 (4096 bytes) Preparing to test memory region 7f801bdad000 (24576 bytes) Preparing to test memory region 7f801bfa1000 (16384 bytes) Preparing to test memory region 7f801bfc4000 (16384 bytes) Preparing to test memory region 7f801c29b000 (16384 bytes) Preparing to test memory region 7f801c487000 (8192 bytes) *** Preparing to test memory region 7f801c4ba000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: gsignal (base: 0x7f801bdf5f40) Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7f801bdb3000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x7f801bdf5f40 -D -b binary -m i386:x86-64 /tmp/dump.bin

9:M 28 Jul 2023 01:15:24.038 # dump of function (hexdump of 331 bytes): f30f1efa4881ec180100004189f831ff660f6f058898170064488b042528000000488984240801000031c04989e1b80e000000488db424800000000f29842480000000660f76c041ba080000004c89ca0f298424900000000f298424a00000000f298424b00000000f298424c00000000f298424d00000000f298424e00000000f298424f00000000f05b92700000089c80f054889c7b8ba0000000f0589c64489c2b8ea0000000f05483d00f0ffff773f4189c041ba0800000031d24c89cebf02000000b80e0000000f05488b84240801000064483304252800000075264489c04881c418010000c30f1f8000000000488b15398e1a00f7d841b8ffffffff648902ebb0e827ca0e000f1f8000000000f30f1efa85ff7808f7dfe97103000090488b05098e1a0064c70016000000b8ffffffffc3662e0f1f84000000000066909066662e0f1f8400000000 Function at 0x7f801bee2a70 is __stack_chk_fail

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

jcq15 commented 11 months ago

It's solved. There is a long int (larger than 2^64) in my python script which caused some basic service crashed.