search

RediSearch / RediSearch

A query and indexing engine for Redis, providing secondary indexing, full-text search, vector similarity search and aggregations.
https://redis.io/docs/stack/search/
Other
5.43k stars 517 forks source link

Redisearch compile for armv7 on raspberry pi3 crashes with segfault #388

Open JefStat opened 6 years ago

JefStat commented 6 years ago

Attempt to run Redisearch 1.2.0 has the following issue.

------ DUMPING CODE AROUND EIP ------
Symbol: strlen (base: 0x76dc4be0)
Module: /lib/arm-linux-gnueabihf/libc.so.6 (base 0x76d4f000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x76dc4be0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
2953:M 16 Jul 20:26:02.161 # dump of function (hexdump of 128 bytes):
0020d0e5073000e20010a0e10f3063e2000052e31600000a0120f0e5013053e200005213fbffff1a000052e31000000a010080e2d820c0e03cc09fe540f0d0f5f22f6ce6f33f6ce680f0d0f5023093e1d820c000f9ffff0a000052e3040040e20320a00104004012322fbfe6122f6fe1a20180e0010040e01eff2fe101010101
itamarhaber commented 6 years ago

Thanks @JefStat - while undocumented, this is certainly expected. AFAIR, RediSearch does unaligned allocations a lot.

JefStat commented 6 years ago

We tried using the following compile option -mno-unaligned-access from the comment about unaligned allocations but that didn't fix the segfault. More info: https://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html

JefStat commented 6 years ago 
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1560:M 26 Jul 15:40:03.561 # Redis 4.0.10 crashed by signal: 11
1560:M 26 Jul 15:40:03.562 # Crashed running the instruction at: 0x76dafbe0
1560:M 26 Jul 15:40:03.562 # Accessing address: 0xc8
1560:M 26 Jul 15:40:03.562 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
/lib/arm-linux-gnueabihf/libc.so.6(strlen+0x0)[0x76dafbe0]
Backtrace:
./redis-server *:6379(logStackTrace+0x30)[0x70fc4]
./redis-server *:6379(sigsegvHandler+0x90)[0x71694]
/lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0x0)[0x76d666c0]
------ INFO OUTPUT ------
# Server
redis_version:4.0.10
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:945bccecc7dbc216
redis_mode:standalone
os:Linux 4.14.34-v7+ armv7l
arch_bits:32
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:6.3.0
process_id:1560
run_id:a2e918101a47b687d147ca5613fa6063e46b3447
tcp_port:6379
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:5910947
executable:/home/pi/redis/redis-stable/src/./redis-server
config_file:
# Clients
connected_clients:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:637306
used_memory_human:622.37K
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:637306
used_memory_peak_human:622.37K
used_memory_peak_perc:inf%
used_memory_overhead:636302
used_memory_startup:636302
used_memory_dataset:1004
used_memory_dataset_perc:100.00%
total_system_memory:971730944
total_system_memory_human:926.71M
used_memory_lua:27648
used_memory_lua_human:27.00K
maxmemory:3221225472
maxmemory_human:3.00G
maxmemory_policy:noeviction
mem_fragmentation_ratio:0.00
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1532637603
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
# Stats
total_connections_received:0
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:master
connected_slaves:0
master_replid:860e1d5ec8fc248648f8f1e4926c8f5c814c295b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.01
used_cpu_user:0.02
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Commandstats
# Cluster
cluster_enabled:0
# Keyspace
------ CLIENT LIST OUTPUT ------
------ REGISTERS ------
------ FAST MEMORY TEST ------
1560:M 26 Jul 15:40:03.564 # Bio thread for job type #0 terminated
1560:M 26 Jul 15:40:03.565 # Bio thread for job type #1 terminated
1560:M 26 Jul 15:40:03.565 # Bio thread for job type #2 terminated
*** Preparing to test memory region f7000 (45056 bytes)
*** Preparing to test memory region 1c0c000 (540672 bytes)
*** Preparing to test memory region 7534d000 (8192 bytes)
*** Preparing to test memory region 75350000 (8384512 bytes)
*** Preparing to test memory region 75b50000 (8384512 bytes)
*** Preparing to test memory region 76350000 (8712192 bytes)
*** Preparing to test memory region 76e76000 (12288 bytes)
*** Preparing to test memory region 76ea0000 (8192 bytes)
*** Preparing to test memory region 76fbd000 (20480 bytes)
.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: strlen (base: 0x76dafbe0)
Module: /lib/arm-linux-gnueabihf/libc.so.6 (base 0x76d3a000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x76dafbe0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1560:M 26 Jul 15:40:05.450 # dump of function (hexdump of 128 bytes):
0020d0e5073000e20010a0e10f3063e2000052e31600000a0120f0e5013053e200005213fbffff1a000052e31000000a010080e2d820c0e03cc09fe540f0d0f5f22f6ce6f33f6ce680f0d0f5023093e1d820c000f9ffff0a000052e3040040e20320a00104004012322fbfe6122f6fe1a20180e0010040e01eff2fe101010101
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
itamarhaber commented 6 years ago

Thanks.

I'm curious - why are you trying to run RediSearch on ARM?

JefStat commented 6 years ago

To offer a lower price point for our product.

mnunberg commented 6 years ago

@JefStat Can you give me some resources/quick pointers to how I might be able to reproduce this without using an RP?

omribahumi commented 5 years ago

I guess you could try reproducing this with qemu with one of these:

Also, a coredump + the binaries might help you debug this without going through the effort of reproducing. Apparently you can open foreign architecture coredump with gdb: https://stackoverflow.com/questions/12621511/gdb-wont-read-core-file-from-foreign-architecture

itamarhaber commented 5 years ago

@omribahumi hey :alien: