search

nmslib / hnswlib

Header-only C++/python library for fast approximate nearest neighbors
https://github.com/nmslib/hnswlib
Apache License 2.0
4.11k stars 607 forks source link

Illegal instruction (core dumped) when load_index #564

Open kuangue opened 3 weeks ago

kuangue commented 3 weeks ago

Hi, folks. I found the problem on hnswlib 0.8.0 as follows:

root@ip-mask:/app# python --version
Python 3.11.2
root@ip-mask:/app# uname -a
Linux ip-mask.ap-northeast-1.compute.internal 4.14.343-261.564.amzn2.aarch64 #1 SMP Tue May 7 02:23:29 UTC 2024 aarch64 GNU/Linux
root@ip-mask:/app# cat check.py
import hnswlib

engine = hnswlib.Index(space="l2", dim=50)
engine.load_index("/tmp/index.bin", allow_replace_deleted=False)
engine.set_ef(1000)

root@ip-mask:/app# gdb -c core.17315 python
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
(No debugging symbols found in python)
[New LWP 17315]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `python check.py'.
Program terminated with signal SIGILL, Illegal instruction.
#0  std::__atomic_base<unsigned long>::store (__m=std::memory_order_seq_cst, __i=0, this=0xaaab012aadc0) at /usr/include/c++/10/bits/atomic_base.h:397
397           store(__int_type __i, memory_order __m = memory_order_seq_cst) noexcept
(gdb) bt
#0  std::__atomic_base<unsigned long>::store (__m=std::memory_order_seq_cst, __i=0, this=0xaaab012aadc0) at /usr/include/c++/10/bits/atomic_base.h:397
#1  std::__atomic_base<unsigned long>::operator= (__i=0, this=0xaaab012aadc0) at /usr/include/c++/10/bits/atomic_base.h:297
#2  hnswlib::HierarchicalNSW<float>::clear (this=0xaaab012aadb0) at ./hnswlib/hnswalg.h:160
#3  hnswlib::HierarchicalNSW<float>::loadIndex (this=this@entry=0xaaab012aadb0, location=..., s=0xaaab0123d180, max_elements_i=max_elements_i@entry=0)
    at ./hnswlib/hnswalg.h:722
#4  0x0000ffffaac3e75c in hnswlib::HierarchicalNSW<float>::HierarchicalNSW (allow_replace_deleted=false, max_elements=0, nmslib=false, location=...,
    s=<optimized out>, this=0xaaab012aadb0) at ./hnswlib/hnswalg.h:85
#5  Index<float, float>::loadIndex (this=0xaaab012bfd50, path_to_index=..., max_elements=0, allow_replace_deleted=false) at ./python_bindings/bindings.cpp:235
#6  0x0000ffffaac36174 in pybind11::cpp_function::cpp_function<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}::operator()(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool) const (args#2=<optimized out>, args#1=<optimized out>, args#0="/tmp/jd_index.bin", c=<optimized out>,
    this=<optimized out>) at /tmp/tmp86094u_6/.venv/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:154
#7  pybind11::detail::argument_loader<Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool>::call_impl<void, pybind11::cpp_function::cpp_function<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}&, 0ul, 1ul, 2ul, 3ul, pybind11::detail::void_type>(pybind11::cpp_function::cpp_function<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul>, pybind11::detail::void_type&&) && (f=..., this=0xfffffab02a88)
    at /tmp/tmp86094u_6/.venv/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1613
#8  pybind11::detail::argument_loader<Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool>::call<void, pybind11::detail::void_type, pybind11::cpp_function::cpp_function<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}&>(pybind11::cpp_function::cpp_function<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}&) && (f=..., f=...,
    this=0xfffffab02a88) at /tmp/tmp86094u_6/.venv/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1587
#9  pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_methodconst&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}, void, Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>,--Type <RET> for more, q to quit, c to continue without paging--
 std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(pybind11::cpp_function::initialize<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, unsigned long, bool)#1}&&, void (*)(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::operator()(pybind11::detail::function_call&) const (this=0x0, call=...)
    at /tmp/tmp86094u_6/.venv/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:297
#10 pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_methodconst&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)#1}, void, Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(pybind11::cpp_function::initialize<void, Index<float, float>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v, pybind11::arg_v>(void (Index<float, float>::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, unsigned long, bool)#1}&&, void (*)(Index<float, float>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
    at /tmp/tmp86094u_6/.venv/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:267
#11 0x0000ffffaac239cc in pybind11::cpp_function::dispatcher (self=<optimized out>, args_in=0xffffaac8ca00, kwargs_in=0xffffaac67640)
    at /tmp/tmp86094u_6/.venv/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:987
#12 0x0000ffffab3682cc in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#13 0x0000ffffab33ee9c in _PyObject_MakeTpCall () from /usr/local/bin/../lib/libpython3.11.so.1.0
#14 0x0000ffffab34b4d0 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.11.so.1.0
#15 0x0000ffffab3496c0 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#16 0x0000ffffab3b9048 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.11.so.1.0
#17 0x0000ffffab4028ac in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#18 0x0000ffffab402814 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#19 0x0000ffffab4021d0 in ?? () from /usr/local/bin/../lib/libpython3.11.so.1.0
#20 0x0000ffffab401d5c in _PyRun_SimpleFileObject () from /usr/local/bin/../lib/libpython3.11.so.1.0
#21 0x0000ffffab401ae0 in _PyRun_AnyFileObject () from /usr/local/bin/../lib/libpython3.11.so.1.0
#22 0x0000ffffab3fa4fc in Py_RunMain () from /usr/local/bin/../lib/libpython3.11.so.1.0
#23 0x0000ffffab3a1d98 in Py_BytesMain () from /usr/local/bin/../lib/libpython3.11.so.1.0
#24 0x0000ffffab043e18 in __libc_start_main (main=0xaaaad0d129b0 <_start+288>, argc=2, argv=0xfffffab033f8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#25 0x0000aaaad0d128c4 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

hnswlib/hnswalg.h:160 => cur_element_count = 0; I don't know what happened. Do you have any idea?

kuangue commented 3 weeks ago

I found the previous problem occurred with init_index().