Panic in Linux thread_parker

[agourlay]

parking_lot version: 0.12.1

While doing performance tests with a lot of concurrent threads and RwLocks, I got the following panic due to a debug assertion.

/parking_lot_core-0.9.5/src/thread_parker/linux.rs at line 120: assertion failed: r == 0 || r == -1

Here is the whole trace.

[2023-06-26T13:46:11.974Z ERROR qdrant::startup] Panic occurred in file /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/thread_parker/linux.rs at line 120: assertion failed: r == 0 || r == -1
[2023-06-26T13:46:11.958Z ERROR qdrant::startup] Panic backtrace: 
       0: qdrant::startup::setup_panic_hook::{{closure}}
                 at ./src/startup.rs:38:25
       1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
       2: std::panicking::rust_panic_with_hook
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:695:13
       3: std::panicking::begin_panic_handler::{{closure}}
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:580:13
       4: std::sys_common::backtrace::__rust_end_short_backtrace
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
       5: rust_begin_unwind
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
       6: core::panicking::panic_fmt
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
       7: core::panicking::panic
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:117:5
       8: parking_lot_core::thread_parker::imp::ThreadParker::futex_wait
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/thread_parker/linux.rs:120:9
       9: <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/thread_parker/linux.rs:66:13
      10: parking_lot_core::parking_lot::park::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/parking_lot.rs:635:17
      11: parking_lot_core::parking_lot::with_thread_data
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/parking_lot.rs:207:5
      12: parking_lot_core::parking_lot::park
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/parking_lot.rs:600:5
      13: parking_lot::raw_rwlock::RawRwLock::lock_common
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115:17
      14: parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:633:26
      15: <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:73:26
      16: lock_api::rwlock::RwLock<R,T>::write
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.9/src/rwlock.rs:480:9
      17: segment::index::hnsw_index::graph_layers_builder::GraphLayersBuilder::link_new_point
                 at ./lib/segment/src/index/hnsw_index/graph_layers_builder.rs:368:33
      18: <segment::index::hnsw_index::hnsw::HNSWIndex<TGraphLinks> as segment::index::vector_index_base::VectorIndex>::build_index::{{closure}}::{{closure}}
                 at ./lib/segment/src/index/hnsw_index/hnsw.rs:556:21
      19: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:274:13
      20: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:310:13
      21: core::option::Option<T>::map
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/option.rs:1099:29
      22: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:103:9
      23: rayon::iter::plumbing::Folder::consume_iter
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:178:21
      24: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/map.rs:248:21
      25: rayon::iter::plumbing::Producer::fold_with
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:110:9
      26: rayon::iter::plumbing::bridge_producer_consumer::helper
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:438:13
      27: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
      28: rayon_core::join::join_context::call_b::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
      29: rayon_core::job::StackJob<L,F,R>::run_inline
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:102:9
      30: rayon_core::join::join_context::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:159:36
      31: rayon_core::registry::in_worker
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
      32: rayon_core::join::join_context
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
      33: rayon::iter::plumbing::bridge_producer_consumer::helper
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
      34: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
      35: rayon_core::join::join_context::call_b::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
      36: rayon_core::job::StackJob<L,F,R>::run_inline
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:102:9
      37: rayon_core::join::join_context::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:159:36
      38: rayon_core::registry::in_worker
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
      39: rayon_core::join::join_context
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
      40: rayon::iter::plumbing::bridge_producer_consumer::helper
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
      41: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:418:21
      42: rayon_core::join::join_context::call_a::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:124:17
      43: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
      44: std::panicking::try::do_call
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      45: __rust_try
      46: std::panicking::try
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      47: std::panic::catch_unwind
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      48: rayon_core::unwind::halt_unwinding
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/unwind.rs:17:5
      49: rayon_core::join::join_context::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:142:24
      50: rayon_core::registry::in_worker
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
      51: rayon_core::join::join_context
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
      52: rayon::iter::plumbing::bridge_producer_consumer::helper
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
      53: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
      54: rayon_core::join::join_context::call_b::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
      55: rayon_core::job::JobResult<T>::call::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:41
      56: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
      57: std::panicking::try::do_call
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      58: __rust_try
      59: std::panicking::try
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      60: std::panic::catch_unwind
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      61: rayon_core::unwind::halt_unwinding
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/unwind.rs:17:5
      62: rayon_core::job::JobResult<T>::call
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:15
      63: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:120:32
      64: rayon_core::job::JobRef::execute
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:64:9
      65: rayon_core::registry::WorkerThread::execute
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:874:9
      66: rayon_core::registry::WorkerThread::wait_until_cold
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:820:17
      67: rayon_core::registry::WorkerThread::wait_until
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:803:13
      68: rayon_core::join::join_context::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:167:17
      69: rayon_core::registry::in_worker
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
      70: rayon_core::join::join_context
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
      71: rayon::iter::plumbing::bridge_producer_consumer::helper
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
      72: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
      73: rayon_core::join::join_context::call_b::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
      74: rayon_core::job::JobResult<T>::call::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:41
      75: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
      76: std::panicking::try::do_call
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      77: __rust_try
      78: std::panicking::try
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      79: std::panic::catch_unwind
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      80: rayon_core::unwind::halt_unwinding
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/unwind.rs:17:5
      81: rayon_core::job::JobResult<T>::call
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:15
      82: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:120:32
      83: rayon_core::job::JobRef::execute
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:64:9
      84: rayon_core::registry::WorkerThread::execute
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:874:9
      85: rayon_core::registry::WorkerThread::wait_until_cold
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:820:17
      86: rayon_core::registry::WorkerThread::wait_until
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:803:13
      87: rayon_core::registry::main_loop
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:948:5
      88: rayon_core::registry::ThreadBuilder::run
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:54:18
      89: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
                 at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:99:20
      90: std::sys_common::backtrace::__rust_begin_short_backtrace
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
      91: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/mod.rs:526:17
      92: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
      93: std::panicking::try::do_call
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
      94: __rust_try
      95: std::panicking::try
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
      96: std::panic::catch_unwind
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
      97: std::thread::Builder::spawn_unchecked_::{{closure}}
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/mod.rs:525:30
      98: core::ops::function::FnOnce::call_once{{vtable.shim}}
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
      99: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
     100: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
     101: std::sys::unix::thread::Thread::new::thread_start
                 at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys/unix/thread.rs:108:17
     102: start_thread
                 at ./nptl/./nptl/pthread_create.c:444:8
     103: __GI___clone3
                 at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

At the time of the error the system was not running out of memory.

I would love to know what kind of invariant was broken to know if I am maybe misusing this library.

Thanks!

[Amanieu]

This should not be possible. futex(FUTEX_WAIT) is documented (by its man page) to only ever return 0 or -1. What kernel version are you running? Do you have anything that might mess with system calls (e.g. seccomp filters, ptrace)?

[agourlay]

Thank you for the fast answer!

Do you have anything that might mess with system calls (e.g. seccomp filters, ptrace)?

Bingo, I was profiling with Intel VTune at the same time, I guess we can assume this is responsible for that broken invariant.

I have never seen this behavior in the past, sorry for the noise.

[Amanieu]

I'm going to close this for now. Do reopen if you manage to reproduce this without vtune.