Amanieu / parking_lot

Compact and efficient synchronization primitives for Rust. Also provides an API for creating custom synchronization primitives.
Apache License 2.0
2.77k stars 217 forks source link

Apparent bug/hang in Mutex #361

Closed cstrahan-blueshift closed 2 years ago

cstrahan-blueshift commented 2 years ago

My system:

I originally opened an issue here (but I'll recap below): https://github.com/nnethercote/dhat-rs/issues/25

I noticed that when using dhat and moka together, the process would hang somewhere inmoka::sync::Cache::new() (or moka::future::Cache::new()).

Here's a small reproducer: https://github.com/cstrahan-blueshift/dhat_moka_hang_repro

Run cargo run --release and it'll hang about 1 in 10 times.

I took a look at the stack with gdb, and this is where it would hang:

(gdb) info stack
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055fe6771c5d1 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait () at src/thread_parker/linux.rs:112
#2  parking_lot_core::thread_parker::imp::{impl#0}::park () at src/thread_parker/linux.rs:66
#3  parking_lot_core::word_lock::{impl#1}::lock_slow::{closure#0} () at src/word_lock.rs:167
#4  parking_lot_core::word_lock::with_thread_data<usize, parking_lot_core::word_lock::{impl#1}::lock_slow::{closure_env#0}> () at src/word_lock.rs:67
#5  parking_lot_core::word_lock::WordLock::lock_slow () at src/word_lock.rs:136
#6  0x000055fe6774e7bc in parking_lot_core::word_lock::WordLock::lock () at src/word_lock.rs:97
#7  parking_lot_core::parking_lot::grow_hashtable () at src/parking_lot.rs:276
#8  parking_lot_core::parking_lot::ThreadData::new () at src/parking_lot.rs:180
#9  0x000055fe6774d800 in parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__init () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.4/src/parking_lot.rs:202
#10 parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit::{closure#0} () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:353
#11 std::thread::local::lazy::LazyKeyInner::initialize<parking_lot_core::parking_lot::ThreadData, parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit::{closure_env#0}> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:809
#12 std::thread::local::fast::Key::try_initialize<parking_lot_core::parking_lot::ThreadData, parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit::{closure_env#0}> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:987
#13 0x000055fe6771bce6 in std::thread::local::fast::Key::get<parking_lot_core::parking_lot::ThreadData, parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit::{closure_env#0}> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:970
#14 parking_lot_core::parking_lot::with_thread_data::THREAD_DATA::__getit () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:345
#15 std::thread::local::LocalKey::try_with<parking_lot_core::parking_lot::ThreadData, parking_lot_core::parking_lot::with_thread_data::{closure_env#0}<parking_lot_core::parking_lot::ParkResult, parking_lot_core::parking_lot::park::{closure_env#0}<parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#0}, parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#1}, parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#2}>>, *const parking_lot_core::parking_lot::ThreadData> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:444
#16 parking_lot_core::parking_lot::with_thread_data<parking_lot_core::parking_lot::ParkResult, parking_lot_core::parking_lot::park::{closure_env#0}<parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#0}, parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#1}, parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#2}>> ()
    at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.4/src/parking_lot.rs:203
#17 parking_lot_core::parking_lot::park<parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#0}, parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#1}, parking_lot::raw_mutex::{impl#3}::lock_slow::{closure_env#2}> ()
    at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.9.4/src/parking_lot.rs:600
#18 parking_lot::raw_mutex::RawMutex::lock_slow () at src/raw_mutex.rs:262
#19 0x000055fe6773ed07 in parking_lot::raw_mutex::{impl#0}::lock () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.12.1/src/raw_mutex.rs:72
#20 lock_api::mutex::Mutex::lock<parking_lot::raw_mutex::RawMutex, dhat::Phase<dhat::Globals>> () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.9/src/mutex.rs:214
#21 dhat::{impl#7}::alloc () at src/lib.rs:1221
#22 0x000055fe6774a7b7 in alloc::raw_vec::finish_grow<alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/result.rs:2134
#23 0x000055fe6771a152 in alloc::raw_vec::RawVec::grow_amortized<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/raw_vec.rs:400
#24 alloc::raw_vec::{impl#1}::reserve::do_reserve_and_handle<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/raw_vec.rs:285
#25 0x000055fe67748fe8 in alloc::raw_vec::RawVec::reserve<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/raw_vec.rs:289
#26 alloc::vec::Vec::reserve<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/vec/mod.rs:841
#27 alloc::vec::Vec::append_elements<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/vec/mod.rs:1834
#28 alloc::vec::spec_extend::{impl#4}::spec_extend<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/vec/spec_extend.rs:85
#29 alloc::vec::Vec::extend_from_slice<u8, alloc::alloc::Global> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/vec/mod.rs:2283
#30 alloc::string::String::push_str () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/string.rs:925
#31 alloc::string::{impl#63}::write_str () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/string.rs:2818
#32 0x000055fe67784d70 in core::fmt::num::imp::fmt_u64 () at library/core/src/fmt/num.rs:273
#33 core::fmt::num::imp::{impl#7}::fmt () at library/core/src/fmt/num.rs:287
#34 0x000055fe677493ab in alloc::string::{impl#40}::to_string<usize> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/string.rs:2489
#35 scheduled_thread_pool::{impl#7}::new_inner::{closure#0} () at src/lib.rs:158
#36 core::option::Option::map<&str, alloc::string::String, scheduled_thread_pool::{impl#7}::new_inner::{closure_env#0}> () at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:929
#37 scheduled_thread_pool::ScheduledThreadPool::new_inner () at src/lib.rs:158
#38 0x000055fe67742f14 in moka::common::concurrent::thread_pool::ThreadPool::new () at src/common/concurrent/thread_pool.rs:38
#39 moka::common::concurrent::thread_pool::{impl#3}::acquire_pool::{closure#0} () at src/common/concurrent/thread_pool.rs:78
#40 std::collections::hash::map::Entry::or_insert_with<moka::common::concurrent::thread_pool::PoolName, alloc::sync::Arc<moka::common::concurrent::thread_pool::ThreadPool>, moka::common::concurrent::thread_pool::{impl#3}::acquire_pool::{closure_env#0}> ()
    at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/collections/hash/map.rs:2549
#41 moka::common::concurrent::thread_pool::ThreadPoolRegistry::acquire_pool () at src/common/concurrent/thread_pool.rs:73
#42 0x000055fe6772ef5b in moka::common::concurrent::housekeeper::ThreadPoolHousekeeper::new<moka::sync_base::base_cache::Inner<alloc::string::String, alloc::string::String, std::collections::hash::map::RandomState>> () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/moka-0.9.4/src/common/concurrent/housekeeper.rs:242
#43 moka::common::concurrent::housekeeper::Housekeeper::new<moka::sync_base::base_cache::Inner<alloc::string::String, alloc::string::String, std::collections::hash::map::RandomState>> () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/moka-0.9.4/src/common/concurrent/housekeeper.rs:73
#44 0x000055fe677383f8 in moka::sync_base::base_cache::BaseCache::new<alloc::string::String, alloc::string::String, std::collections::hash::map::RandomState> () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/moka-0.9.4/src/sync_base/base_cache.rs:185
#45 0x000055fe6772ab9b in moka::sync::cache::Cache::with_everything<alloc::string::String, alloc::string::String, std::collections::hash::map::RandomState> () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/moka-0.9.4/src/sync/cache.rs:856
#46 0x000055fe67725e42 in moka::sync::builder::CacheBuilder::build<alloc::string::String, alloc::string::String> () at /home/cstrahan/.cargo/registry/src/github.com-1ecc6299db9ec823/moka-0.9.4/src/sync/builder.rs:141
#47 0x000055fe6772b530 in jemalloc_test::main () at src/main.rs:13

So I tried swapping parking_lot::Mutex with std::sync::Mutex, and the problem went away. So I suspect there's a very subtle bug in parking_lot's Mutex implementation.

I'd be more than happy to provide any additional information that could be helpful for diagnosing this.

Amanieu commented 2 years ago

I think you are hitting the same issue as #305, for which there is no real good solution. The problem is that parking_lot itself needs to allocate memory, which is a problem if the allocator uses parking_lot.

cstrahan-blueshift commented 2 years ago

Ah, I see! I'll explore resolution from dhat's end then, thanks!