search

tokio-rs / tokio

A runtime for writing reliable asynchronous applications with Rust. Provides I/O, networking, scheduling, timers, ...
https://tokio.rs
MIT License
25.74k stars 2.35k forks source link

tokio panicked at 'inconsistent state in unpark' #2088

Closed nanne007 closed 3 years ago

nanne007 commented 4 years ago

Version

v0.2.6

Platform

ubuntu 18.04

Description

Running test in github workflow, come across the following panic which shows that is caused by https://github.com/tokio-rs/tokio/blob/34e0a2c04d83877e8bb2e0228818ef0592221e24/tokio/src/runtime/park.rs#L206

2020-01-10T10:36:05.5317727Z C0110 10:36:05.531294 140349055829760 libra/common/crash-handler/src/lib.rs:38] details = '''panicked at 'inconsistent state in unpark; actual = 140349330989872', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/park.rs:206:23'''
2020-01-10T10:36:05.5318720Z backtrace = '''
2020-01-10T10:36:05.5319295Z stack backtrace:
2020-01-10T10:36:05.5320160Z    0:     0x564de2cf3f3d - backtrace::backtrace::libunwind::trace::h9534e738497f3e0c
2020-01-10T10:36:05.5320562Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
2020-01-10T10:36:05.5320754Z                            backtrace::backtrace::trace_unsynchronized::hfaeb352168d67f4f
2020-01-10T10:36:05.5321128Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2020-01-10T10:36:05.5321615Z    1:     0x564de2cf3ebe - backtrace::backtrace::trace::h4debebf7a7481f52
2020-01-10T10:36:05.5321976Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:53
2020-01-10T10:36:05.5322449Z    2:     0x564de2cf924d - backtrace::capture::Backtrace::create::hb5d38b1db6575962
2020-01-10T10:36:05.5322809Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/capture.rs:164
2020-01-10T10:36:05.5323276Z    3:     0x564de2cf9165 - backtrace::capture::Backtrace::new::h8d4306883159fabf
2020-01-10T10:36:05.5323635Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/capture.rs:128
2020-01-10T10:36:05.5324252Z    4:     0x564de147405d - crash_handler::handle_panic::h477d3635198860d8
2020-01-10T10:36:05.5324538Z                                at libra/common/crash-handler/src/lib.rs:35
2020-01-10T10:36:05.5325736Z    5:     0x564de14742f8 - crash_handler::setup_panic_handler::{{closure}}::h7de518867292c991
2020-01-10T10:36:05.5326012Z                                at libra/common/crash-handler/src/lib.rs:27
2020-01-10T10:36:05.5326456Z    6:     0x564de2d7eeac - std::panicking::rust_panic_with_hook::h825f041245da8739
2020-01-10T10:36:05.5326586Z                                at src/libstd/panicking.rs:477
2020-01-10T10:36:05.5327023Z    7:     0x564de2d7e962 - std::panicking::continue_panic_fmt::hbe0378e33481e81b
2020-01-10T10:36:05.5327150Z                                at src/libstd/panicking.rs:380
2020-01-10T10:36:05.5327580Z    8:     0x564de2d7e8bc - std::panicking::begin_panic_fmt::hecb5f06c798651bd
2020-01-10T10:36:05.5327713Z                                at src/libstd/panicking.rs:335
2020-01-10T10:36:05.5328158Z    9:     0x564de24bef33 - tokio::runtime::park::Inner::unpark::ha02e092b4769417b
2020-01-10T10:36:05.5328506Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/park.rs:206
2020-01-10T10:36:05.5329001Z   10:     0x564de24be073 - <tokio::runtime::park::Unparker as tokio::park::Unpark>::unpark::hd7249af8e6b13806
2020-01-10T10:36:05.5329347Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/park.rs:110
2020-01-10T10:36:05.5329816Z   11:     0x564de24f1d92 - tokio::runtime::thread_pool::shared::Shared::unpark::hf4cf98328f3f41da
2020-01-10T10:36:05.5330173Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:47
2020-01-10T10:36:05.5330757Z   12:     0x564de24dcd18 - tokio::runtime::thread_pool::slice::Set::notify_all::h180a075f8e729999
2020-01-10T10:36:05.5331154Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:94
2020-01-10T10:36:05.5331662Z   13:     0x564de24dcb2a - tokio::runtime::thread_pool::slice::Set::inject_task::{{closure}}::h9cbab9389e92f74d
2020-01-10T10:36:05.5332016Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:79
2020-01-10T10:36:05.5332476Z   14:     0x564de249cb35 - tokio::runtime::thread_pool::queue::global::Queue<T>::push::h7f2cbb4fa7d6e16c
2020-01-10T10:36:05.5332839Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/queue/global.rs:95
2020-01-10T10:36:05.5333314Z   15:     0x564de24e7e07 - tokio::runtime::thread_pool::queue::inject::Inject<T>::push::h7a400c68fa68f653
2020-01-10T10:36:05.5333682Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/queue/inject.rs:19
2020-01-10T10:36:05.5334330Z   16:     0x564de24dcaa1 - tokio::runtime::thread_pool::slice::Set::inject_task::hdc45be5fc82ea76a
2020-01-10T10:36:05.5334704Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:69
2020-01-10T10:36:05.5335277Z   17:     0x564de24dcdcd - tokio::runtime::thread_pool::slice::Set::schedule::{{closure}}::hafeaf06061189725
2020-01-10T10:36:05.5335645Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:106
2020-01-10T10:36:05.5336124Z   18:     0x564de248e15c - tokio::runtime::thread_pool::current::get::{{closure}}::hf80ceb7d4a5e48bc
2020-01-10T10:36:05.5336470Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/current.rs:61
2020-01-10T10:36:05.5336950Z   19:     0x564de24a3756 - std::thread::local::LocalKey<T>::try_with::h3c32f61a510e7a89
2020-01-10T10:36:05.5337093Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:262
2020-01-10T10:36:05.5337836Z   20:     0x564de24a29c8 - std::thread::local::LocalKey<T>::with::h7cedc4c38b5bdf62
2020-01-10T10:36:05.5337975Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:239
2020-01-10T10:36:05.5338427Z   21:     0x564de248e0e9 - tokio::runtime::thread_pool::current::get::hb6ff52c77f9dfe85
2020-01-10T10:36:05.5338782Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/current.rs:59
2020-01-10T10:36:05.5339401Z   22:     0x564de24dcd51 - tokio::runtime::thread_pool::slice::Set::schedule::h8ade22bce0f639a0
2020-01-10T10:36:05.5339912Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:99
2020-01-10T10:36:05.5340783Z   23:     0x564de24f1d4d - tokio::runtime::thread_pool::shared::Shared::schedule::h5e970150eec6d7fe
2020-01-10T10:36:05.5341166Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:43
2020-01-10T10:36:05.5341694Z   24:     0x564de24f1fdd - <tokio::runtime::thread_pool::shared::Shared as tokio::task::Schedule>::schedule::hd1e7e6cbdf1466a1
2020-01-10T10:36:05.5342070Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:90
2020-01-10T10:36:05.5342556Z   25:     0x564de0f8ac89 - tokio::task::harness::Harness<T,S>::wake_by_ref::h4ca4df98aba2a999
2020-01-10T10:36:05.5342918Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:315
2020-01-10T10:36:05.5344272Z   26:     0x564de0f8bbce - tokio::task::harness::Harness<T,S>::wake_by_val::hbac87b5d7c1936a5
2020-01-10T10:36:05.5344684Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:299
2020-01-10T10:36:05.5345164Z   27:     0x564de1053014 - tokio::task::waker::wake_by_val::hdc8dc10a97fc578f
2020-01-10T10:36:05.5345512Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/waker.rs:84
2020-01-10T10:36:05.5345972Z   28:     0x564de1459202 - core::task::wake::Waker::wake::h7f114068623785d5
2020-01-10T10:36:05.5346093Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libcore/task/wake.rs:252
2020-01-10T10:36:05.5346577Z   29:     0x564de144d9da - futures_timer::atomic_waker::AtomicWaker::wake::h2bea43a6601b0f32
2020-01-10T10:36:05.5346941Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-timer-2.0.2/src/atomic_waker.rs:171
2020-01-10T10:36:05.5348103Z   30:     0x564de1446820 - futures_timer::timer::Timer::advance_to::ha24c9e911b885270
2020-01-10T10:36:05.5348589Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-timer-2.0.2/src/timer.rs:130
2020-01-10T10:36:05.5349195Z   31:     0x564de1446507 - futures_timer::timer::Timer::advance::h964f7e7ff75403c0
2020-01-10T10:36:05.5349635Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-timer-2.0.2/src/timer.rs:105
2020-01-10T10:36:05.5350214Z   32:     0x564de1453754 - futures_timer::global::run::h27f8a7d5c4943b5a
2020-01-10T10:36:05.5350651Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-timer-2.0.2/src/global.rs:65
2020-01-10T10:36:05.5351579Z   33:     0x564de145333c - futures_timer::global::HelperThread::new::{{closure}}::h26b4f8a98677c40d
2020-01-10T10:36:05.5352091Z                                at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-timer-2.0.2/src/global.rs:28
2020-01-10T10:36:05.5352565Z   34:     0x564de1455192 - std::sys_common::backtrace::__rust_begin_short_backtrace::hddeec95e4aa828ce
2020-01-10T10:36:05.5352822Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/sys_common/backtrace.rs:126
2020-01-10T10:36:05.5353496Z   35:     0x564de1455d31 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::hd311b801044c2a0d
2020-01-10T10:36:05.5353628Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/mod.rs:470
2020-01-10T10:36:05.5354284Z   36:     0x564de1455151 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h550cd838e65c2311
2020-01-10T10:36:05.5354420Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panic.rs:315
2020-01-10T10:36:05.5354863Z   37:     0x564de145613b - std::panicking::try::do_call::h9e8134b376e1f731
2020-01-10T10:36:05.5354997Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:292
2020-01-10T10:36:05.5355271Z   38:     0x564de2d8318a - __rust_maybe_catch_panic
2020-01-10T10:36:05.5355379Z                                at src/libpanic_unwind/lib.rs:80
2020-01-10T10:36:05.5355807Z   39:     0x564de1455fa0 - std::panicking::try::h7791b956a2dc6551
2020-01-10T10:36:05.5355935Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:271
2020-01-10T10:36:05.5356362Z   40:     0x564de14551d3 - std::panic::catch_unwind::h7508a664d18b5124
2020-01-10T10:36:05.5356491Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panic.rs:394
2020-01-10T10:36:05.5356947Z   41:     0x564de1455b2a - std::thread::Builder::spawn_unchecked::{{closure}}::h8f1f5ac0ac466ffe
2020-01-10T10:36:05.5357165Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/mod.rs:469
2020-01-10T10:36:05.5357667Z   42:     0x564de1456234 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h71506c2b23e4fc09
2020-01-10T10:36:05.5357810Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libcore/ops/function.rs:227
2020-01-10T10:36:05.5358300Z   43:     0x564de2d6fd8f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h483711add4ba2330
2020-01-10T10:36:05.5358441Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/liballoc/boxed.rs:922
2020-01-10T10:36:05.5358925Z   44:     0x564de2d824f0 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h7605b45eb29ed0be
2020-01-10T10:36:05.5359064Z                                at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/liballoc/boxed.rs:922
2020-01-10T10:36:05.5359196Z                            std::sys_common::thread::start_thread::h557b0c2dc4449562
2020-01-10T10:36:05.5359326Z                                at src/libstd/sys_common/thread.rs:13
2020-01-10T10:36:05.5359458Z                            std::sys::unix::thread::Thread::new::thread_start::h7c2a7f9b68fe4bba
2020-01-10T10:36:05.5359568Z                                at src/libstd/sys/unix/thread.rs:79
2020-01-10T10:36:05.5359836Z   45:     0x7fa5dfd576db - start_thread
2020-01-10T10:36:05.5360083Z   46:     0x7fa5df86888f - __clone
2020-01-10T10:36:05.5360507Z   47:                0x0 - <unknown>
2020-01-10T10:36:05.5360908Z '''

I have little experience on low level concurrent programing. Have to post it here for help.

carllerche commented 4 years ago

Do you have a way to reproduce this?

nanne007 commented 4 years ago

It's the only case by now. Will post back if it happen again. Maybe it is caused by underlining system(the test was run in a virtual machine)

cynecx commented 4 years ago

@lerencao What @carllerche meant is, whether you could provide us a small rust-project with code which can reproduce this issue. Having just a backtrace in this case is not really useful in finding the cause for this.

nanne007 commented 4 years ago

@cynecx @carllerche sorry for my misunderstanding. I'm afraid that I had failed several times, and it's the only case. Have never come across the error that the atomic has a totally different value(while only flags value of a fixed set is expected). I thought maybe you guys have experienced this.

Darksonn commented 3 years ago

Closing due to lack of minimal example.