Closed Dirbaio closed 3 years ago
I would have expected infinite loops like these to be caught by this check. Would be interesting to log LR, PC and other registers as probe-run unwinds the stack.
I'm seeing this behavior with -C force-frame-pointers=no.
did you compile the whole Rust code with -C force-frame-pointers=no
or is it just the assembly (asm.rs
?) what was compiled w/o frame pointers? The backtrace looks fine for the first 17 frames or so.
Hi @Dirbaio, Could you please provide us with some code to reproduce this error? Thanks in advance! π
Attached ELF: repro.tar.gz Built from this commit https://github.com/akiles/embassy/commit/f8c434400393d74fb51acb130bbaaf835e7edf38
[dirbaio@mars embassy-nrf-examples]$ probe-run --version
0.2.1
supported defmt version: 0.2
[dirbaio@mars embassy-nrf-examples]$ rustc --version
rustc 1.52.0-nightly (a15f484b9 2021-02-22)
[dirbaio@mars embassy-nrf-examples]$ cargo build --bin rtc_async
Finished dev [optimized + debuginfo] target(s) in 0.05s
rustc 1.52.0-nightly (a15f484b9 2021-02-22)
[dirbaio@mars embassy-nrf-examples]$ probe-run --chip nRF52840_xxAA ../target/thumbv7em-none-eabi/debug/rtc_async
(HOST) INFO flashing program (11.52 KiB)
(HOST) INFO success!
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
0 INFO Hello World!
ββ rtc_async::__cortex_m_rt_main @ src/bin/rtc_async.rs:42
1 INFO tick
ββ rtc_async::run2::task::{{closure}} @ src/bin/rtc_async.rs:31
2 INFO BIG INFREQUENT TICK
ββ rtc_async::run1::task::{{closure}} @ src/bin/rtc_async.rs:22
3 INFO tick
ββ rtc_async::run2::task::{{closure}} @ src/bin/rtc_async.rs:31
4 INFO tick
ββ rtc_async::run2::task::{{closure}} @ src/bin/rtc_async.rs:31
5 INFO tick
ββ rtc_async::run2::task::{{closure}} @ src/bin/rtc_async.rs:31
6 INFO tick
ββ rtc_async::run2::task::{{closure}} @ src/bin/rtc_async.rs:31
stack backtrace:
0: HardFaultTrampoline
<exception entry>
1: rtc_async::run1::task::{{closure}}
at src/bin/rtc_async.rs:24
2: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /home/dirbaio/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:80
3: embassy::executor::Task<F>::poll
at /home/dirbaio/akiles/embassy/embassy/src/executor/mod.rs:71
4: embassy::executor::timer_queue::TimerQueue::update
at /home/dirbaio/akiles/embassy/embassy/src/executor/timer_queue.rs:35
5: embassy::executor::raw::Executor::run_queued::{{closure}}
at /home/dirbaio/akiles/embassy/embassy/src/executor/raw.rs:135
6: embassy::executor::run_queue::RunQueue::dequeue_all
at /home/dirbaio/akiles/embassy/embassy/src/executor/run_queue.rs:66
7: embassy::executor::raw::Executor::run_queued
at /home/dirbaio/akiles/embassy/embassy/src/executor/raw.rs:117
8: cortex_m::asm::wfe
at /home/dirbaio/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.7.1/src/asm.rs:49
9: embassy::executor::Executor::run
at /home/dirbaio/akiles/embassy/embassy/src/executor/mod.rs:202
10: cortex_m::asm::wfe
at /home/dirbaio/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.7.1/src/asm.rs:49
11: embassy::executor::Executor::run
at /home/dirbaio/akiles/embassy/embassy/src/executor/mod.rs:202
12: cortex_m::asm::wfe
at /home/dirbaio/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.7.1/src/asm.rs:49
13: embassy::executor::Executor::run
at /home/dirbaio/akiles/embassy/embassy/src/executor/mod.rs:202
14: cortex_m::asm::wfe
at /home/dirbaio/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.7.1/src/asm.rs:49
15: embassy::executor::Executor::run
at /home/dirbaio/akiles/embassy/embassy/src/executor/mod.rs:202
16: cortex_m::asm::wfe
at /home/dirbaio/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.7.1/src/asm.rs:49
.... infinite loop
Thank you @Dirbaio for providing the reproducable!
Jonas mentioned in #163
that following condition is likely the cause for this bug: https://github.com/knurling-rs/probe-run/blob/d7029443bc91860d316efc39d547a5313264cbdc/src/main.rs#L801
reporting back for the record and also as a note for myself for when I pick this back upβ
the problem is indeed that stack_corrupted
doesan't paint the full picture, but this one is tricky to detect β after 9:
, the program counter doesn't change (checked by lr & !THUMB_BIT == pc & !THUMB_BIT
), but the CFA (Canonical Frame Address)
keep shifting, as the HardFault occurs in a loop https://github.com/embassy-rs/embassy/blob/f8c434400393d74fb51acb130bbaaf835e7edf38/embassy/src/executor/mod.rs#L202 .
We can't just make the &&
a ||
because in other cases like for example.... several recursive function calls, this is a perfectly legal state that should not interrupt the backtrace printing.
We've checked if gdb does some detection magic, but it seems to run into the same problem:
(paging prevents the endless scrolling here though, which is nice)
As a band-aid, I'll implement a backtrace line limit after which the backtrace is cut off (length reconfigurable by flag) to prevent it from scrolling forever. If we're feeling fancy we could additionally cut out the middle and only print, say, the top 10 and bottom 10 lines separated by a [...]
by default for better glance-ability.
Closing this now due to #179 being mergedβ @Dirbaio , feel free to re-open if you disagree or have new input
I'm seeing this behavior with
-C force-frame-pointers=no
.I think it's to be expected that backtracing doesn't work correctly with it, but I think at least this should be detected and fail with
error: the stack appears to be corrupted beyond this point
instead of looping forever.If there's interest I can try cooking a binary that reproduces this.