Closed tttapa closed 3 weeks ago
I'm really "happy" to see this issue here because I had the same problem 20 minutes ago concerning LTO (wrapper functions not found). So yeah, maybe it could be cool to add an option to the pico sdk to support LTO (something like PICO_SDK_LTO
which would add -DPICO_SDK_LTO
to the compilation command-line to conditionally define WRAPPER_FUNC
to __attribute__((used)) __wrap_ ## x
). However, I guess inconsistent behavior when enabling LTO may rather be caused by unwanted optimizations (I don't know exactly how nor why) or wrong code (but I think pico-sdk's code has been sufficiently tested to verify such things) rather than data races (at least in this case, since there's apparently no multicore code nor multithreading possibilities)
Note #97 for LTO
I understand that LTO/CMAKE_INTERPROCEDURAL_OPTIMIZATION is not officially supported, however, the fact that it fails with LTO enabled may point to an underlying problem such as a data race or another source of undefined behavior.
are you saying it doesn't fail without LTO
However, I guess inconsistent behavior when enabling LTO may rather be caused by unwanted optimizations (I don't know exactly how nor why) or wrong code (but I think pico-sdk's code has been sufficiently tested to verify such things)
In my experience, what you refer to as "unwanted" optimizations have almost exclusively been caused by invalid code invoking Undefined Behavior.
For correct code without UB, the compiler and the optimizer should not alter the observable behavior of the code (for the standard's definition of observable). As such, "unwanted" optimizations are not really a thing, it's just that if you write code that violates the preconditions of the optimizer, the result won't satisfy its postconditions either. This is of course ignoring the real possibility of compiler bugs, but those are relatively rare, and shouldn't usually be the first hypothesis.
UB is unfortunately not something you can rule out by testing: because of its very nature, problems may only come up in certain scenarios or at sufficiently high optimization levels.
rather than data races (at least in this case, since there's apparently no multicore code nor multithreading possibilities)
True, but in the cases where I encountered the bug, there have always been timer interrupts involved. In the example code I posted, the state of the timer/alarm pool has to be updated both from the main thread and in the interrupt handlers. Combined with the nondeterministic behavior, a data race doesn't seem unlikely, but there could of course be other causes.
are you saying it doesn't fail without LTO
Indeed. Or at least, I haven't observed the same glitch (yet?) without LTO enabled. Given the sporadic nature of the bug, it may just be more likely with LTO enabled, but not impossible without.
Even with LTO enabled, it occasionally takes two or three tries to reproduce the issue.
@tttapa yeah indeed, maybe you're right, especially if it hasn't been tested with LTO enabled before
It could be that enabling/disabling LTO is simply changing the timing and ordering of events, which makes the bug apparent (or not).
We are also seeing a very similar issue in MicroPython and I think we have tracked down the bug to the way the hardware timer target is being configured, sometimes an interrupt is missed and that puts things into an inconsistent state. See #1552.
are you saying it doesn't fail without LTO
I just tried this and it seemed that the issue was ONLY reproducible with LTO
The alarm code was rewritten for the last SDK, please check on that
Below is a simple program that uses a recurring timer interrupt using
add_alarm_in_us
to blink an LED as a liveness check. In the main loop, it simply reads characters from stdin (USB CDC), usessleep_us
, and echos them to stdout.When sending enough data to the Pico, it eventually stops blinking, which should never happen.
The Pico is still alive, but the blink timer interrupt somehow stopped firing. After some time (minutes to hours), it suddenly starts blinking again, so it seems like it's just the alarm period that somehow gets corrupted.
The reason that I wrote this small program is that in a much larger program, I sometimes encountered
sleep_us
hanging in a similar way when the USB traffic was high. I would have asleep_us(100)
somewhere, for example, and I would notice that the Pico would hang on that call for minutes. When attaching a debugger, it turned out that it was indeed waiting inside ofsleep_us
(all relevant variables were optimized out, so I didn't investigate further, this only happens in release mode).main.cpp
CMakeLists.txt
Build
Test
Tools
Pico SDK patch
Required to support LTO (otherwise, the linker can't find the
__wrap_
functions).I understand that LTO/
CMAKE_INTERPROCEDURAL_OPTIMIZATION
is not officially supported, however, the fact that it fails with LTO enabled may point to an underlying problem such as a data race or another source of undefined behavior.