dagar
commented
2 years ago Until we have a better understanding I would try to collect enough information to debug these as separate issues.
For H7 there was an alignment fix that went into v1.13. https://github.com/PX4/PX4-Autopilot/pull/19724
- Cube Orange
- can you reproduce with v1.13 or current main branch?
- Kakute f7
- would you be able to connect a serial console (and possibly debugger) to take a closer look?
- nuttx shell crashing could be completely unrelated (we might need to increase the stack)
Etruscian
Describe the bug
We're using a build server that uses the official docker containers to build the firmware and does a clean pull every time a build is started. Lately, we've noticed unpredictable behavior with builds. We're building v1.11.3 with some additions for our own drone that runs a cube orange. We've also tried different machines but they all give the same problems.
Two firmwares where the only difference is the values of some tuning parameters in the start up script behave very differently. The version before the change works fine, no issues. The version after the change gives a lot of UAVCAN errors.
We've noticed that the binaries aren't the same size (the second one is 8 bytes larger). If we take the working version and perform the same shift at the same location in the flash before uploading (with modifying the address pointers so the binaries are identical apart from git hash and build timestamps), the exact same problem occurs.
The difference of 8 bytes is determined to be due to the timestamp of compilation of certain c files being set in the binary. If this is the same timestamp (second accuracy), they all point to 1 timestamp in the binary. if these are different, 9 bytes get added to store the second timestamp. This timestamp was added at 0x139872 (relative to 0x08020000) and because of alignment, a single padding byte is removed at 0x166b8b. These offsets might be case specific, but might also indicate a problem somewhere.
Another way we notice this is that a later build with also minimal changes seems to freeze at touch down. We've made ram dumps and verified with a debugger that at least most of the PX4 code is still running (i.e. interrupts and at least some threads are still running). There is just no communication with the outside world. UAVCAN stops sending messages, uarts stop sending messages and the logger stops as well.
Using the ram dumps, we were able to verify that the incoming messages are still put in the rx buffers of the uarts (which indicates that the peripherals and DMA's are still running when the outward communication stopped). The tx buffer of the telemetry radio for instance contains only all the messages up to the last one we received over telemetry. We've also checked the BSRAM to see if any hard faults are logged but it was empty.
I personally also have a PX4 drone that runs 1.13 using a holybro kakute f7 with some modules turned off to enable the ekf. Minor changes in these builds also result in unpredictable behavior where sometimes the gnss driver refuses to start or the entire nuttx shell stops working. Trying to communicate to the nuttx shell using qgroundcontrol then results in a hard crash of the flight controller where the usb connection is stopped by the flight controller. These firmwares are built on a completely different machine.
At first we were thinking of it being due to our code changes, but since it is also happening for my personal drone with different hardware and we've heard other people in slack noting the same problems, we believe that something is fundamentally going wrong in the build process.
How can we best proceed to further investigate the cause of this issue?