search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.52k stars 13.51k forks source link

PX4 software crashes and Pixhawk restarts mid-air #16390

Closed Chris-Bee closed 3 years ago

Chris-Bee commented 3 years ago

Describe the bug We are using a Pixhawk Holybro V4 for a multicopter setup. It seems that the platform restarts mid-air which causes the vehicle to crash (Motors stop and the Pixhawk reboots immediately). These test flights are done in a MoCap environment and we are using the internal EKF2 by feeding the MoCap pose at 30Hz via the mavlink vision pose topic. Each individual flight produces 3 logfiles:

I'm not sure about the smaller logfiles before and after the flight test but I attached them anyway. The second file however points out a software crash. We are using the latest and unmodified firmware release (v1.11.2) and I can not see a clear cause for the error in the log file.

To Reproduce Steps to reproduce the behavior:

  1. We are using the internal EKF2 estimator and feed 30Hz MoCap pose information via the mavlink vision topic
  2. Takeoff and let the copter hover in position control for a while
  3. In our case, the Pixhawk restarts after 20 seconds or 2 minutes (this varies)
  4. Motors turn off and the multicopter performs a hard landing

Expected behavior The PX4/Pixhawk V4 should not restart

Log Files and Screenshots

Drone (please complete the following information):

Additional information I'm not sure, but this could be related to issue #14075

dayjaby commented 3 years ago

We had a similar issue some weeks ago. Our current analysis is that the SD card must be very good, otherwise PX4 hangs up sometimes. Can you verify that by testing sd_bench -r 50 in the MAVLink console/NuttX shell and see whether your maximal block writing times are ~20ms?

dagar commented 3 years ago

@Chris-Bee that second file shows a hard fault in the init task. Do you have any modifications on top of PX4 v1.11.2? Are you using PX4 flashed through QGC or did you build it yourself?

@dayjaby Can you open a separate issue with any discoveries of SD card issues? At a bare minimum we should be able to handle these failures gracefully across missions, etc.

Chris-Bee commented 3 years ago

We had a similar issue some weeks ago. Our current analysis is that the SD card must be very good, otherwise PX4 hangs up sometimes. Can you verify that by testing sd_bench -r 50 in the MAVLink console/NuttX shell and see whether your maximal block writing times are ~20ms?

@dayjaby thank you for the fast response, sd_bench -r 50 indeed shows max writing times between 30-900ms, we performed a test without the SD card mounted, and the issue persists.

Chris-Bee commented 3 years ago

@Chris-Bee that second file shows a hard fault in the init task. Do you have any modifications on top of PX4 v1.11.2? Are you using PX4 flashed through QGC or did you build it yourself?

@dagar We did not modify the PX4 v1.11.2 firmware. For the logged flight, we used QGC (v3.5.6) to flash the firmware directly, we did not compile the firmware our self. Since the QGC version is a bit older (latest working with ubuntu 16.04), we just used the latest QGC version and reset all parameters to prevent version/parameter set discrepancies. This is still in progress. Let me know if there is something else we can test to further debug the issue.

Chris-Bee commented 3 years ago

The general cause of the problem is found. We have two copters with supposedly the same configuration. One showed this issue and the second copter did not. We checked all the wiring and the vehicle which showed the issue had 5V connected to the FMU DEBUG port (GND and RX) pins. I'm not sure why this was done by the manufacturer. Anyway, we disconnected the FMU DEBUG connector and the issue seems to be gone. This is verified by 4 successful tests without debug connected, and one non-successful test with debug connected as a sanity check.

I assume the odd input to the FMU debug RX pin caused the software to crash.

The vehicle that caused the issue is the same as mentioned in issue #14075, I guess this can be closed as well.

LorenzMeier commented 3 years ago

5V to GND is electrically a situation where all guarantees on the system behavior are off. I'm closing this issue accordingly.

Chris-Bee commented 3 years ago

@LorenzMeier Apologies, what I wrote was incorrect - Supply GND was connected to GND of the FMU Debug port and 5V was connected to the RX pin of the FMU Debug port.