VTOL Crash 02/01/18

[r0gelion]

We flew a VTOL (Delta - Quad) with Pixhawk 1 (no silicon errata), executed auto (mission) the VTOL climbed, made the transition and reached the first waypoint, then it started to fly in a straight line, tried to recovered and activated kill switch but it continued flying and eventually crashed.

Firmware loaded was master (102c0ea)

Here is the log: https://review.px4.io/plot_app?log=7b00d0db-adf7-4a22-a9ba-8f8d55bb0419

Thank you!

[PX4BuildBot]

GitMate.io thinks the contributor most likely able to help you is @sanderux.

Possibly related issues are #7675 (Some questions about “mc_att_control” hope to get help), #6788 (Some questions after a crash..), #5700 (How to control drones with companion computer?), #7390 ('fw_pos_control_l1 stop' crashes the system in v1.6.2), and #7719 (suddenly drone was fall ...).

[dagar]

I briefly looked at this log already shortly after it was uploaded, it's an estimator issue. I'll take a closer look to identify the root cause. @priseborough FYI

Notice the jump in altitude estimate.

[dagar]

This jump in barometric pressure is odd.

[dagar]

No ms5611 errors registered by the perf counter.

[RomanBapst]

@r0gelion @dagar I had a look at the log and came to the following conclusion (see figure below). Just a few seconds before the vehicle changed the status from rotary wing to fixed wing the navigation state changed from AUTO to POSCTRL. I checked the parameters and you didn't have the POSCTRL switch mapped so the only way for you to have switched is via the tablet. This made me suspect that you might have switched the modes by accident when you were using the tablet to emit the transition command? So while you were assuming the missing flew away in AUTO mode is was just flying a straight line just like it supposed to be in POSCTRL.

[RomanBapst]

And using the kill switch in fw mode has the disadvantage that the servos probably go to neutral and the plane can still glide.

[sanderux]

@RomanBapst if the vehicle was commanded to transition from mission, switching to POSCTL may have commanded it to backtransition

[RomanBapst]

@sanderux The problem is that the mode switch happened 0.4 seconds before the transitions command came :)

[RomanBapst]

So the vehicle started the transition in POSCTRL

[sanderux]

right, new QGC has a transition command interface. did the radio command anything?

[RomanBapst]

no, POSCTRL was not even mapped on the radio

[dagar]

@RomanBapst that's for the old/advanced style switch config.

The new simple switch config (default) uses these slots.

    uint8 MAIN_STATE_POSCTL = 2

[dagar]

Why was the kill switch activated in air?

[RomanBapst]

@dagar Affirmitive

[RomanBapst]

Oops, read it as a question, sorry

[priseborough]

Apologies for missing this query when it was posted. There is no estimator failure, however the baro sensor data has a step pressure change that persists until logging stops. This causes the reset event

2:54:085 | WARNING | [lib__ecl] EKF baro hgt timeout - reset to baro

This coincides with a large disturbance in reported baro temperatue:

There is no indication of a change in the 5v supply.

This indicates either a HW fault in the baro or a change in the baro supply voltage. I would not be flying this board.

[RomanBapst]

I assumed the pilot did not know the plane was in posctrl and thought the plane was doing a flyaway from the mission. But only the pilot can tell us what really happened

[r0gelion]

@RomanBapst According to the pilot, the POSCTRL was mapped to the RC, but the flight started in auto, the VTOL was on the ground and the pilot activated AUTO, it might pass the switch by POSCTRL but the VTOL took off in auto. The pilot didn't command the POSCTRL with the RC (unless not that he was aware) and, since he was using a laptop, he didn't command POSCTRL from QGC.

Once he saw the vehicle was flying away from the mission it activated the kill switch.

@priseborough The autopilot used was purchased recently, is there a way to confirm the board is defective?

Thank you!

[r0gelion]

@RomanBapst @priseborough

We had another incident with the VTOL where it flipped when taking off in auto. https://review.px4.io/plot_app?log=1bd9f8e0-b286-48ea-890c-9abd5d01065c

Here are other logs where it took off but it didn't do the transition to plane: https://review.px4.io/plot_app?log=cf6c3278-15bf-4b29-a036-5cb24e3fb81a https://review.px4.io/plot_app?log=bcfe5e4a-8b4a-48d4-a7e7-f22659f1c3b4 https://review.px4.io/plot_app?log=ac5c0da7-6453-4160-87ee-035378610bbe

This VTOL used a Pixhawk 3 Pro, but we replaced that autopilot with a Pixhawk 1 (no silicon errata), but we haven't had a successful flight since then.

[LorenzMeier]

@priseborough Could you comment on the current state of things?

[priseborough]

The original posted log showed the estimator performance was nominal and the root cause was a step change in raw Baro output. I will inspect the recent logs and report back.

[sanderux]

Possibly related issue, a very sudden glitch in heading during a vtol transition: Note: this vehicle was flown after a DIY repair after a pilot error crash so not ruling out hardware failure.

https://logs.px4.io/plot_app?log=3b10b618-c1b6-4647-bace-fe73c0953506

[priseborough]

@r0gelion Can you please indicate which logs were flown with the Pixhawk 3 Pro and which were flown with the Pixhawk 1. The logs show that all of the last four logs flown were with the Pixhawk 1 board, so does that mean they are all on the same board and are the first flights on that airframe with that board installation?

[priseborough]

@r0gelion The https://review.px4.io/plot_app?log=1bd9f8e0-b286-48ea-890c-9abd5d01065c log shows the inertial nav solution has diverged before logging starts and the estimator is going through a series of reset cycles. There is no evidence of estimator time slippage. I would require pre-arm logging data to identify the root cause. The pre-arm checks should be inspected to determine why they allowed the vehicle to arm when the estimator was reporting such large errors.

The other three logs have reasonable innovation levels and show no evidence of estimation problems.

[priseborough]

@sanderux The mag field length shows large fluctuations indicating a problem with calibration of the mag sensor or magnetic interference that needs to be fixed.

[priseborough]

Please preform a mag calibration and verify the steady state heading is correct at four 90 degree separated compass points (eg pointing N,E,S,W) before flying again.