PX4 1.7.3: Total crash in loiter.

[tubeme]

Hello,

We've had a total crash today with 1.7.3 when we engaged loiter in FW flight. We could not find the reason behind this crash in the Log...

While we were flying 1.7.3 for a week we've had no unexpected problems until today.

VTOL deltaquad PX4 1.7.3 QGC 3.3 Windows AUAV X2.1 HW Without airspeed sensor

When the pilot activated Hold the plane just nose dived without any control whatsoever. During the nose dive the pilot switched to MC flight but did not switch back the loiter switch, so it did not work as expected to safe the VTOL. (We did not test this but what is happening when we switch the transition switch while in loiter?) There are no HW problems with the machine.

LOG: https://logs.px4.io/plot_app?log=89bd8fa6-6506-4e9a-87d5-81ee2cc9243a

The log seems to be cut at the end... but the video shows it all.

Video: https://www.youtube.com/watch?v=7iCPhtmdQq8&feature=youtu.be

Any ideas?

[dagar]

What's the history of this vehicle? Does it have previous successful flights manual/stabilized/loiter/mission/etc?

[dagar]

The pink line is the manual control pitch input. Why is it close to max during manual flight? It seems like you have an enormous TRIM missing. In manual you were flying with about 80% throttle, but when starting an AUTO mode TECS initializes at FW_THR_CRUISE (50%), which seems way too low for this vehicle.

[tubeme]

NO the pitch is because we are climbing to a safe altitude in the beginning of the flight, so we push manually the throttle to about 80% for the climb. Once we reach the desired altitude we engaged the loiter.

The motor is with a very very good ratio for the plane, even in this case the plane does not carry camera so it is even lighter.

In this configuration the plane can fly perfectly well with a FW_THR_CRUISE (42%) tested and guaranteed. We usually fly it in auto modes with 60% but this time we used 50%. With 50% throttle the plane is flying just perfect, we have a headroom of 10% throttle until we reach the problematic 40% throttle. At 40% throttle we begin to see swings (too low speed).

The CG of the plane is perfect as well, because we test it in each flying session prior the main test flights.

Is there possibility to have a relation with the TECS problems we've experienced? https://github.com/PX4/Firmware/issues/9078

[tubeme]

The plane did not have a single crash for two months with 1.6.5 and one week with 1.7.3 until today.

All modes tested to their extremes! we've had about 45 flight sessions with average about 4-6 flights each session. We've had also about 30 survey missions with very expensive camera. Thanks god we've had the camera removed for this flight...

[dagar]

Can you post a good log for comparison?

[RomanBapst]

Things checked so far

• [x] nothing visible in system power
• seems like pitch control was starting to degrade shortly before switch to loiter
• [x] cpu and RAM usage seem ok
• log stops 1.5 seconds after switching into loiter when vehicle was still in the air
• [x] controller demanded pitch up and servo signals reflect the intention -> still plane could not hold pitch

[dagar]

From the video are you able to discern if the left elevon is moving? It looks fixed to me, but it's hard to tell.

If you have the original video and can step through frame by frame see if there's any movement during the dive.

[RomanBapst]

@tubeme I was watching the video and I think you can clearly see how the elevon close to the camera goes into the up position due to the pitch up command from the controller. However, I can't really see if the servo on the other side is really doing anything.

[RomanBapst]

Also I think you can see how the rate controller was still doing something during the dive because the elevon close to the camera starts to oscillate which is normals since you don't have an airspeed sensor and thus no scaling takes place.

[tubeme]

Good flight.

https://review.px4.io/plot_app?log=c3b1a88d-227a-4a18-aa17-067c680591ec

[dagar]

I'd say it's not moving. There's a roll rate everytime the pitch rate controller commands a change.

Roll actuation is trying to compensate the entire time.

[tubeme]

@RomanBapst I thought the same thing, if the other elevon stop working. But then the pattern in

Should look different if servo malfunction. They should clip. Especially the broken servo channel. But here servos are just ok...

[RomanBapst]

Well, we the other huge question is why did the logging all of a sudden stop? Doesn't sound like a coincident does it?

[RomanBapst]

@tubeme I assume it can't be something stupid like the airflow pushing in- and out the SD card? It seems like you have something mounted on top of the wing.

[RomanBapst]

@tubeme Also keep in mind that you don't see any clipping because the log stops too early. At the time the log stops the pitch error is not very large but clearly in the video you can see it decreasing until it's nose down. For the vehicle to pitch down like that we can come up with several failure options:

• one servo stopped working, the other one was pulling the elevon up pretty far as seen on the video
• something came loose in the plane, e.g. battery moving forward and the elevons did not have enough authority
• the log stops all of a sudden which could hint to a autopilot failure

[tubeme]

@RomanBapst That is the GPS stand because of the MAG we had to move it away from electronics.

Agree this stopping of the logging is strange... But if it is related to the SD card this is strange too...

Also I thought that this might be the problem that we are not seeing the clipping. But why the log stopped on the first place...

From the video it is visible that despite the logging stopped the pilot switched back to MC mode and tried to recover the quad just until the plane crashed. So the autopilot somehow worked at that time, but not logging for some reason and this led to some other failure??

The batteries are secured well.

[dagar]

Verifying the wiring would be another valuable clue. If the right elevon is output 4 (as stated in the 13006 airframe configuration), then it's the left elevon (output 5 - orange) that's supposedly moving wildly according to the log.

[RomanBapst]

@tubeme I assume the SD card was still in the autopilot after the crash?

[tubeme]

@RomanBapst It was!

[dagar]

14:40:56

14:41:01

14:41:08

14:41:14

14:41:23

14:41:39

14:41:57

[dagar]

The loss of a few seconds of logging is concerning, but it's something I've seen before. The pixhawk logger buffer is quite small (usually around 16 kB depending on configuration), and the default logging rate is probably ~ 40 kB/s.

So I'm wondering if it's corruption of the fat filesystem.

[dagar]

Have you powered up the pixhawk since? If not, it's still worth a try with an sdcard in and serial console attached. If the autopilot actually locked up or crashed the hard fault logger should have caught something. If the timing was right (aka wrong) it might be possible that the hardfault wasn't written to the sdcard before impact.

[RomanBapst]

@tubeme @dagar Something you can clearly see is that in the log from the crash you have a constant offset between roll_rate_setpoint and roll_rate (I'm talking bodyrate). Btw, not that it's relevant for you crash but your rate I gains are quite small, I think you can increase those in the future.

[dagar]

@tubeme any chance you have the corresponding mavlink log from QGroundControl?

[tubeme]

@dagar We powered it up. Even made some tests on the camera trigger.

How can I get this hard fault log?

[dagar]

How can I get this hard fault log?

Check for any additional log files on the sdcard.

[tubeme]

@dagar @RomanBapst We just tested the servos with bottles of water on top. They push very hard and are in perfect condition. Our way of wiring on our controller is such that there is no possibility to have a faulty connection.

For worse we don't have the other logs by mistake and we did not save the mavlink log...... :( stupid mistake

[dagar]

Yes please create another issue if you'd like to pursue it. Showing side by side logs with that type of corresponding explanation can be quite helpful.

For this issue one crude thing you could do on the bench with everything set up the same (camera same location and recording) is manually move the control surfaces and call out the corresponding PWM values. Some simple calibration mapping which motor back to which servo output, and rough PWM ranges would put the video in context. Watching it yet again I still find it hard to believe the left elevon is moving at all.

[tubeme]

I watched the video at least 20 times and the servo is moving.

Here is another video that is with better light condition and you can better see the oposit elevon moving. Then again watch the crash video and you will se that the servo is moving. The camera is wide angle.

https://www.youtube.com/watch?v=HpgNfv1rG44&feature=youtu.be

[dagar]

Actually in that video I can definitely see it moving, and it's easier to see in motion than these still frames. In the crash video the left elevon servo still looks rock solid to me.

17:09:42

17:09:53

17:10:23

17:10:24

Can you verify the motor connections at the time of the crash? https://logs.px4.io/plot_app?log=89bd8fa6-6506-4e9a-87d5-81ee2cc9243a

Was output 4 the left or right elevon servo? Was output 5 the left or right elevon servo?

[Antiheavy]

integrator windup?

[tubeme]

I can confirm that 4 (ch5) is the elevon close to the camera. and 5 (ch6) is the elevon far from the camera.

ie. it is like in the original configuration so no reverse was used.

[tubeme]

@dagar You can better see the elevon's movement at the moment of transition to both flights, the crash one and the previous one.

What do you mean by motor connections? If you mean the pusher, it just flew out and was about 10m away from the plane.

[tubeme]

Here is the original video:

https://drive.google.com/file/d/1JfpMQiwmufwSKTfRoeub6MdiUOEhdklv/view?usp=sharing

I Played it with VLC and slowed it down to 35%. You can clearly see the eleveon moving. I can see that there is a critical moment regarding the pitch. Once we climbed out and enabled loiter I can clearly see that the elevons are making right turn movement, the close elevon is deflecting and the opposite one is pushing down. But the movement of the elevons and the command is not strong enough to make the turn, When the plane starts to pitch down, the command in the elevons continues to be right turn and I don't see a strong pitch up command to compensate and hold the altitude. Then very shortly after that because of the speed we see the flutter...

[tubeme]

Could FW_P_LIM_MAX = 10 be the problem?

[LorenzMeier]

Depending on how the airframe is tuned (controller gains) and the mixers are ordered its possible that it ran into either controller output or mixer saturation.

[tubeme]

Thanks @LorenzMeier. We dedicated a lot of time on tuning and the plane is performing good to our taste.

The only thing that really changed since recently is the FW_P_LIM_MAX = 10 from FW_P_LIM_MAX = 25.

We did this to fight #9078

We are really stuck now. We like the 1.7+ but we are afraid to make a mission with our expensive camera. But at the same time we don't want to go to 1.6.5 because there are a lot of mission related fixes in 1.7.3 we need,

Here is how the mixer looks like. We use only the main bus.

VTOL quad X mixer for PX4FMU

=============================

R: 4x 10000 10000 10000 0

Elevon mixers

Three scalers total (output, roll, pitch).

On the assumption that the two elevon servos are physically reversed, the pitch input is inverted between the two servos.

M: 2 O: 10000 10000 0 -10000 10000 S: 1 0 7500 7500 0 -10000 10000 S: 1 1 6500 6500 0 -10000 10000

M: 2 O: 10000 10000 0 -10000 10000 S: 1 0 7500 7500 0 -10000 10000 S: 1 1 -6500 -6500 0 -10000 10000

Throttle mixer

M: 1 O: 10000 10000 0 -10000 10000 S: 1 3 0 20000 -10000 -10000 10000

Z:

Nothing unusual. A little more weight to the roll in the mixer.

[JohnSnowball]

Hello, I checked your log and found that, the logger stopped (maybe the whole PX4) in air, just as mine. The firmware I'm using is derived from master in July 2017.

[tubeme]

@JohnSnowball Thanks man. I think just the logging stopped because we were able to switch back to MC just before we hit the ground...

[RomanBapst]

@tubeme I know this is abit late but do you happen to have the telemetry log of this flight?

[tubeme]

@RomanBapst Hmmm It will be really hard to find out this one. I will try.

From the link I gave

LOG: https://logs.px4.io/plot_app?log=89bd8fa6-6506-4e9a-87d5-81ee2cc9243a

I was able to download ULG file...

Other than that I will try to find it...

[RomanBapst]

@tubeme Hi Vasil, do you know if the autopilot lost power during impact?

[tubeme]

@RomanBapst On impact the Batteries just flew away. It is seen in the video.

But up until the crash the vtol had power, the pilot switched the transition switch but too late. Quad-copter was working before the impact.

[LorenzMeier]

Closing: https://github.com/PX4/Firmware/issues/9271#issuecomment-392344702