Position Hold and Navigation issue

[giacomo892]

Hi. I've already wrote about this issue in the forum and no, unluckly, I do not have a blackbox log but a video. I'm posting here just because a better analysis could maybe fix the issue and cause i've found a previous similar issue that is now declared as closed. (#431)

First of all. Setup.

1.6.1 NAZE. F450 QUADX. UBLOX M8N,HMC5883,BMP280,MPU6050.

And the conf diff:

feature GPS
feature TELEMETRY
feature PWM_OUTPUT_ENABLE
serial 0 17 57600 38400 57600 115200
serial 1 2 57600 38400 0 115200
aux 0 1 0 950 2050
aux 1 1 1 950 2050
aux 2 3 0 1600 2025
aux 3 3 1 1600 2025
aux 4 9 1 1600 2025
aux 5 8 3 1650 2050
aux 6 19 2 1850 2050
set acc_hardware = MPU6050
set acczero_x = 164
set acczero_y = 16
set acczero_z = -12
set accgain_x = 4071
set accgain_y = 4082
set accgain_z = 4012
set align_mag = CW270FLIP
set mag_hardware = HMC5883
set mag_declination = 230
set magzero_x = 55
set magzero_y = -133
set magzero_z = -33
set mag_hold_rate_limit = 45
set baro_hardware = BMP280
set rx_min_usec = 989
set rx_max_usec = 2100
set failsafe_procedure = RTH
set vbat_scale = 50
set nav_max_speed = 700
set nav_max_climb_rate = 400
set nav_manual_speed = 500
set nav_manual_climb_rate = 300
set nav_mc_hover_thr = 1575
profile 1

set mc_p_pitch = 79
set mc_d_pitch = 45
set mc_p_roll = 79
set mc_d_roll = 45
set mc_p_yaw = 80
set mc_i_yaw = 40
set max_angle_inclination_rll = 350
set max_angle_inclination_pit = 350
set rate_accel_limit_roll_pitch = 4000
set rate_accel_limit_yaw = 1800
set roll_rate = 35
set pitch_rate = 35

Basically I had some other times when the quad was in position hold and magically started to drift away (fast!) like @xdigyx reported in #431 "Today I had a situation when at pos hold mode my hexacopter rapidly started to fly away. "

So maybe the fact that this time happened during a mission (and at the final stage = position hold) urged me to post this issue since there maybe something left from that issue.

I've checked for I2C errors and GPS errors after landing and they were at zero, so I assume all the sensors were working.

More, I did another flight without rebooting the FC also in position hold, even fully yawing the multi to check for mag issues, and it held correctly without toilet bowling. (an issue I never had before to be clear)

Here is the video on what happened https://www.youtube.com/watch?v=_kv2N9qdhfc

I've annotated the video , please check it using desktop version of youtube.

[stronnag]

I'm glad you mentioned #431, as i's a canonical example of how to diagnose, fix and demonstrate as fixed such problems. In particular:

• Unusual occurrence on a machine known to fly iNav successfully over 9 months prior
• Logging of every instance
• Rigour and repeatable set of test cases

All the above contributing to a discussion that led to a collaborative and successful analysis as a direct result of rigorous log collection and repeatable test cases facilitating the elimination of false trails before we reached the correct outcome after, iirc, 23 days --- probably the longest continual bug hunt we've had.

431 was not just "declared closed" it was closed because evidence was presented to demonstrate that a software fix had corrected a bug.

Bottom line: We need evidence.

[Redshifft]

Exact same experience with an Afromini on older firmware 1.3 and ublox 7 - this was a test bed and was perfect for a few weeks when built. After a 4 week layup it started exhibiting this fault again no logs but I will say the movements are quite fast and can be controlled (fast as in quicker than nav speed)

I would often test machines by physically dragging out of position hold and the machine would fly back to position much slower than exhibited above so that would indicate to me something more that intermittent gps data, it is certainly not vibration induced for me anyway,

[giacomo892]

On the same exact day (20 minutes before) with another battery I had something like >10 minutes continous position hold with a nice breeze and the quad was solid at it's place. I haven't yet found an external condition that let the issue begin.

[digitalentity]

My best guess without logs is GPS hardware or coverage issue.

[digitalentity]

@Redshifft @giacomo892 please record logs of your issues - troubleshooting by video is next to impossible.

[Redshifft]

@digitalentity , @giacomo892 asked me to contribute to this issue, we all have the lack of ports issue for logging i'm assuming. If it manifests after a minute or 2 in PH it should not "move fast" unless it's glitching out of PH ? PH is well filtered and slow to do anything - you know the code, how long will PH wait without valid (say GPS) before it flys away at speed to some location ? I say it should never and just rely on inertial sensors ?

The test would be to let it go and see where it stops (if it stops) :) like I say anything more than small position changes need to be filtered in PH - the mystery is why this fault only started happening and never a hint of the problem before, and I test stuff like you would not believe ;)

@giacomo892 Did yours just start doing this having been ok previously

[giacomo892]

I always had this issue since I'm using INAV. (I started since 1.5.1). I also do a lot of tests. Even tested the GPS alone on the bench leaving it on for many hours and it seemed to work.

I tried to leave it alone during the issue and you can see result in the video. At first it seemed like a normal behaviour on WP overshot (I never seen an overshoot before to be clear) but then it started to go around the final WP point at speed. Even worse when I switched back to position hold, at that point it started to fly away at navigation speed.

@digitalentity I've ordered and SPI microsd slot 4 months ago... haven't yet received it... If I receive it I can solder a openlog blackbox.

It's strage issue because if you disable pos hold and you enable it again it works as far I can tell.

The very first time I had this issue on a very windy day and I tough my quad could hold the wind. It started to fly away on position hold. I panicked and disabled it and recovered manually the quad.

About the coverage: The very first happened in a place were I had "only" 12 satellites but now happened with 18-19 in view without any obstacle as you can see from the video. There aren't pratically any interferences in that place. No power wires, no base stations and so on.

It could be due GPS outputting bad positions but as I said there aren't logged GPS errors and position hold works if you immediately disable and enable it again.

I could try maybe a self compiled version with glitch detection enabled that increments GPS error count in case of a glitch so I can easily monitor it from ezgui without the needing of a blackbox.

But @digitalentity already told me that glitch detection isn't very tuned. If you can provide better glitch radius and acc parameters I can try it in order to check is the GPS is sending crazy positions.

[Feldsalat]

Similar problem with my quad. Displayed speed in OSD disappears for 6 or 7 video frames then a coordinate jump around 30m and speed 246km/h. Inav responds with very fast correction in poshold. Happend after 9 minutes flighttime with Inav 1.6.1 No log, because older F1 board :-( GPS: Beitian 880 FC: Flip32 without Mag I think it is perhaps a hardware problem but why this is not noticed by glitch detection?

[giacomo892]

Could maybe switching to INAV computed GPS speed (from lat and lon change) instead of using the GPS module one fix this issue?

[Redshifft]

@Feldsalat yes as your osd display a high speed departure from PH.

It would seem (GPS) fault that triggers the wild reaction - If the GPS only sent good solid data (which is unlikely) we possibly would not see this issue but there is more to it as fortunately my Micro iNAV quad (Afromini F1 ublox7) does not exhibit this problem, and this has had lots of Flying. The difference is it has "Stronnag special" fw (stripped out with Blheli passthrough and Frsky telemetry added) Also a better Power supply for the board with a nice capacitor across the regulator. I could try that fw on the test quad I suppose ;) I would in the meantime recommend people adding at least a 330uf cap on the boards 5v rail/earth to protect the thing from back emf from the motors/esc's

[giacomo892]

I've enabled glitch detection logic using built-in parameters by decomenting line 638 in navigation_pos_estimator.c

#if defined(NAV_GPS_GLITCH_DETECTION)
    isGPSValid = isGPSValid && !posEstimator.gps.glitchDetected;
#endif

and also adding gpsStats.errors++ around line 382. This should let us easily without blackbox if glitch detection logic fired up by simply checking GPS errors with phone/configurator after landing.

if (detectGPSGlitch(currentTimeUs)) {
                    posEstimator.gps.glitchRecovery = false;
                    posEstimator.gps.glitchDetected = true;

                    //add an error if glitch is detected
                    gpsStats.errors++;

                }

I've build the firmware using gcc-arm-none-eabi-6_2-2016q4. I hope this is the correct compiler. I flashed it. (https://dl.dropboxusercontent.com/u/18187764/inav%20custom.jpg)

GPS works but I cannot fly until this weekend.

If someone want to try it I think it will be totally fine since those are the only modifications.

Download Link: https://dl.dropboxusercontent.com/u/18187764/inav_1.6.1_NAZE_GLITCH.hex

Edit:

I left the board near a windows for some hours and glitch detection counter has increased: (i'm using an ublox 7M now)

https://dl.dropboxusercontent.com/u/18187764/gps%20errors.png

[WaspFPV]

I had the same behaviour in the past and tracked it down to a malfunctioning compass. It did about the same as in the video, first passing waypoint, spiral around it, then flying away.

[giacomo892]

I'll check the compass from sensors tab when I land if the problem happens again.

[stronnag]

Probably not going to help if it's a transient issue caused by power related mag interference. Hence the frequent requests for a bb log.

[giacomo892]

Yes, you are right. But if the mag issue is transient it should not trash the whole PH. I'll report also with the new firmware that i've compiled.

[Redshifft]

Right, I have just put some fw on the problem test Quad that has been no problem on something else. I might be able to test tonight if it does not rain. @giacomo892 I still have to wonder why the Copter moves so fast when it's in PH and this issue manifests..it is just illogical to me !!

[giacomo892]

@Redshifft have you had time to fly my firmware?

That's the real question. Maybe it receives wrong coordinates or speed from the GPS module.

[Redshifft]

Ok, had about 6 flights 90% in PH and a couple of RTH, the wind and gusts were about the limit for iNAV and the 250 Quad, no problems encountered - Hardware was not touched in any shape or form from before and original configuration just copied across, down to you software boys to figure out now ;)

[giacomo892]

@Redshifft pardon, what software modification have you done ?

[Redshifft]

I said previously, my mate "software" compiled me a stripped out build.

[giacomo892]

So without any code modifications? Just stripped out things? Can you list them please.

[Redshifft]

Stronnag built it, I asked everything be pulled out and Frsky telemetry and Blheli passthrough added

fw attached rename it to rar ;)

[giacomo892]

I've crashed my F450. Luckily I was on grass and nothing damaged. It did a roll in the air and started falling.

[Redshifft]

@giacomo892 What does that mean ? what fw ?

[giacomo892]

I have no idea why it crashed. Some minutes before I did lot of yawing to check if mag was working and the position hold was working. Then suddenly it flipped and crashed. I was luckly at 10 meters. I was running my build 1.6.1 with glitch detection. I saw glitch detection working at least one time but the quad tried to escape like usual and at least two times but I stopped it forcing it in the opposite direction with the remote control

[Redshifft]

Flip = motor Put my fw on and test as yours in consistently showing this problem

[giacomo892]

I've tried to takeoff immediately after the crash and the copter stays in the air. I'll try yours soon.

[giacomo892]

@Redshifft I've flashed your firmware. If the weather allows i'll try to fly it today. Have you tried mine if you still have PH issues? @digitalentity Regarding my mid air flip and crash. Is there something related the mag that can screw up the IMU and crash in ANGLE mode? The motors seems to run fine.

Do you all have experience of esc+motor that fails randomly in air but works right after?

I've HobbyWing Xrotor 20A ESC. Should I put BLHELI on them?

[digitalentity]

@giacomo892 mag data can't screw roll/pitch data and cause a flip. This is most likely hardware issue caused by one of the following:

1. Compass chip messing up with I2C wiring and causing gyro hickup - which usually results in imminent crash.
2. Signal issues with one of the ESCs, usually caused by no signal ground from FC to ESC.

[Redshifft]

@giacomo892 The weather has turned to crap for a few days here :( More important you test my fw which has not displayed problems on 2 Quads. Do not change anything on your quad, sure it's unfortunate if you have other problems but we have to back-back the fw.

@digitalentity can you Just confirm in PH the max nav speed in all directions?

I have some ideas but await @giacomo892 results first.

[giacomo892]

@Redshifft I crashed again the quadcopter, dunno why it started to crash :) At least I was running your firmware and it did 4 minutes of position hold before crash. No issues reported but it is too early to say that all is working correctly. It was fighting wind, i did some yawing and all ok. Still unconclusive.

@digitalentity Thanks. So. After this crash the battery didn't pulled out (so i broke a landing gear, damn!) but I had access to the FC. I checked that there weren't any I2C errors and all the sensors still functional. So I'm really mad now. I flied like 5h with that quad and now all in sudden flyaways, crashes and so on.

At least I have the video of the crash. https://youtu.be/tekUD4AGcjE

Do you have any idea? Strange cause I re armed the quad right after the crash and all motor went online with all the power needed to do another takeoff :(

[Redshifft]

@giacomo892 I told you if it flips it's a motor ;) Now I'm obviously not able to tell you which part of the circuit from battery through to propeller but I will tell you it's the rear right motor, you will have to go through it carefully I'm afraid.

[giacomo892]

Sorry all for being sightly off-topic.

I've flashed BLHeli 14.9 to all my ESC's and WOW, the response of the engines is much much better. Loving damped light and i'll switch to ONESHOT125 soon 🥇

@Redshifft This is my quad running your FW (sorry for the light, it is 18.30) https://youtu.be/ICp3mm5iAt8 doing throttle punching for like 6-7 minutes like this and ESC just sightly warm and motor too.

All engines wire are cold and soldering point are cold and protected with hot glue. I've re-checked all ESC connections too. The next weekend, i'll try another position hold flight and see if I crash or fly away too :) Thanks for your suggestions.

[Redshifft]

Lol, you software boys :D I saw the motor stop, all you needed to do was turn the props over and move them around 1 position then test the thing on the ground - you don't fault find by changing other things !

[giacomo892]

@Redshifft I immediately tested all the motors right after the crash and it was working :)

I've another suspect. Till now I used to fly with temperatures from 0 to 10 degrees. But now outside is like 18-20 degrees during day so might also be a temperature issue. So I reversed all ESC and mounted them in a way that thermal dissipator is towards the sky and installed BLHeli to see if it self-protect less 💃

[Redshifft]

@giacomo892 It could be many things, I just think you could find the problem quickly and without risk running it with the motors and ESC's under load on the ground.

[giacomo892]

@Redshifft holding the quad with props on the ground and sending msp motor command to keep engines at full speed for minutes or without props? What trouble shoot do you suggest? Thanks thanks

[Redshifft]

Not full power obviously, but with the propellers moved around one position and fitted upside down it will just try to press itself into the ground. The object of the test is just to simulate load on the Motors and ESC's which would hopefully show up any intermittent problems. Obviously be careful and even tie it down to be extra safe. Just arm and control it with the RC as normal.

[giacomo892]

Ok! What do you mean when you say "with the propellers moved around one position"?

Yes, i'll tie to copter to ground so in cause of failure it does not capsize.

[Redshifft]

So the props are on the wrong motors they then push the Quad down instead of up and turning them upside down further reduces generated thrust.

[giacomo892]

@Redshifft Ah ok, perfect. Basically install CW on CCW and viceversa. Thanks. On the next weekend i'll do a 20 minute static burn test and if all work i'll run to the field to try PH. And ofc report back here.

Again sorry all for the OT but having a functional quad is the only way I have to test around PH bug :)

[Redshifft]

Lol, Yes swap CW and CCW also if you can also mount the propellers upside down they make less thrust but will still give everything a work out. Don't do long static tests without keeping an eye on temps as you are not going to have the same cooling air as normal because the props will blow up instead of down.

[giacomo892]

@Redshifft Ok. I'll monitor the temperatures and I hope to discharge a battery on the bench without issues :)

[digitalentity]

@giacomo892 I'm 99% certain that it's the motor/ESC/wiring. To reproduce the issue do as @Redshifft suggests - switch props and turn them upside down so the quad is pressed into the ground. This will put exactly the same stress on motor/battery etc but without acutally flying the machine.

@Redshifft if in PH machine exceeds the maximum speed - it's confused and moving in wrong direction.

I.e. GPS says "you should move North to correct", but compass has gone crazy and sais "North is at 180deg azimuth". Quad starts moving South but thinks that it's moving North. GPS reports increasing speed in South direction but as quad wants to go North it will continue to push harder in the "assumed north" direction believeing it's trying to fight the wind. The result - fly away in South direction at maximum tilt.

The problem here is that quad can fly in ANY directoion and GPS only reports direction of motion, not where the nose is pointed. Compass is required to figure out how the copters "front" is oriented relative to North and thus where to tilt the motors to move as required by GPS.

[Redshifft]

@giacomo892 you can also check mag swing when doing that ground test.

@digitalentity, Sure classic mag issues, and I assume the Mag data is not weighted in PH ? I have 2 reasons that make me feel it's not the mag.

1. in Mag tests - fly out 360's left and right etc then RTH it never displayed any glitches.
2. Fly off from PH seemed instantly pretty fast not a "building error" and it happened in still air with a virtually stationary Quad - so there was nothing to correct and certainly no rotation element ever observed and no consistent departure direction.

We all know above may not count for much ! but it's all we have atm ;)

[giacomo892]

@digitalentity I'll check also the soldering on the PDB (f450 bottom plate)

@Redshifft i'll check mag values via the sensors tab too if the change when giving throttle too :)

But for all the tests need to wait the weekend, and i'm struggling :(

[giacomo892]

Today I might have the chance of trying the motors. (Finally). Is there a way to run motors with the remote control without the pid loop running? (Avoiding the I term to grow and speed up only a some motors ) or is it better to use directly telemetry and MSP_MOTOR?

[Redshifft]

Try it first and if it's a problem use the motor test tab.

[giacomo892]

Hi!

Finally I've done the test. (Putting the I PID term to 0 :) :) :) so I could use the RC throttle)

During the first try the FC rebooted after two minutes and all motor stopped. I don't really know why it happened since it never happened before. (some INAV 1.3 hard bug?) .

Beside that, i've done many other minutes of test as you can see from the logs and none of the motor stopped.

ESC were pretty cold and motor sightly warm.

What I've noted is that gyro and acc are not noisy and mag readings don't change even at full throttle as again you can see from the logs.

To prevent reboots, if they are vaguely connected to lack of power i've pushed my BEC to output 6V instead of 5V.

So I think that I might try to go the field again and try the POS HOLD :)

Tethered MSP logs:

https://www.dropbox.com/sh/59zq5c37varyt15/AABi0fyIeO6IkCfobSNiV45xa?dl=0

[stronnag]

And it's in no way dodgy hardware?? Remind me, exactly which FC hardware is this?