Unresponsive drone then chaos

[TheSoundMan000]

What happened:

Drone placed in open yard away from buildings etc as per standard operating procedure. Battery was connected (1) and the ESCs powered up but did not register a signal. Flight controller lights were on but not responding to any attempts to arm etc. GPS lights indicated a 3d lock after several seconds (2). This remained for 60 seconds or so without the FC booting up then the battery was removed and reconnected. These same symptoms repeated themselves but I gave up after 20 seconds and repeated. 3rd time over it armed instantly and performed as expected for a brief hover test (3).

I then moved to a screen to do some FPV flying (4), armed the craft and began take off. The craft failed to ascend and instead began a steady circle roughly around the far forward left arm. After a few seconds I realised that it was not responding to inputs and further yet was ignoring all disarm instructions etc (5). This extremely low altitude circle continued for a few seconds while I struggled to think of a solution then the craft went completely ballistic and smashed itself to pieces across the lawn (6).

Known prior issues

The arming issue as described has been showing up a bit lately. After a successful flight (of any duration, a few seconds or a complete battery) the FC will shut down upon disarm. By that I mean the following symptoms will be observed: Telemetry ceases, RC inputs are ignored and ESCs chirp indicating loss of signal. My diagnosis in this case has been a complete shutdown/hanging of the FC following a disarm condition in some instances. This has a roughly 50/50 strike rate and the only solution is a power cycle however no issues of any sort have shown up in flight.

A mild motor short was identified and rectified this morning. The craft has been operational in this state for many months unaffected and I got sick of the overheating arm this morning and fixed it. The motor continues to function after this incident completely fine and the issue appears completely unrelated.

Other important information

The drone had just completed one battery pack immediately prior to this flight. The disarm hang problem did occur twice but other than that the system was completely stable. RTH was consistently tested and performed without fault as did alt hold and pos hold. Telemetry link was solid and reliable and all indicators were positive in this regard. There were no crashes or jolts of any sort (This craft, until this incident was in mint condition despite many hours of operation).

1. Battery was tested immediately prior and after and both indicate a fully charged and functional battery. No issues here.

2. Telemetry indicated 20+ satellites on powerup, directions indicated and altitude were all as expected. The craft was correctly indicated on the map.

3. The craft took off, yawed, pitched and rolled correctly then landed.

4. The FPV system is a new addition. The operates completely independent of the main system (seperate battery) with the only link being via the signal controller to the FPV switcher consisting of GND and signal connections. FPV has been having a few issues but nothing related and seeing as its completely independant I can't see it causing a problem. It was certainly functional for a first person view of the crash if so desired.

5. As mentioned RTH was tested moments before, in fact the craft took off from the site of its last RTH. Failsafe has been well tested and is known to work reliably. The transmitter was fully charged and had made no indication of poor signal, it was functional immediately after the issue and still is. Failsafe procedure for the record is no signal output from the RX (SBUS).

6. The craft appeared to simply engaged full throttle on all 8 channels and hit the deck hard on its side completely shearing the propellor on the impacting arm. It then cartwheeled and smashed most remaining props (fortunately removing the battery in the process (without damage even, bonus!)) before settling right side up and rather sore.

System specifications

AnyFC F7 Custom mix for octo (verified over many hours of flight) Lora Telemetry (LTM) TGY-IA6B RX (SBUS) connected to a Jumper T12 TX flat octo in the standard layout (custom mix uses a logical numbering system, the only reason for that change) Very overpowered craft, 6kg at the time of test with roughly 15kg of thrust at 100% External BMP085 foam covered and sheltered from blasts, very stable and reliable alt hold UBLOX-M8N GPS puck with Magnetometer (El Cheapo but also well calibrated and thus far reliable save for some minor bugs initially with cable proximity) 3A bec on the FC, filtered and caps included to ensure stability. Separate bec for camera system. Blackbox has not worked (known bug that clearly isn't reliably fixed yet) due to the selected board type so no recording is available.

My thoughts

Possibly a long way from the truth but my thoughts as they stand:

The symptoms appear to echo that experienced with the shutdown on disarm problem. It appears as though the craft went into its own little world there for a moment then shutdown completely at which point the ESCs (or some atleast) briefly registered false readings due to the undefined shutdown behaviour at the pins and spooled accordingly (This has been observed in the past by simply powering down the FC whilst connected).

[TheSoundMan000]

Oh and also thanks for any assistance anyone can provide, I will do my best to conduct any experiments etc as needed in lieu of a blackbox, I am really hoping to get this fixed asap.

[rb1205]

My best guess would be an hardware issue, probably in your FC or maybe in the PSU/wiring. Try swapping the FC and check if the problem represents.

[teckel12]

@TheSoundMan000 Were you logging to SD card?

[TheSoundMan000]

@teckel12 No unfortunately though I certainly did try get it working prior. In my initial post I did mention the board and the non functioning SD card.

@rb1205 wiring hasn't been an issue over the prior 20 hours or so of operation and the loom is in good nick still. Of course it is a possibility but it's passed all tests thus far and for several reasons I suspect the issue lies elsewhere.

Hardware failure is a more likely option and it's definitely on the cards but I would like to investigate the possibility/probably of a software issue.

During the non responsive phase the drone was definitely "functional" in the sense that it was leveling and very consistently circling just simply non responsive and not behaving in accordance with any defined failsafe procedure. The chaos following I don't believe was a result of this status however, it appeared as if the FC had simply shut down or similar and the ESCs had received false messages.

I haven't been over the inav source code too thoroughly but the behaviour was as if the main software loop had stalled whilst maintaining the stabilisation side of things. A watchdog then kicking in for a hard reset could be responsible for the next stage? I don't know if this is possible in the inav implementation but something along those lines looks to me to be the cause of my troubles.

Perhaps a partial hardware failure could have contributed to this state but where would I be looking if that were the case? If you have suggestions for potential causes I'll have a sniff around and see if I can find anything anomalous.

[teckel12]

@TheSoundMan000 Sorry, your post was really long :wink:

Actually, non-functioning SD card may need more clarification. Did you have an SD card in, set to logging, but it didn't work?

[TheSoundMan000]

@teckel12 yeah it was a little hey 😂 wanted to get all the information down that I could while it was fresh in my mind.

The SD card I suspect is to do with the issues that come up a bit with the AnyFC F7 clone boards. I had to invert the one setting to get inav to recognise the SD card slot but nothing could get the SD card itself to register. There was no card in during that flight and the device was not logging. (Unfortunately 😐 I really wanted to be logging in case something like this happened but alas)

[teckel12]

@TheSoundMan000 I ask because sd logging has been known to cause FC crashing issues. Wondering if this is the source of your issue.

[TheSoundMan000]

Alrighty yeah well it would definitely make sense but unfortunately I don't think it's my problem. That being said I do have black box enabled and set to SD card mode, could a false reading of an inserted SD card (despite it not being there) have caused this issue?

[teckel12]

@TheSoundMan000 Possibly. The fact that you were having SD card reader issues and you were making changes to accommodate make it seem more likely even. Black box recording is known to cause total FC lock-ups. I would suggest turning BB off and have it controlled via a switch if you need to log something for testing. Also, use fast SD cards. Slow cards seen to have this problem more frequently.

[TheSoundMan000]

Thanks @teckel12 I'll definitely disable that then and see how we go.

it'll be a while before I fly it again so if anyone else has any suggestions that would be great is well, I would like to avoid a total wipeout again so anything that improves my chance's will be appreciated. I'll be a bit chicken next time I fly I think 😂

[TheSoundMan000]

Right guys, an update on this situation. I finally got the courage to fire up the beast this afternoon (Yippee :D) and results so far are positive but a bit of a back story first as to what I've done and what I think the issue was (pending a LOT(!!!) of testing).

A somewhat recent addition to the drone was a 433 1W LORA module for telemetry purposes. The addition was a fair few packs back and I had discounted it... until today when my fathers plane ran into a serious issue as a result of that exact module.

Upon connecting the signal connector to his module the servos would begin wiggling madly. The components were all on completely separate regs with proper grounding etc and plenty of filtering across the board... It wasn't enough to affect the FC or receiver but it was enough to wipe out the video. Despite being difficult to detect anything too concerning with a 'scope the problem was apparently noise of some form (Though I don't really know whats going on as we didn't bump the transmit down much and the issue is completely gone). Turning the power on the module down a click completely removed the issue and it got me thinking.

So... in addition to the disabling of blackbox and a complete strip down of all non-essential systems (to be gradually reintroduced at a later date stage by stage and with plenty of testing in between) the telemetry was removed and, when added, will be turned right down! I plan on putting a great many packs through with the drone tethered to a bundle of bricks to slow any unwanted departures until I'm satisfied the issue is resolved and will update on any issues or progress as I go.

[TheSoundMan000]

In other news the E32-TTL LORA modules perform incredibly! At 9k air data rate and 250mw of power with a lying down (incorrectly aligned) antenna they got past 5km with all but faultless signal.

I'm inclined to suggest quite confidently that (presuming a larger model with space for an independent power system on the radio module and appropriately aligned antenna) the 1W modules would quite solidly achieve link ranges out to the 15-20km range. I've personally tested the modules at 250mw and 19k air-rates out past 10km with a mountain in between (and at ground levels both ends) so perhaps even higher figures are possible.

Those things are just incredible!

[KnuckleUpFPV]

Ive seen this same scenario happen personally. My cpu load was maxing out. Was fine at times, then other times it would not arm. Then I got it to arm and take off. Once in the air it just spun in circles. No ability to disarm or take control in any way. Thankfully it wasnt to high and I stuck the broom handle up in the props. More so tossed the broom at it so it wouldn't come down on my head. Quad hit the ground and broke all the props. Lifted it from the battery and carried it inside. Motors still spinning. plugged the fc into the pc. And disconnected the battery. Wanted to keep the fc powered. Hit connect and it actually connected and opened the main screen. Then was completely unresponsive. Cpu load was reading 700ish percent. Don't remember the exact number. I was using a 433mhz openlrs module for control. Unplugged the gps, no change, unplugged the mag, no change. Unplugged the receiver and the cpu usage dropped down to 12%. Configurator unlocked and was able to navigate around. Fearing the fc had I problem I removed it.

[expipiplus1]

@teckel12 Is there an issue tracking this SD card lockup?

[teckel12]

@expipiplus1 There is an issue. Not sure if it's still open or the issue was resolved or not. I know it can happen if the card isn't fast enough. It's good to use a faster class 10 SanDisk 16gb card. Many are tempted to use an old slow 4gb card, but that's also the same people who have experienced lockups.

[expipiplus1]

Thanks, in that case might it be a good idea to include a warning when turning on logging. Or for inav to benchmark the card before arming.

On Sun, 10 Mar 2019, 01:26 Tim Eckel, notifications@github.com wrote:

@expipiplus1 https://github.com/expipiplus1 There is an issue. Not sure if it's still open or the issue was resolved or not. I know it can happen if thr card isn't fast enough. It's good to use a faster class 10 SanDisk 16gb card. Many are tempted to use an old slow 4gb card, but that's also the same people who have experienced lockups.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/iNavFlight/inav/issues/4136#issuecomment-471203192, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0U3H-RviNezwTdtyAc0ctzvNzakQVUks5vU-6ygaJpZM4ZlY_e .

[michaelvanderhoek]

@TheSoundMan000 , a bit offtopic, but im interested in the way you wired up your LoRa module. I'm using LoRa as well on my wing and hexacopter but somehow on the hexa i cant get it to work flawlessly.

You said it is set up as LTM telemetry? What settings do you use on what ports?

[stale[bot]]

This issue / pull request has been automatically marked as stale because it has not had any activity in 60 days. The resources of the INAV team are limited, and so we are asking for your help. This issue / pull request will be closed if no further activity occurs within two weeks.

[stale[bot]]

Automatically closing as inactive.