transmitter enters in emergency mode while in flight

[scott-adamson1975]

on my x12s running opentx 2.2.3 (eu/lua/no heli) the transmitter has entered in emergency mode twice while in flight.

on the first occasion i was at low altitude performing a roll and the tx entered in emergency mode, no time for emergency mode to boot fully before plane met floor so a totalled plane was the result

the second occasion i had sufficient altitude, flew out 700 meters fpv and on the return noticed that i was dropping a wing with an unresponsive model, initially thought that i had lost signal, but it recovered and noticed that rssi on vtx telemetry had frozen, quickly landed and saw the tx had entered in emergency mode while in flight.

have not had emergency mode while not running the scripts

have left transmitter running for 3 hours on ground connected to plane in armed state and have not been able to reproduce, may be loading and playing of multiple sound files at the same time as on both occasions i was at some form of limit (in the roll i at extremely low altitude arsing around and my battery was reaching low status, the other i was on the return leg of a 700 meter journey in a radio noisey environment)

will give it another try this weekend (with height of course), are there any logs that i can turn on to try and give you some information regarding this?

To reproduce

fly on an x12s with the script enabled

Expected behavior

not enter emergency mode?

Screenshots

https://photos.app.goo.gl/aHHxKpR6mtbHrNHM6

[teckel12]

@scott-adamson1975 I assume you're running Lua Telemetry as a widget? It could be some issue with OpenTX as well. There's been some history of this error:

https://github.com/opentx/opentx/issues/4589

This particular issue was a memory leak. You may want to review the used RAM to see if it's related. In other words, there could be a memory leak in OpenTX which is causing emergency mode. Not to say there isn't a problem with Lua Telemetry. But, I've flown quite a bit with my Horus X10S without any issues like this.

I'll try a flight with a bunch of mode changes to trigger a bunch of sound file playbacks. You may also want to be sure you're using a fast enough SD card. Using a Class 10 card should insure that it's not a SD card speed issue.

[scott-adamson1975]

yup, using as a full screen widget

i will buy a fresh new sd card, the one i was using has been "used and abused" in a raspberry pi before

i think i recall that on both occasions when this has happened I had switched between models without powercycling

[teckel12]

@scott-adamson1975 When you switch between models Lua Telemetry is restarted. But, that actually could be what's causing the memory leak (if that's what's happening).

The error disabled would be more on the lines of a bug with Lua Telemetry.

The error CPU Limit would also be the fault of Lua Telemetry, but not so much a bug but that it's using too many CPU cycles so OpenTX kills it so you can still fly.

I believe EMERGENCY MODE is when OpenTX crashes. A lua script shouldn't be able to cause this, as there's protections against lua using all resources or crashing the entire controller.

You may want to create an issue in OpenTX about this. As a heads up, some of the OpenTX developers are not so keen on lua scripts and especially lua scripts that do many things (like Lua Telemetry). They may be quick to blame the script, just so you know. It may be a better tactic to not disclose you're using Lua Telemetry at first (as I honestly don't think a lua script should be able to cause this). Then disclose it only if they ask for more details. But who knows, there are some very friendly to lua developers with OpenTX as well, so maybe you'll get lucky.

Keep in mind that I'm not saying that it can't be Lua Telemetry causing this or that for sure it's OpenTX. Just that it seems very possible as OpenTX shouldn't crash due to a lua script as they've kept lua scripts fairly contained and controlled.

[teckel12]

@scott-adamson1975 You probably want to try the latest development build. I made some performance changes that could avoid the problem you were having (unless it is an OpenTX memory issue).

[fpoullet]

Hello, i have horus 12s OT2.2.3 EU and same problem, sd x10 ok, lua full screen widget, no another widget active. Can i have a link for the last development build ? Regards

[janmj-skd]

I would like to chime in here too. I have a Horus X12S OpenTX 2.2.3, and also experienced emergency mode twice. Second time it even crashed the whole radio, had to reboot to regain link to receiver. with resulting chrash and dameged plane, very annoying!

A nice widget, but i will certainly not risk my planes and potentialy the safety of myself and others for debugging. I am however more than willing to help debug on the ground. Problem so far is that i can't reproduce this unless the plane is airborne. Have anyone successfully reproduced this in a controled environment?

I see the issue was closed 3 days ago, can anyone confirm that the issue was properly identified and fixed?

Edit: Sorry i see it's still open, but there is a pull request that's supposed to fix the issue.

[teckel12]

@fpoullet There's a wiki on upgrading to the development build. It's super easy too:

https://github.com/iNavFlight/LuaTelemetry/wiki/Upgrade-to-Development-Build

@janmj-skd There's a possible fix in the development branch, see the above link for instructions on installing. I don't have the X12S controller and I've never triggered an emergency mode error on my X10S, so I can't confirm that the changes in the development branch resolve the problem.

There's safeguards in place with OpenTX to limit lua from crashing the entire controller. It's kinda strange that only the X12S is having this problem while it seems the X10S is working correctly. There very well could be an OpenTX bug on just the X12S. If the latest development branch doesn't resolve the problem, I would suggest opening an issue with OpenTX because I'm sure they don't want the transmitter crashing. If it's the fault of a lua script or not, OpenTX crashing is not desirable.

[teckel12]

Issue re-opened to see if problems on X12S are resolved with v1.6.1

[fpoullet]

Issue re-opened to see if problems on X12S are resolved with v1.6.1

Génial, nice i download and test tomorrow. Regards.

[janmj]

@teckel12 I will try the 1.61 on the ground to see if i can reproduce the emergency mode. Even if i couldn't reproduce the problem i might have a theory to what caused it. The voltage/fuel measurements was quite unreliable, an caused so many warnings i eventually had to turn off haptic/voice because it was too annoying :). Maybe a memoryleak in opentx was triggerd because of this? And since it was so excesive the LUA failsafe didn't kille the script in time, resulting in a total crash of the radio. Anyways, just a theory. I will test some more and report back if i can find some way to trigger this again.

Edit: Btw. I'm also janmj-skd, my work account. This is my private one :)

[teckel12]

@janmj What telemetry protocol are you flying (SBUS or CRSF)? Also, what do you mean by the voltage/fuel readings were unreliable? Lua Telemetry just gets these values from INAV.

Maybe you're not averaging the voltage sensor on your transmitter? I always do this and I would assume everyone would because the voltage will be all over the place otherwise.

Im going to guess that's its voltage and not fuel, as fuel only goes in one direction and only gives audible feedback every 10% (until it's below your low fuel setting).

If your getting a lot of voltage dips, you probably need to have averaging turned on for the telemetry sensor. Or, you can just turn off audible feedback for voltage (there's individual settings for different warning types, you don't need to turn everything off if just the voltage is the one giving the messages).

I only really use telemetry for 5-7" quads, so my smallest battery is around 1500 mAh. I always set all my voltage telemetry sensors to averaging. Even with all voice alerts on, I've never been peppered with audio alerts. I would suggest if you do, note which value is flashing or red and land. Then figure out what's wrong before flying again. It shouldn't be freaking out with messages, if it does something isn't right as that's not what my quads do with default settings.

I am, however, guessing. It would be nice to get all your radio files and Lua Telemetry cfg files so I could see your setup. Even better would be telemetry log files (which are super useful for SBUS because I can play back your entire flight).

[janmj]

@teckel12 I'm using sbus. I mean that values presented in the widget was wildly different to the actual telemetry values. I don't really use the fuel value, but wanted to test it out. But as i said earlier it was all over the place. I have never before had a need to average to get correct voltage readings. Btw, this is on a plane not a quad. So the voltage varies a lot less, but even so i have never felt the need to average before. Anyways, point being it seemed the widget wasn't really getting correct values and was swamped with error messages, i did eventually turn the warnings off. But by then i have already crashed and i'm not about to try this again in the air :). Unfortunately i have no blackbox logging for the plane that chrashed it have a matek 411 wing so no log at all. I will try to make one of my more "expendable" planes with a omnibus F4 pro on it airborne and provide you with logs and config files.

[teckel12]

@janmj There's no reason to test anything else other than the voltage. That's the core problem here and we shouldn't be looking at anything else. The voltage is just being read from the telemetry sensors. It makes no sense that the voltage would be all over the place.

Anyway, let's ONLY concentrate on the voltage.

When I asked for logs, I mean telemetry logs that your transmitter can save. You may have it turned off, but I would suggest turning this on as you can get a LOT of information from this. With a telemetry log and your transmitter settings, I can playback your entire flight and see exactly what your transmitter was showing. Without the telemetry logs and without your transmitter config files we can only guess.

If you select to open a new Lua Telemetry issue and select bug, it will give you instructions on how to get the information that I need to simulate your flight and diagnose the bug. I need this info as I have never experienced anything even close to what you're reporting. I would guess if tge voltage is all over the place, it would be on the bench as well?

As to guessing... What battery size we're you using? Like 4 or 6 cell or whatever.

[slash24nl]

I have the same issue with my X12S, as well as a friend of mine. I read that you need logging and settings of the transmitter. Can you let us know how to gather the required info, as we are very happy to deliver this info if it can solve the issue! We really like your LUA but it's a little tricky to use with this issue still happening on the X12S.

[teckel12]

@slash24nl This appears to only be happening only on X12S transmitters and not the X10S, which is odd as I would think the OpenTX firmware would be mostly the same.

When opening a bug report issue, I provide instructions for dumping your transmitter settings, see the following:

https://github.com/iNavFlight/LuaTelemetry/issues/new?template=Bug_report.md

The telemetry logs are just part of OpenTX, nothing special to Lua Telemetry. Just turn on logging for all of your telemetry sensors and set a function to start logging when your arm switch is activated. This is something that I would figure everyone would do all the time, as keeping log files for any type of problem is crucial. But, maybe it's not as standard a practice as I would guess.

[teckel12]

From reports on this issue, it seems possible that the sound file playback on the X12S either has a bug or is at least different than the X10S and Taranis.

I'm in the process of creating a branch that hopefully will address this. It also won't cause any adverse effects for other controllers.

I'm still working on it, but when I feel it's ready for testing I'll post instructions here.

[teckel12]

@scott-adamson1975 Check out the latest development branch. I made a change that should prevent a volume message overflow. While this should never happen, it seems it can if there's multiple sensors with the same name (OpenTX randomly picks one in this case).

[teckel12]

@slash24nl Instructions for upgrading to the development branch:

https://github.com/iNavFlight/LuaTelemetry/wiki/Upgrade-to-Development-Build

[PeterBeretta]

Whilst flying a $2000 F5J model today, my X12s went into emergency mode and my plane crashed into the ground from 150m. I had an R9M module with R9MM rx and a 2.4GHz G-RX8 receiver. I had recently updated the firmware on the R9M and R9MM to the latest and same version. I had the power setting on the R9M module to "Auto <=1W". Just before the radio went into "Emergency Mode" I got a telemetry message saying "Sensor Lost", then I heard buzzing on my radio as the R9M module ramped up the power, then it went into emergency mode. My TX is running OTX 2.2.3. Any ideas about what may have caused this? I noted on a thread on Github that there is a known memory leak problem with OTX 2.2.3 and LUA scripts with sensors on the X12s which can cause the radio to go into emergency mode. Apparently "emergency mode" happens when OTX crashes.

[teckel12]

@PeterBeretta That's a terrible story. The Emergency Mode is an OpenTX crash and does seem to happen from a memory leak or overflow of some kind.

Were you running INAV Lua Telemetry at the time? I don't think it's related directly to Lua Telemetry, as others have experienced Emergency Mode without Lua Telemetry running. If you were running Lua Telemetry, what version? An older version (in rare cases where the voltage was all over the place) could cause a bunch of audio low battery alerts (like hundreds). While this shouldn't be a problem, if there's a bug in OpenTX which causes an overflow or fragmented memory because of this I could see this happening.

OpenTX sets lots of limits on lua scripts. If the script uses too many CPU cycles, too much memory, etc it disables the script (which is the 'disabled' message). A script should never be able to crash OpenTX.

Emergency Mode is more of an OpenTX bug as far as I can tell. If you're using a GPS and have failsafe setup and tested, you should be able to even turn off your transmitter and your model will return to you. I always test this by doing exactly that in a controlled situation. Position hold about 10 feet above soft ground and turn off the transmitter (or unplug the transmitter module). This is part of my testing before I fly a new model.

[PeterBeretta]

Hi,

I was not running INAV as this is a fixed wing glider and was thermal soaring directly above myself, so the plane was well within range. The 2.4GHz rx that I use has a variometer built in (G-RX8) and I just have a rule under special functions to play the altitude value.

The model configuration I use is one used by hundreds of pilots and was created my Mike Shellim http://rc-soar.com/opentx/setups/f3j/index.htm

This is a fairly complex model configuration but I have never had issues with it previously and have used it for years.

[teckel12]

@PeterBeretta At least it seems this Emergency Mode issue on the X12S is not caused specifically by INAV Lua Telemetry. I know this doesn't help your situation, but does mine.

As it appears this issue is only being reported on the X12S, it could also be a hardware issue as OpenTX between the X12S and X10S is so close, and I assume the X10S is more popular so it should have more reports of problems.

In any case, this is a good warning for those using the X12S to insure failsafe before flying. In your situation, I assume failsafe should cause a slow downward spiral, giving you enough time to restart the controller or at least a "gentile" crash.

One thing you mentioned which other's have as well is that Emergency Mode seems to happen when an audio alert is playing. It may be wise to disable all wav file playback on the X12S till OpenTX sorts this out (unless of course it's hardware). If I had an X12S, I'd probably go this route.

[PeterBeretta]

According to the telemetry logs, the RSSI was around 90, RX voltage 4.9v, TX voltage 10.5v, so was definitely not a battery or signal strength issue.

[PeterBeretta]

Thanks Tim, will do so