cyoung / stratux

Aviation weather and traffic receiver based on RTL-SDR.
BSD 3-Clause "New" or "Revised" License
1.07k stars 365 forks source link

Sighting: Stratux drop out #631

Closed jzeevi closed 7 years ago

jzeevi commented 7 years ago
  1. Stratux version:1.3b3

  2. Stratux config:

    SDR

    • [ ] single
    • [ X] dual

    GPS

    • [X ] yes
    • [ ] no type: GPYes

    AHRS

    • [ X] yes
    • [ ] no

    power source: Airplane (USB Port)

    usb cable: um, yes?

  3. EFB app and version: (e.g., WingX Pro7 8.6.2) FF, beta

    EFB platform: (e.g., iOS 9.2) 10.3.x (latest, whatever version that is)

    EFB hardware: (e.g., iPad Mini 2) iPad Mini 2

  4. Description of your issue: A sighting: while on a set of long flights (2-3 hours each) over this last weekend, I saw FF showing ADS-B towers (i.e. connected to Stratux), but NO ADS-B data. Speed went to zero, position went to zero, the FF airplane went to a blue pulsing dot, "remaining" data was missing in the navlog. In one instance, the data came back by itself within a minute or so. In another case, I rebooted the Stratux (power cycle) after 2-3 minutes. When it re-occured, I simply powered off the stratux and utilized the iPad GPS for FF data (at this point, I was 2.5 hours out from my destination. In total over three days, I flew 18+ hours). Earlier, I had checked Stratux temperature and seen that it was around 51 degrees C. Note: in my plane (Aircoupe) the GPS has sky view through the canopy.

If possible, enable "Replay Logs", reproduce the problem, and provide a copy of the logs in http://192.168.10.1/logs/stratux/ and http://192.168.10.1/logs/stratux.log.

Ergonomicmike commented 7 years ago

This just happened to me today, although mine might not be a representative sample. (I've sh'd up to an Eric build, and then back down to a Chris build. ) Also, my Stratux has been on for about 4 hours in the house (AC set for 83) and is quite warm in a Chris box, with no fan and copper foil shield around to make it a little oven. Am also getting "stuttering" of the web UI, as if packets aren't making it thru or are being delayed or sent real slow. https://vid.me/bGy2d

daveholland86 commented 7 years ago

I also had mine drop out completely, no traffic no GPS but the box was still running and wifi on v1.3b2 c6e96b5. I was flying my checkride so I didnt trouble shoot it but it really threw me off my game (Instrument check), still passed though. I have since updated to the new b3 but have not flown since.

jpuetthoff commented 7 years ago

daveholland86, you got to use stratux during the instrument checkride? Ipad?

daveholland86 commented 7 years ago

Yes I was using Foreflight with electronic terminal procedures.

jpuetthoff commented 7 years ago

Any limitation what you cannot use the ipad/ foreflight for? Doing my IFR training, and i like my ipad. My panel mount certified gps is not a moving map.

daveholland86 commented 7 years ago

No. Its only to be used to for situational awareness, so the gps approaches will be loaded in the the panel mount gps and nav from the VOR dial. You can verify your position on the ipad but its not official. I used it for flying my enroute, and approach and departure plates and keeping track where I was on the approach. Its good to see the aircraft on the leg segment and get the stepdown altitudes.

cyoung commented 7 years ago

I've confirmed this issue last weekend. On a 1.25hr flight, ForeFlight showed "ADS-B Disconnected" 3-4 times. Uptime on Stratux (gdl90_daemon) showed 55min+ at the end and no crashes/errors. It happened on my iPad Mini 2, but my passenger had ForeFlight on his iPad Mini 4 and experienced zero disconnects.

Tagging this as a ForeFlight issue. It popped up in ForeFlight 9.2.

cyoung commented 7 years ago

@daveholland86 - which iPad were you using?

pmaag commented 7 years ago

@cyoung , I wonder if this is hardware/iPad performance related. I also have a Mini 2 and see the same behavior. Do you have Hide Distant Traffic enabled? I am wondering if we hit a certain volume of ADS-B messages that just overwhelms the Mini 2.

My next flight I am going to try and hide Distant Traffic to see if I see the same disconnect messages. Theory is that hiding distant traffic might free up some resources on the iPad.

cyoung commented 7 years ago

Do not have "Hide Distant Traffic" enabled.

Thanks for your report, so far all the confirmed disconnects are on Mini 2's.

jpuetthoff commented 7 years ago
    I hide distant traffic on my Ipad mini, and still had disconnects, HOWEVER, on recent flights in light traffic environments haven't had a single disconnect. 

    Get Outlook for iOS

On Tue, Aug 22, 2017 at 11:38 AM -0400, "pmaag" notifications@github.com wrote:

@cyoung , I wonder if this is hardware/iPad performance related. I also have a Mini 2 and see the same behavior. Do you have Hide Distant Traffic enabled? I am wondering if we hit a certain volume of ADS-B messages that just overwhelms the Mini 2.

My next flight I am going to try and hide Distant Traffic to see if I see the same disconnect messages. Theory is that hiding distant traffic might free up some resources on the iPad.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

pmaag commented 7 years ago

I am flying in a plane tomorrow that has a Stratus 2s. If the Mini 2 performance theory is accurate, I should see these disconnect messages tomorrow during that flight. Obviously the Stratus could be doing some device-side throttling, but will be an interesting test.

I'll report back after the flight tomorrow.

cyoung commented 7 years ago

@jpuetthoff - thanks for the report. Which Mini do you have? On the back of the iPad it should say "Model A____".

jpuetthoff commented 7 years ago
    I will check as soon as I can. Thought it was a mini 2. Will verify. Case fits a mini 1 and 2. 

    Get Outlook for iOS

On Tue, Aug 22, 2017 at 11:44 AM -0400, "cyoung" notifications@github.com wrote:

@jpuetthoff - thanks for the report. Which Mini do you have? On the back of the iPad it should say "Model A____".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

pmaag commented 7 years ago

I am not sure this is a Foreflight issue. See the attached packet capture taken on the Stratux during an "ADS-B Disconnected" event. Here is my summary:

Foreflight reported the disconnect at 08-21-2017 22:01:14Z

Packet #10 in the capture is the last UDP packet sent from Stratux to the Foreflight client. Packet #13 is the Stratux attempting to ping the iPad. Never receives a reply. Packet #72 is the Stratux attempting to ping the iPad. Never receives a reply. Packet #91, the iPad sends a broadcast UDP packet with DST port of 50113, content: "i want to play - ffm - udp" Appears to be some sort of heartbeat. Packet #156 is the Stratux attempting to ping the iPad. Never receives a reply. Packet #219 is the Stratux attempting to ping the iPad. Never receives a reply. Packet #239 another iPad heartbeat (same as packet #91) to broadcast, so we know the iPad is on the network. Packet #291 another Stratux ping to the iPad, no response received. Packet #355 another Stratux ping to the iPad, no response received. Packet #375 another iPad heartbeat to broadcast, iPad definitely on the network. Packet #426 Stratux ICMP to iPad, this ICMP is replied to by the iPad. Packet #435 Stratux starts sending UDP once again to the iPad.

So from the looks of it, the iPad will stop replying to ICMP messages during times of high CPU usage/performance issues? This causes the Stratux "heartbeat" to the iPad to drop, which stops the flow of UDP. Once the iPad starts replying to ICMP once again, things begin to flow correctly.

Suggestion: Use ARP instead of ICMP to test device reachability. Usually ICMP is the first protocol to be dropped once CPU usage gets high. ARP is normally not deprioritized and can simply be checked by issuing an ARP request and looking for the entry within the Stratux's ARP table.

Thoughts?

ads-b_disconnected_filtered.zip

daveholland86 commented 7 years ago

Sorry, I just got notification today. I have a Mini 2 64gb.

pmaag commented 7 years ago

@cyoung - I have been working on the code for this fix and have something working. Looking for some direction on how you want to proceed, as there are some trade-offs (at least for iOS clients):

Here is how my code currently is working (in the following order): 1) If client has issued an ICMP Unreachable to UDP 4000 within the past 5 seconds, immediately report as sleeping.

else

2) If client has sent a FF heartbeat broadcast in the past 10 seconds, report as awake.

else

3) If client has not responded to ICMP request and has no MAC address in ARP table, report as sleeping.

else

4) If client has responded to ICMP request within the past 30 seconds, and has a MAC address in ARP table, report as awake.

2 & #4 are essentially the "second chance" for the older hardware that might be resource constrained at the ICMP layer but still active within the EFB.

The downside to #4 is that when a client is physically put to sleep (press of the power button), iOS has strange/inconsistent behavior. In my testing the iPad sometimes responds to ICMP (very intermittent and high latency), but always responds to ARP. This results in the client flapping between awake and sleep, since #4 marks the client as awake but then the client immediately issues ICMP unreachables when the stratux starts up the GDL90 stream (FF is not running). The flap is just a second, so the client will stay in sleep for 4 out of 5 seconds, but the flapping does occur.

Part of me thinks we should just have #2 and remove the check in #4, however other EFBs might not send a FF heartbeat (they likely don't), so we could see issues with those clients disconnecting when/if they start dropping ICMP due to hardware performance.

Any suggestions are appreciated. I am taking my code flying tonight and tomorrow and I definitely think it will fix the "ADS-B Disconnected" issue, however I don't sleep my iPad during flights so I am not sure how the flapping would affect me.

pmaag commented 7 years ago

Just an FYI, flew for 4.4 the past two days with my updated code described above and received zero "ADS-B Disconnected" errors. Normally I would have seen at least 3-8 instances during that length of time.

rgutlon commented 7 years ago

Just want to add my name to the list. Flew this Saturday on (2) 2 hour legs and for the first time started seeing the "ADS-B Disconnected" message in FF. These were longer flights than usual. Distant traffic hidden. However, I did have the Radar (Composite) Layer active (which is usually off). UI behavior as others indicated. At the time running 1.4r1 (just upgraded to r2) on a Mini 2 (32GB).

AMNegron commented 7 years ago

Throwing my name in. 1.4r2 / iPad Mini / FPG. Flight 1.0 hour. Stratus shows running 1.5 Counted 8 disconnects in flight.

mrdoornbos commented 7 years ago

Still an issue yesterday. Since I was flying it was hard to troubleshoot the source of the problem.

cyoung commented 7 years ago

@mrdoornbos - which app? iPad Mini/Mini2?

JohnOCFII commented 7 years ago

I flew for about 90 minutes yesterday. I saw between 8 and 10 disconnects. iPad Pro 10.5 with iOS 11.0.2. I didn't see any disconnects for about 20 minutes into the flight, which made me wonder if it was heat related, even though I have the PWM controlled fan.

jpuetthoff commented 7 years ago
    I have asked myself the same thing if it is heat related on the iPad side. My ipad is in a case on a cradle.  

    Get Outlook for iOS

On Sat, Oct 7, 2017 at 10:01 AM -0400, "JohnOCFII" notifications@github.com wrote:

I flew for about 90 minutes yesterday. I saw between 8 and 10 disconnects. iPad Pro 10.5 with iOS 11.0.2. I didn't see any disconnects for about 20 minutes into the flight, which made me wonder if it was heat related, even though I have the PWM controlled fan.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rgutlon commented 7 years ago

Noticed this post on FF Facebook page under the topic "We're testing iOS 11.0.2 with ForeFlight" and it made me wonder if this actually a FF issue ... https://www.facebook.com/pg/foreflight/posts/

I've experienced connectivity issues between Stratus 2S and Foreflight. I'm running the latest iOS. 2 hour night flight ... iPad showed the wifi connection was still active and online. Foreflight's device page showed Stratus connected and sharing data but the map said zero towers at 3500ft. GPS accuracy 5m. (iPad) No traffic on the map. If I killed the wifi from the settings and turned it back on, everything came back alive: 3 towers, lots of air targets, 1m accuracy. Had to do this three times in mid air. This happened at least once with the previous version of iOS. Only three flights so far with stratus.

mrdoornbos commented 7 years ago

@cyoung iPad Mini 4, Foreflight 9.4.1, iOS 11.0.2, Stratux 1.4r2

I'll grab it out of the plane on my way home and look at the logs.

mrdoornbos commented 7 years ago

I'll try it with a Trial version of Garmin Pilot just for fun on the same iPad to see if I see anything different.

cyoung commented 7 years ago

Sounds like ForeFlight hasn't finished their compatibility testing with iOS 11.0.2 in the week that it has been out. My guess is that they're preparing a fix for these issues. We don't have any other users (FlyQ, WingX, etc) reporting this issue yet.

cyoung commented 7 years ago

This appears to have gotten quite a bit worse.

Reproduced it even with sleep mode completely disabled, so there's something more going on.

(change isSleeping() to return false always): https://github.com/cyoung/stratux/blob/master/main/network.go#L147-L162

@pmaag - have you found anything else? Seems like it's much easier to reproduce now with iOS 11 and latest FF (9.4.1).

Use this to help reproduce: ./gen_gdl90 -replay -uatlog test-data/cyoung-09062015-noproblem-stratux-uat.log -speed 10

Tag #660.

pmaag commented 7 years ago

I have been swamped recently, I never heard back from anyone on my proposed changes. From my testing it really did help the issue.

Can you run your tests on the PR code i submitted to validate?

On Thu, Oct 12, 2017 at 4:47 PM cyoung notifications@github.com wrote:

This appears to have gotten quite a bit worse.

Reproduced it even with sleep mode completely disabled, so there's something more going on.

(change isSleeping() to return false always): https://github.com/cyoung/stratux/blob/master/main/network.go#L147-L162

@pmaag https://github.com/pmaag - have you found anything else? Seems like it's much easier to reproduce now with iOS 11 and latest FF (9.4.1).

Use this to help reproduce: ./gen_gdl90 -replay -uatlog test-data/cyoung-09062015-noproblem-stratux-uat.log -speed 10

Tag #660 https://github.com/cyoung/stratux/pull/660.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/cyoung/stratux/issues/631#issuecomment-336311527, or mute the thread https://github.com/notifications/unsubscribe-auth/AB-EcKGbS_oAerXCJ5DJsRG-hwMaFQHiks5srqUEgaJpZM4OOTmo .

cyoung commented 7 years ago

It might have been coincidental because if the disconnect happens with no sleep mode, then changing the sleep mode criteria shouldn't help much.

cyoung commented 7 years ago

https://youtu.be/VkypNVu03l8

iPhone 6s, iOS 11.0.2, ForeFlight 9.4.1.

Stratux UAT log replay @ 10x.

Stratux "sleep mode" disabled. See /getClients at the end (no clients are in sleep mode). Unit is sending GDL90, Wi-Fi is working, but ForeFlight detects no devices.

pmaag commented 7 years ago

Very interesting. I am running Foreflight 9.4.1 on iOS 10.x on a Mini 2. I will try and get some time to run some tests as well.

On Oct 12, 2017 5:09 PM, "cyoung" notifications@github.com wrote:

https://youtu.be/VkypNVu03l8

iPhone 6s, iOS 11.0.2, ForeFlight 9.4.1.

Stratux UAT log replay @ 10x.

Stratux "sleep mode" disabled. See /getClients at the end (no clients are in sleep mode). Unit is sending GDL90, Wi-Fi is working, but ForeFlight detects no devices.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/cyoung/stratux/issues/631#issuecomment-336314800, or mute the thread https://github.com/notifications/unsubscribe-auth/AB-EcLk1yAxt_d3eo1xFxJe9vxztj8f_ks5srqo3gaJpZM4OOTmo .

jdaero commented 7 years ago

Foreflight 9.4.1 IOS 10.3.3.

I reset my iPad to see if it was a conflict, and it improved but still having issues. I re-downloaded the software for Stratux. To make it work, I have to close the FF, and re-open it, and it will find the ADS-B. It will only work for a few minutes before it gets disconnected.

dkloiber commented 7 years ago

Any solutions yet? I am using an iPad mini 4 with the newest iOS and Foreflight. I am flying every day approx 6h. I have disconnect about every minute. Makes no sense for me to use it anymore.

mikehaire commented 7 years ago

I am having the same issue. Disconnect ADS B notice every couple of minutes. I have to close FF app each time and relaunch to get it back momentarily before it disconnects again. It’s so bad now that it’s become a distraction in the cockpit. It’s worthless in its current state. Any fixes yet?

cyoung commented 7 years ago

This is a ForeFlight bug. I've been trying to figure out what triggers it, but it's not obvious.

Stratux is working great (as usual) in every other app.

Best bet right now is to send a note to team@foreflight.com and let them know that they broke it.

dkloiber commented 7 years ago

Yes, I tested it with FlyQ EFB and Wing X. No problems. Works great. It’s just damn foreflight.

cyoung commented 7 years ago

I've let them know since the beginning of the ticket but it has gotten worse with the recent releases.

Best bet is to send them a note so that they know that it's affecting more than just a few select users.

dkloiber commented 7 years ago

I just sent them an email and received already a reply.

Hi Dan,

William here - thanks for the email.

ForeFlight's stance on Stratux is neutral at the moment. If you want to use it, that’s great. You can view in-flight weather and traffic data from Stratux in ForeFlight Mobile, and we have no plans to block it from connecting to ForeFlight Mobile. That said we don’t have a formal agreement or partnership with the Stratux developers so you’ll need to look to the Stratux community for support resolving any Stratux-related issues.

The Stratux experimental AHRS data (https://github.com/cyoung/stratux/wiki/FAQ#is-ahrs-working-can-i-use-stratux-to-power-my-efbs-synthetic-vision) isn't currently displayed by ForeFlight Mobile and I do not have any timeline for if or when that might change.

If you haven't yet built a Stratux and are looking for a low-priced ADS-B receiver option, I encourage you to check out the Scout: it's a super-compact dual-band ADS-B receiver that runs all day on a single-cell battery and works great with ForeFlight Mobile, and only costs $199. Find out more at flywithscout.com.

Regards,

William Pilot Support Team team@foreflight.com

cyoung commented 7 years ago

Nice canned response support and upsell. I think they don't even read incoming support emails unless you're following up for the second or third time. At least that has been my experience.

rgutlon commented 7 years ago

Any idea why Steve @ Seattle Avionics indicates "Do not use version 1.4r2 as it’s experimental and does not work w/ FlyQ EFB. 1.4r3 has been released but we are still testing it" on their ADS-B / GPS Device Status page? https://seattleavionics.wordpress.com/2017/10/12/ads-b-gps-device-status/

@cyoung May be worth reaching out to Steve for clarification ... I hope the timing of when we all noticed these ADS-B disconnects in FF is not related to when Scout was released (despite the comment from Support in Dan's reply).

cyoung commented 7 years ago

That was confusion on release naming, Steve was referring to the dangerzone v1.4r2 build which sends two types of packets and was confusing FlyQ. All of the issues here concern the main branch.

crj2driver commented 7 years ago

Hello, I have experienced the same adsb disconnections. I use foreflight / stratux about 5hrs a day, when it disconnects I loose my GPS position and traffic but weather is still displayed. sometimes it comes back by itself, sometimes I need to restart FF. It is pretty bad and distracting. Sent an email to FF support.

zanecane53 commented 7 years ago

Here’s the reply email I received..

Hi Zane,

Steven here. Thanks for the email and for the thoughts. ForeFlight's stance on Stratux is neutral at the moment. If you want to use it, that’s great. You can view in-flight weather and traffic data from Stratux in ForeFlight Mobile, and we have no plans to block it from connecting to ForeFlight Mobile. That said, we do not have a formal agreement or partnership with the Stratux developers so you’ll need to look to the Stratux community for support resolving any Stratux-related issues.

The Stratux experimental AHRS data isn't currently displayed by ForeFlight Mobile and I do not have any timeline for if or when that might change.

As you have learned, the Scout is fully integrated with ForeFlight. And another option if you want the AHRS and other units is the Stratus 2S.

Let us know if there is anything else that we can help you with.

Blue Skies,

Steven Pilot Support Team team@foreflight.com

I’m sure if there was $$ involved things would start working very quickly.. I’m thinking about just changeing EFB’s. This basically just says buy one of our products if you want it to work.

cyoung commented 7 years ago

I've just reproduced the issue with the Scout.

I didn't catch the exact moment when it disconnected (just left the iPad on for 10-15 minutes and it happened). But in the screen capture below, you'll see how the iPad is connected to Scout and I need to swipe to close ForeFlight and re-open it for the ADS-B connection to reestablish.

https://youtu.be/Spaf2ed7nwI

iPad Mini 2, iOS 11.0.2, ForeFlight 9.4.1.

Scout Wifi version: 1.3.1. Scout Firmware: 2.3.6.

ijessen commented 7 years ago

+1 same issue. Everything was perfect before the update to FF 9.4.1. Stratux v1.4r1, iPad Mini 4 WiFi, iOS 10.3.3.

During a flight tonight FF was losing its connection every 10-15 seconds. Force close / restart and it came back with a connection, which would drop almost immediately. I've reported to ForeFlight support and linked to this thread. Expect a canned response in 3..2..1...

ijessen commented 7 years ago

Response from ForeFlight support:

The logging function & messages were developed for supported devices so the in the logs info re: Stratux may not be reliable, but we are going to take a good faith look at this to see what's going on.

zanecane53 commented 7 years ago

I received a follow up email yesterday..

Hi Zane,

Following-up with Steven's email, though Stratux is not one of our officially supported devices, on a "time available" basis we are looking into the reports of device disconnections from you and other pilots.

I don't have a timeline for when we'll have any updates, but it would be helpful if you could send us the following:

1) As best you can remember, when you started seeing the "Disconnected" messages.

2) Which version of iOS is on the device(s) where you've been seeing it. If you've seen it on multiple versions of iOS, please let us know that too.

Etc..

They asked if I could send another diagnostic log from Foreflight and Stratux so they could look more in depth into the situation. I happen to be flying this afternoon so I should have one to them by tonight.

Keep the emails rolling because I think we’re maybe getting somewhere.

MenFly2 commented 7 years ago

My 2 cents. From my point of view "ADS-B has disconnected" issue doesn't appear limited to iOS version or hardware type.

Our rig: Raspberry Pi 3 Model B GPS V-162 (G-Mouse RoHS IPX6) DC 3.3V-5V

  1. Stratux version:1.3b3

  2. Stratux config:
SDR

    • single
    • [ X] dual w/ Dan Murray High Gain Antennas
  3. GPS

    • [X ] yes
    • no
    • type: Vk-162
  4. AHRS

    • [ X] yes
    • no
  5. power source: Anker PowerDrive 5 usb cable: Anker

  6. EFB app and version: FF 9.4.1
 (16238) EFB platform: iOS 10.3.3 EFB hardware: iPad Mini 4 &
    EFB platform: iOS 11.02 EFB hardware: iPad Air &
    EFB platform: iOS 10.3.3 EFB hardware: iPhone 6

  7. Description of your issue: After four plus hours of flying over the past two days we had numerous (more than 20) dropouts. we tried rebooting everything and finally pulled out the Bad Elf GPS and enjoyed looking out the window. The dropouts happened individually on each device. The Stratux seemed to be working great. Lots of traffic, up to 8 towers seen and gps at 3m. Except when it dropped out. Also, on the iPad Air received another error: "Download Error, Unable to connect to the network while connected to ADS-B." This year we have flown the Stratux from the Maryland to Oshkosh and back, to New Maxico and up the East Coast with no hiccups until this recent trip. We are currently not happy with FF.