Huawei: Large sporadic location deviations during GPX track recording

[scaidermern]

The following issue has been mentioned by three different users in the German OsmAnd #DE Telegram group, I didn't experience it myself so far.

Sometimes recorded GPS tracks contain huge location deviations. They appear sporadically. These users mention that it does not correlate with a bad GPS signal. However one user is pretty sure it happens at the same moment he switches the smartphone display on, i.e. waking the smartphone up.

Screenshots from user 1 (devices: Huawei Mate 20 lite and Honor):

Screenshot from user 2 (Moto G4 Plus, Android 7.0):

Screenshot from user 3 (Mi 8, Android 10, logging interval: 0 seconds, logging minimum displacement: off, logging minimum accuracy: 50 m):

• Does OsmAnd ever use the network location provider for track recording if the GPS location provider doesn't return a location?
• Is it possible to filter out these huge deviations? E.g. by comparing time & distance against the previous location or the previous and the next location.

[pebogufi]

I also had such jumps with a single wrong point. device samsung S7 android 8.

[sonora]

I remember such occurrences from the old days, but have not seen this in any hardware younger than 10 years. I would not think this comes from using the fused location provider, but maybe people would like to rule this out by temporarily switching their phone's Location setting to GPS only.

If anyone has access to such a GPX file, I would be interested to take a look at the data recorded to see what it speaks.

If those data points are accompanied by large hdop values, setting the minimum accuracy filter could be a workaround, although I do not normally advocate any filtering at the time of recording because the risk of losing valuable data.

[scaidermern]

If anyone has access to such a GPX file, I would be interested to take a look at the data recorded to see what it speaks.

This is the track from one of the users: 2020-06-01_10-27_Mon.gpx.txt

This is the data point with the large deviation, it has a pretty high hdop value:

      <trkpt lat="51.2815674" lon="7.3400054">
        <ele>0</ele>
        <time>2020-06-01T10:03:42Z</time>
        <hdop>1462</hdop>
      </trkpt>

The other hdop values range from 3 to 31 whereas the mean is around 8 and the median around 6.

The previous hdop values are 12.7 + 13.7 and the following ones are 4.9 + 16.6. This means it should be possible to filter out these large deviations by tracking the typical hdop value of the last few track points and ignoring location reports with a much higher hdop values. Or keeping them in memory and either using them if the large hdop value persists or dropping them if the next points are of significantly higher accuracy.

[sonora]

An hdop over 100 for GPS indicates something rather strange. Note that at the same time the reported ele dropped to zero, and no speed value is being reported. I wonder if this is a chipset artifact, or in fact a weakness caused by the fused location provider, so a test using 'GPS only' would be valuable.

In any case, it looks like on devices producing this our 'minimum accuracy' filter should be able to blank these out.

[sonora]

PS: Sudden changes in GPS hdop are not entirely unusual, e.g. when entering a canyon (screenshot 1?), a forest with wet leaves (screenshot 2?), or a street flanked by high rise buildings or similar features causing lots of reflection (the quarry in screenshot 4?).

[vshcherb]

@sonora probably we can filter out values with such high hdop by default

[sonora]

@vshcherb Probably. But I just see that screenshot 4 above was allegedly taken with the accuracy filter set to 50, and it seems it did not help in that case. I personally would like to see a little more data and research to understand what's going on...

Also: Setting an arbitrary default may cause issues with present or future chipsets and getAccuracy() implementation or behavior of reporting (or not reporting any) hdop.

Plus: this may not really be a mainstream issue, by my experience.

[paolobenve]

I'm experimenting this issue (I reported #9691), the wrong point is usually the same point

These are my trip recording settings

This bug is quite annoying, because it makes osmand produce track graphs which are not usable: the min/max values of altitude/speed are fake, the trip length is wrong, too.

Why not simply discard points which correspond to a speed greater than a fixed value, different according to the profile (walking, car, ecc.)?

A trip track uploaded to openstreetmap.org: it's very annoying, too

[paolobenve]

3394150.gpx.gz this is the zipped gpx track

hdop is 2299.999 in the wrong points

[sonora]

@paolobenve Well, the maximum speed recorded in your file is only 7.8km/h, quite within the bounds for walking, so limiting the upper speed value would likely not be a good filter for you here.

Inspecting your file, it in deed seems that where your phone produces the artifacts the hdop value is in the triple digit range, so a workaround for your would likely be to set OsmAnd's "Minimum accuracy" filter to some 2-digit value like maybe 20 and it should suppress recording the wrong points.

Still, if this is a phone under warranty, it would be worth investigating. I would suspect a phone hardware or software issue here worth researching. You may want to confirm if or if not this equivalent issue happens also when using another app to record a GPX track. Off hand I see no reason to suspect this is an OsmAnd related issue ...

[sonora]

PS: I just read in the other issue you opened that this seems to happen when turning off the display, so chances are it may be related to the power saving implemented in your device.

Depending on your version of Android and the phone brand, there are different ways to turn off power saving (or at least use the lowest possible grade), or at least exempt certain apps, so you can create an exception that OsmAnd should not be touched. Experimenting with this may also solve your issue.

I e.g. mostly test on Samsung devices, and there, under Android 10, these settings give me 100% reliability for good background trip recording:

• Device care / Battery:
  • Power mode = Optimized
  • Power mode / Adaptive power saving = OFF (leaving ON may periodically use Medium power saving which inhibits OsmAnd logging)
• Device care / Battery / App Power Management (under And9 called 'Settings'):
  • Adaptive battery = ON (candidate for 'OFF', but no problem detected so far)
  • Put unused apps to sleep = OFF (check list of sleeping apps)
  • Auto disable unused apps = OFF (seems not to exist anymore in And10)
  • Optimize settings = OFF (in And10 under Device care / Advanced)
• Device Care / Advanced:
  • Notifications = ON
  • Auto optimization = ON
  • Auto restart = OFF
  • Optimize Settings = OFF
• Apps / ... / Special access / Optimize battery usage / All = Leave all unchanged (looks like OsmAnd does not need to have Battery optimization disabled here)

[paolobenve]

@sonora, my device is a Honor9Lite (Android 9), I can't find all those optimization you are saying...

I'm going to set the minimum accuracy and verifying if anything change.

But I was wondering whether I myself set it to "not selected" or that is the default value.

In case it's the default value, I'd change this default value to the maximum value, 100m

Besides that, if it's really simple to exclude points with high hdop value, why cannot we implement it? anyway, a high hdop value means that the point is absolutely a fake one.

[sonora]

Hi Paolo, yes, please try if setting a filter solves this particular issue for you. We are careful pre-setting any filters by default, because different devices have different issues, pre-setting filters may cause unforeseen and hard-to-debug issues in other situations.

I had a closer look at the file you provided, and I have to say that I have never before seen such behavior:

Your device failed to properly record exactly 3 different points, given by these 3 places in the file:

      <trkpt lat="44.5631446" lon="9.2420117">
        <ele>0</ele>
        <time>2020-08-26T08:12:58Z</time>
        <hdop>2299.999</hdop>
      </trkpt>

      <trkpt lat="44.5631446" lon="9.2420117">
        <ele>0</ele>
        <time>2020-08-26T08:41:57Z</time>
        <hdop>2299.999</hdop>
      </trkpt>

      <trkpt lat="44.5631446" lon="9.2420117">
        <ele>0</ele>
        <time>2020-08-26T08:51:55Z</time>
        <hdop>2299.999</hdop>
      </trkpt>

• Only these 3 points in your recording seem to have an issue. (Meaning if you manually find these 15 lines in your file and remove them in an editor, your recording is perfectly fine.)
• Immediately before and after these points the recording is ok, and the reception quality seems very good, so I doubt this is an issue of bad GPS reception.
• In these 3 points, please note that there is no speed recorded, the elevation is reported to be 0, the accuracy is reported to be 2299.999, and most notably, the false position reported for all 3 points (although recorded in completely different locations at different times), is exactly the same!

Given that this appears only 3 times in singular places, and at irregular intervals, I actually doubt now that it is a power saving issue. I wonder if you remember doing anything specific at the 3 locations, respectively the 3 times specified (8:12, 8:41, and 8:51, all Greenwhich Mean Time, so plus 2 hours in Bella Italia. :) ).

I would not know what else to say other than that your device very occasionally seems to produce this artifact. In any case I am optimistic that setting an accuracy filter in OsmAnd could avoid these points being written to future recordings. (The second largest hdop in your recording was 33.866, so maybe 100 is a reasonable cut-off for your situation.)

And maybe you can make the manufacturer aware using the data I specified above, perhaps they are aware and/or can fix this in a future firmware update.

[paolobenve]

I wonder if you remember doing anything specific at the 3 locations, respectively the 3 times specified

the 3 issues happened when switching of the screen

[sonora]

@scaidermern Would you know if we have any indication from the other users reporting this that it may be related to manually turning off the screen?

I would still have no idea how this can be or how to fix it, but it would be an interesting finding.

@paolobenve Is it EVERY TIME you turn off the screen, or just some times? And: If OsmAnd is in the foreground when you turn off the screen, maybe try to tap the home button first to first put it in the background, does that make a difference? (Just as a side node: Looking at your recording, these are not "extra" points: They perfectly fit in the 3 seconds intervals you recorded with, so it is not that turning off the screen some how caused an extra bogus point being recorded outside the regular sampling interval.)

[paolobenve]

Is it EVERY TIME you turn off the screen, or just some times?

no, on that trip I switched off the screen some 20 times, and only 3 times it happened

If OsmAnd is in the foreground when you turn off the screen, maybe try to tap the home button first to first put it in the background, does that make a difference?

I'm going to check it next time

[scaidermern]

@scaidermern Would you know if we have any indication from the other users reporting this that it may be related to manually turning off the screen?

According to the user with the Huawei device yes, this happened when switching off the screen. Hower this user just told me that the problem has been resolved by activating GPS only in the location service settings.

[vshcherb]

Probably such 2299.999 should be filtered out or we could use default setting value for hdop different than ignore but it is unclear how will it work on different devices then

[paolobenve]

@vshcherb I think that such a value can be discarded without any fear... what positive meaning could it have?

[sonora]

It could e.g. be Huawei's way to indicate a cell tower was used for a reading and not GPS. In any case it seems like a singular and not understood issue on a not very common chipset, and you can filter it if you want.

While admittedly it seems not very probable in this case, hard-coded fixes for not well understood issues can still be problematic in other scenarios. Imagine there is e.g. another chipset giving good readings but producing bogus hdops like this. Or a future use case where you would do freight tracking just based on cell towers?

[scaidermern]

I'm wondering why the title has been changed. Now it indicates that this is a problem specific to Huawei. However according to the comments this also occurs with Motorola, Xiaomi and Samsung devices.

[paolobenve]

It could e.g. be Huawei's way to indicate a cell tower was used for a reading and not GPS.

I liked the idea, but checking my fake points, they all converge to some particular point, but this points are not cell tower ones:

I think that the solution of letting the user filtering this points out according to a minimum accuracy value in the settings id not the best solution, for two reasons:

• like it happened to me, setting that value could mean to be aware of the nature of the problem, but we cannot suppose that the average user be such aware of what is happening;
• in the case that the user's path is near that singular point, are we sure that the point will be discarded with the minimum accuracy value?

So I'd propose to discard directly all the points with hdop > a given high value

[sonora]

Exactly true, looks like the only common theme in this issue may be turning the screen on/off (not even that seems certain), for several models and vendors.

And note that the screenshot from user 3 above was taken with the hdop filter set to 50 and still exhibits the issue. So if the filter works to fix the Huawei issue, it is a valid workaround exactly how I recommend to use it.

But as long as the issue is not understood and the workaround only fixes some cases, it is time to research, not code.

[bokke87]

Experiencing this issue quite often now, I am more than willing to supply data to try to solve this issue.

@scaidermern You are right, it definitely appears on devices other than Huawei as well. My device is a SM-G973F Galaxy S10, OsmAnd+ 3.8.4.

@sonora Same as @paolobenve I cannot reproduce the failure but I can say it has to do with switching the screen on and/or off.

I also experienced that twice within one track but the jumps don't converge in one point.

[sonora]

Interesting, the S10 is tested intensively, I have never seen the effect... Do you use the device settings I recommend in the documentation?

[bokke87]

• Device care / Battery:
  • Power mode = Optimized
  • Power mode / Adaptive power saving = OFF (leaving ON may periodically use Medium power saving which inhibits OsmAnd logging)
• Device care / Battery / App Power Management (under And9 called 'Settings'):
  • Adaptive battery = ON (candidate for 'OFF', but no problem detected so far)
  • Put unused apps to sleep = OFF (check list of sleeping apps)
  • Auto disable unused apps = OFF (seems not to exist anymore in And10)
  • Optimize settings = OFF (in And10 under Device care / Advanced)
• Device Care / Advanced:
  • Notifications = ON
  • Auto optimization = ON
  • Auto restart = OFF
  • Optimize Settings = OFF
• Apps / ... / Special access / Optimize battery usage / All = Leave all unchanged (looks like OsmAnd does not need to have Battery optimization disabled here)

I double checked all the settings just now and they are exactly the same what you recommended.

[sonora]

Ok, that helps, thanks!

• And you are confident the occurrences happened exactly when turning the screen on? (Or turning it off, or both?)
• And also, it is not well reproducible, but occurs only sometimes? Any correlation with anything which could help reproduce?

(I would like to go to some colleagues I know regularly record tracks on Samsung S10 and ask them to try to reproduce .... I myself heavily use Samsung XCover 4 and 4s (rather similar to the S10), and have certainly not seen this a single time, ever...). I am mostly using 15 sec and 30 sec recording intervals, with no recording filtering active. What are your parameters?

• Last but not least: When you look at the "off" point recorded, does it show any anomaly like unusually high hdop or speed value which could have been filtered at recording time, at least as a workaround?

[sonora]

PS: I have now verified with a colleague who has logged over 1500 km of hiking in the last 12 months on a Samsung Galaxg S10 at 1 sec intervals that he in fact has not seen a single instance of what is described here.

So, sadly, I have to say I am currently at loss where this particular issue originates from, and consequently if and what anything in OsmAnd could be done about it...

[bokke87]

I tried to reproduce with intent but didn't manage to reproduce this behaviour when walking outside for 2 hours yesterday.

When extracting the according data to my screenshots above I found huge hdop values with elev 0:

As a workaround filtering may fix this issue - but I'm actually not too much aware of how to set the values for a reasonable filtering.

• And also, it is not well reproducible, but occurs only sometimes? Any correlation with anything which could help reproduce?

I cannot tell whether it happens when turning the screen on or off. Sometimes I think it might have to do with using PeakFinder application in parallel but I cannot reproduce messing around with this app in parallel. Maybe someone who is experiencing this error is using PeakFinder as well?

Hope this helps.

[Zefling]

I have a similar issue with altitude with a Huawei P30. For some time now, the altitude displayed has been wrong, whereas a few months ago I did not have this kind of problem.

I bought a graft for the attitude of the ground, I thought he would use it if the data sent by the GPS is wrong, for example.

Normally for this course it is ~320 m and not 702 m

[scaidermern]

I have a similar issue with altitude with a Huawei P30. For some time now, the altitude displayed has been wrong, whereas a few months ago I did not have this kind of problem.

This sounds more like #10864.