altitude appears to be in Height Above Ellipsoid, rather than height above sea level as expected

[gdt]

I went for a walk and used the app in walking mode; the good news is that it worked well.

The altitude graph shows a lot of variation, but this is how GPS is, and I'm not complaining about that.

The app reports altitude and this should be in orthometric height, distance along the plumb line to sea level more or less, because this is the altitude that people use, and because the app's use of altitude for fitness tracking is all about gravity.

Looking at the altitudes, I see values that are very close to WGS84 Height Above Ellipsoid (HAE) for my area, which are 30m lower than WGS84 Altitude (Height above Geoid). While I realize this is only one data point, it is pretty convincing to me.

To reproduce, record a walk at a place where you know the Height above Ellipsoid and the WGS84 Altitude, someplace where those two are different enough.

I suspect the issue is the use of Android's location API which returns HAE when people would expect a height above sea level. OsmAnd deals with this by having a conversion grid. Outside of Android, essentially all GNSS receivers have this conversion built it and provide heights above sea level to users.

Test done on Pixel XL, LineageOS 15.1.

[dennisguse]

Yes, you are right. Altitude/elevation is provided from the GPS as WGS84 HAE: https://developer.android.com/reference/android/location/Location#getAltitude() OpenTracks takes value this directly taken from the GPS without any further modification.

So, yes that is definitively an important improvement. Before we introduce this correction, we need to figure out what we should export to KML/GPX.

For the implementation, we have the following options: a) change OpenTracks to store the corrected value in it's database b) just modify the user visible

[gdt]

Not sure why you think wrong elevation is an enhancement vs bug, but that doesn't matter too much. Yes, Android provides HAE, and IMHO this is an interface bug, because it means users need to convert if they want orthometric height, which is what everybody wants and expects. If you want to call this an enhancement, we should open a new bug that the the current values are mislabeled in the UI. But there is no way to label elevation as HAE in GPX or KML.

I have tried to find a precise statement about GPX. I think the lack of a clear statement is a clue that until Android introduced this strange notion of HAE, everybody thought it was obvious that elevations were in height above geoid. (== altitude) This is how NMEA reports, and then there is also a geoid height.

For KML, it's pretty clear that the sense of it is above sea level, at least when the extension to name it that way is used: https://developers.google.com/kml/documentation/kmlreference So gpx:altitude_mode should be set to relativeToSeaFloor. They also have a strange notion of "above the earth's surface" for the field without that, and I'd suggest never doing that.

I therefore think it's pretty clear that the right approach is to first find a magic library that converts lat/lon/hae to lat/lon/altitude, and then to use that as values are read from the Android GPS provider, store them already converted, and use those height above geoid values for display, GPX, KML. I know GPSTest does this, and OsmAnd does it too.

I have never met anyone who relates to HAE values at all, other than extreme surveying nerds who also know the geoid height in their region and can explain how their national vertical datums relate to WGS84 altitude.

[dennisguse]

Some more infos:

• Location.getAltitude() is always WGS84 ellipsoid: https://developer.android.com/reference/android/location/Location#getAltitude()
• GPSTest has a nice issue for this: https://github.com/barbeau/gpstest/issues/22 and the patch is here: https://github.com/barbeau/gpstest/commit/bc7d25976606b717f0aecc267152cae5fa053edc

We have the option to either parse the GPS returned data directly (GPSTest approach) or correct the value using a lookup table (FitoTrack approach). FitoTrack patch: https://codeberg.org/jannis/FitoTrack/commit/3a22c970189782f67f3e9c9158ccd6223b564227

[gdt]

If you are willing to require new enough android to get NMEA, or even to use NMEA when present, that sounds good. The lookup table approach is sensible too. Thanks for listening; there was just a long discussion on the tagging list for OSM about the nature of elevation and I really appreciate any reduction in confusion out there in Free Software geo land.

[dennisguse]

@gdt I slept over this one night and we really need to implement this. For now, I will add add a comment in the help section that the elevation is reported as WGS84 ellipsoid.

About the actual implementation: I honestly don't which approach is the better one to take. @gdt Do you have a suggestion?

[gdt]

I would also suggest changing the label from Altitude to HAE. That will confuse people, but IMHO that's better than having them make the wrong assumption. "Ellipsoid Height" is another good word if it fits. But if this can get fixed fast, that's not time well spent!

The gravity model approach is nice because it will work straightforwardly on all Android versions. However a full gravity model is not tiny in terms of bits. A non-full model will have errors.

The NMEA approach, reading geoid separation, is nice because it is easy. There actually is the gravity model inside the GPS chip, most likely, and this is just getting at it. The problem with reading NMEA is that I'm not entirely clear on which Android versions support it. I thought it was new, but it seems there is an ancient one, and a new one in API 24. Assuming you are targetting an API < 24, I wonder if you can use the old one and have the android compat libs work for newer versions. If that can be made to work relatively simply, it seems like a better approach. I have emailed with the GPSTest author and they know what they are doing -- so GPSTest having done this is a clue that it's a reasonable approach.

Using NMEA could also lead to recording DOP and putting that in GPX too, but that's beyond fitness tracking into pseudo-surveying.

There is a further nuance, which is that people can hook up an external GPS via bluetooth or similar, and use it instead of the internal, perhaps by mocking in developer options, or perhaps there is another way. I have seen situations where this causes the getAltitude call to return orthometric height instead, because of bugs in the way the external data is handled, which would result in the geoid height correction being applied twice. But this bug is really in external GPS interfacing, and thus beyond what I think you should worry about. I just wanted to mention it so you'd be on the lookout for it if you used an external GPS receiver.

[dennisguse]

NMEA is available as API level 5: https://developer.android.com/reference/android/location/GpsStatus.NmeaListener and was then renamed for API 25.

OpenTracks targets API 21.

@gdt If you have some more info let us know - but I cannot estimate the actual implementation effort needed at the moment. Accessing NMEA might be easy or not...

[gdt]

My impression is that if you target 21, you can use the function added in API 5, and there is some automatic compat wrapper on newer systems.

I would think using the approach from GPSTest should be fairly easy. Basically, set up NMEA listener, and look for the sentence with the geoid offset, and every time it arrives update a variable. Then use that variable to convert HAE into altitude. That code is is GPSTest, and can be copied if a compatible license or used for inspiration regardless.

I don't know what you mean "accessing NMEA might be easy or not". As I understand it, you make the listener call and then your function gets called with strings. If you ignore checksums, matching on GGA and pulling out the right field separated by commas does not seem super tricky. But I may be missing something.

That maybe leaves a window of the first report being before NMEA, so perhaps require 5 fixes with no valid geoid offset before HAE is just copied into altitude, and throw a popup to the user. Or perhaps make allowing ignoring geoid height configurable, and decline to record/report altitude without it. But that's probabliy an mprovement for later.

[dennisguse]

That is how GPSLogger is doing it: registering a LocationListener and an GpsStatus.NmeaListener while NMEA infos are added to each received location (I assume location frequency is higher than NMEA frequency): https://github.com/mendhak/gpslogger/blob/fa5b3d22d22dd5396f2a120aa8610c949b62d0e8/gpslogger/src/main/java/com/mendhak/gpslogger/GeneralLocationListener.java

Could we also get all information via NMEA? Only API issue: Android's NMEA API does not allow for setting the frequency while location API does. However, NMEA should contain all required information but we would need to parse it manually.

The German Wikipedia has some nice data samples: https://de.wikipedia.org/wiki/NMEA_0183#Einheiten_und_Datenformate I guess, we would need to go for: Global Positioning System Fix Data (GGA). A lot of more details: https://gpsd.gitlab.io/gpsd/NMEA.html#_gga_global_positioning_system_fix_data

[gdt]

Why do you want to get all information via NMEA? What's the gain?

A big issue is how often position is requested. The location API seems tuned for getting position occasionally and not running the GPS receiver at all times, which would result in excessive battery drain. Any logger app needs a strategy for adequate logging with adequately low battery use. Using only NMEA requires turning on GPS somehow, and this seems to require reinventing a lot. Simply adding a listener that gets GGA when it happens, if we really can do that, seems to address the geoid height issue without any other major changes.

[dennisguse]

Primarily, because Android's API is far from being optimal for this case. We have to listen two events and we don't know in which order these two APIs provide events:

• is it always the same?
• and if they happen always together?

Now my assumption is simply that the GPS unit is communicating with Android via NMEA already (why should they implement something custom for this communication; you can just use of the shelf hardware if the computation is done on the GNSS chipset).

If we react to both events (Location and NMEA), we have the issue that we extract lat/long/timestamp from Location while NMEA provides elevation above sea level. And if we don't know the order of these events OR if the frequency is different, it is totally annoying to implement on application level. We have three options for implementation in this case:

1. We have to wait for each coordinate until both events reached the app (assuming same frequency, different order)

2. On location event, we use the most recent NMEA elevation (basically the last one we got) and add/replace the WGS84 elevation of the location. This allows for different frequencies, but we might use the "wrong" elevation (the one from the previous coordinate rather the current one).

3. Or we integrate both events using their timestamp (again assuming the GNSS chipset is the source for events directly and they have the same timestamp).

Or we just use the NMEA data directly and extract all information from one event. From the application perspective it is simpler as we don't have to mess around with all these weird timing issues.

[gdt]

I did not mean to get "WGS84 orthometric height" (~height above sea level) from NMEA. I meant to get "geoid separation", which is the height of the geoid relative to the ellipsoid. This has two nice properties. One is that it is a gravity model output, not dependent on measurements. The other is that the nature of the model is that it changes very slowly. I just did a few test points (near Boston) and moving an entire degree of latitude or longitude changes the geoid height by about 60cm. Over 1km it is maybe a cm. So error in associating the geoid height report with a current location report will be entirely negligible; if it's within a few minutes even then the error from this will be imperceptible compared to GPS measurement error.

Thus, I really meant to say "start a listener, and every time you get a GGA, store the geoid height component in a variable. Whenever you get an android location report, take the ellipsoidal height that it gave you, and subtract the value in the geoid height variable to convert it to WGS84 orthometric height".

The only tricky part here is that until you have received a GGA value, the variable might be zero. But if you persist that variable, then this happens only the first time you turn on the GPS in the app. Or if you turn on the app after flying thousands of km. I would advise just not worrying about this, but it is also ok to store time of geoid height report and ignore location reports if there isn't a within-an-hour geoid height value.

My real concern in moving to NMEA is that Android location API seems to be the architecturally correct way to get location (even if it's defective about height), and there are issues about how often to turn on GPS etc. that are perhaps already handled. There are perhaps unknown troubles arising from a big switch.

[gdt]

If getting geoid height from NMEA seems too hard -- and it is starting to seem that way -- then perhaps we should reconsider and think about including a gravity model as a lookup table. Having done the experiment of moving 1 degree and seeing 60m, I think that having a table that is just 360 x 180 -- which seems quite feasible - is very likely entirely good enough, even with rounding before lookup instead of interpolation. That takes all the timing and API issues off the table.

The fitotrack approach looks entirely fine. Also this looks like a library that does exactly what you want (which is not shocking because ~every app that uses GPS on android should want this too!): https://github.com/matthiaszimmermann/EGM96

It seems that if you add that as a dependency and just call it, that might be entirely satisfactory.

[dennisguse]

I plan to use the EGM2008. NGA provides undulation data here: https://earth-info.nga.mil/wgs84/egm-download.php?file=EGM2008/2.5_minute_interpolation_grid/EGM2008_Interpolation_Grid.zip

We might need to strip down the resolution to 10' and also store decimeters (2 byte) instead IEEE754 (4 byte) due as we need to include the file into the APK. Also the conversion code provided by NGA is Fortran :sunglasses:

[gdt]

EGM2008 is exactly the standard approach, so that sounds good.

Not sure why you don't like fortran; it's possible to have efficient programs, ompared to needing 1 GB of RAM to run "hello world" in java!

[dennisguse]

@gdt I don't have any trouble with Fortran - I really like it, if something that is ages old, still works and is understandable. Re-implementing everything just because there is a new technology/framework/whatever doesn't make any sense (to me).

[dennisguse]

@gdt I created a draft for EGM2008 (5 minute resolution, adds about 20MB to the APK). If you have some time and knowledge at hand, I could use some help with the interpolation. A detailed description is available here: https://geographiclib.sourceforge.io/html/geoid.html (My C reading skills are not yet that great...)

See https://github.com/OpenTracksApp/OpenTracks/pull/672

[dennisguse]

Released in v3.18.0

@gdt Have fun :)

[dennisguse]

@gdt FYI when building the EGM2008 correction, I created a nasty bug. For negative longitudes, I fetched the wrong correction data. Technically, one could say that affected half the world....

Luckily I got approached by the community (incl. test data and a proposed fix). https://github.com/OpenTracksApp/OpenTracks/pull/1343