search

scubajorgen / TomTomWatch

Tool for exporting GPS files from TomTom Sports Watches
75 stars 16 forks source link

Ephemeris data format #33

Open remcokranenburg opened 8 months ago

remcokranenburg commented 8 months ago

Now that the service is being discontinued, I fear that my watch may become a brick (a small brick... a pebble?) when the GPS data is not updated anymore. Has anyone had any luck decyphering the format that the watch uses? The data is obviously public, so we should be able to feed the watch our own crafted files.

I contacted TomTom but they were not very forthcoming about the data format.

scubajorgen commented 8 months ago

Ephemeris

Interesting document explaining Ephemeris

TomTom ephemeris service

The TomTom link https://mysports.tomtom.com/service/config/config.json mentions the ephemeris services:

"service:ephemeris":"https://gpsquickfix.services.tomtom.com/fitness/sifgps.f2p{DAYS}enc.ee", "service:ephemeris_glonass":"https://gpsquickfix.services.tomtom.com/fitness/sifglo.f2p{DAYS}enc.ee" You need the first one, where {DAYS} is 3 or 7.

TomTom won't create the data themselves (maybe the repack it...) most probably and will obtain the ephemeris data from some provider. So we have

  1. to find the original or an alternative provider
  2. optional: to find the file format
  3. optional: repack to the proper format for the watch

file format

On the format:

Overview of various formats: Not much help

Where do the files come from?

Sirf application note: This is interesting: it mentions 'When in SGEE mode, SiRFInstantFixTM the EE server generates predicted ephemeris (SGEE) files providing aiding over 1, 3, 7 and 14 days that can be consumed by the SiRF StarIV GPS receiver.'. There we have our 3 and 7 days... It mentions file sizes: 33 and 77 kByte for 3 and 7 days. Corresponds with the TomTom files :-) It mentions 'Currently the SiRF server calculates EE data for a prediction interval of 14 days; this EE data is then used to generate all the SGEE files f2p3enc.ee and f2p7enc.ee'. Aha! corresponds with the TomTom files sifgps.f2p3enc.ee resp. sifgps.f2penc.ee! You need access to a Telit AGPS Server: '...open an FTP connection to Telit AGPS Server (please contact Telit Technical Support Center [1.3] for Telit AGPS Server Access Credentials)'

Remains to find such an AGPS server... They keep it pretty well hidden at Telit...

Hey, this is interesting! I find the files also on https://choaristudio.com/agps/. However, the files are not exactly equal to the TomTom files when downloading at the same moment... But the document above shows 'New SGEE files are published on the SiRF server at the start of each UTC day'. Maybe TomTom and this site are out of sync. This site downloads at 02:24 hours aparently...

scubajorgen commented 8 months ago

Unfortunatelly the https://choaristudio.com/ files won't work for the TomTom. I uploaded the 3 day file to one of my watches that had an empty battery for days. I recharged, placed the file and took it outside. After 20 seconds no fix yet... Probably the files are for Sirfstart IV and not for V or something... Or stale?

Other links, however requiring credentials...

http://sgee.samsung.csr.com/diff/packedDifference.f2p3enc.ee http://instantfix.csr.com/diff/packedDifference.f2p3enc.ee http://instantfix.csrlbs.com/sifglo.f2p3enc.ee

remcokranenburg commented 8 months ago

Wow you got very far in a short while! I just wrote a small script to download the four available files (GPS 3 & 7, Glonass 3 & 7) every 10 minutes, to see if I can discover some patterns. From your discoveries, I expect the files to only be updated once a day though.

(For completeness:

#!/bin/bash

while true
do
    ts="$(date -Iseconds)"
    mkdir "$ts"
    pushd "$ts"
    wget "https://gpsquickfix.services.tomtom.com/fitness/sifgps.f2p3enc.ee"
    wget "https://gpsquickfix.services.tomtom.com/fitness/sifgps.f2p7enc.ee"
    wget "https://gpsquickfix.services.tomtom.com/fitness/sifglo.f2p3enc.ee"
    wget "https://gpsquickfix.services.tomtom.com/fitness/sifglo.f2p7enc.ee"
    popd
    sleep 600 
done

)

remcokranenburg commented 8 months ago

Likely we won't be able to decypher the format completely, but I'm trying to get as much info as possible. It might be worthwhile to ask Telit for help once we know exactly what we need.

scubajorgen commented 8 months ago

I encountered a site where they had the ephemeris data in plain text. Might be helpful when deciphering. I thougt I read in the telit doc that the files contain the ephemeris data for 4 hour intervals

remcokranenburg commented 8 months ago

I found a driver for a Sirf chip. I don't fully understand it, and I'm not sure if it's for the SirfstarV, but here it is: https://gitlab.com/gpsd/gpsd/-/blob/master/drivers/driver_sirf.c

krzysztof-michalak commented 7 months ago

As of Nov 1st the ChoariStudio files seem to be identical to the TomTom ones. I am testing to see if they work (including the 14 days file...),

scubajorgen commented 7 months ago

@krzysztof-michalak please check my findings on #42 at the bottom. I find Choaristudio to differ from TomTom, like earlier. How can this be?

@remcokranenburg do you still download the TomTom files? At what time does TomTom refresh the files? Choaristudio apparently does it at 14:24 at whatever timezone.

alois-git commented 7 months ago

Some findings

https://github.com/felixge/node-ar-drone/issues/74

https://www.linkedin.com/pulse/how-does-global-positioning-system-work-talvinder-singh/?trk=mp-reader-card

remcokranenburg commented 3 months ago

Sorry for dropping off the face of the earth! I actually stopped running for a while, so this project faded from my memory. I see that this application can still get the ephemeris files from TomTom without trouble, that's good news! Never know how long that might last though.

I downloaded the 4 different files every 10 minutes starting 2023-10-06 09:15 until 2023-10-08 20:31 UTC, in total 1424 files. They zip quite terribly, but 7zip got it down to a few kilobytes, so it looks like that algorithm found some repetition!

I attached it here, but I had to rename it to .zip for GitHub to accept it. Rename to .7z before opening!

TomTomQuickGPS.zip

scubajorgen commented 3 months ago

@remcokranenburg Thanx for the response. In the meantime I had written a download script myself and watched the files for a while. We now have two alternatives:

ZX-Commodore-ST commented 2 months ago

https://choaristudio.com doesn't work anymore. There is a new address. Looks a lot like the last one. https://www.landquest.fr/agps/ https://mail.hieroglyphe.net/agps/

scubajorgen commented 2 months ago

https://choaristudio.com doesn't work anymore. There is a new address. Looks a lot like the last one. https://www.landquest.fr/agps/ https://mail.hieroglyphe.net/agps/

Thanx. All three links resolve to the same IP, so I guess they're the same owner. I once contacted them by mail but no response.

ZX-Commodore-ST commented 1 month ago

Just a quick feedback on 1.12 version - tomtomwatch.properties: "https://mail.hieroglyphe.net/agps/agps/f2p{DAYS}enc.ee" throws out some error on my PC. http://mail.hieroglyphe.net/agps/f2p{DAYS}enc.ee is working fine with 14 days. http instead of https.

scubajorgen commented 1 month ago

Just a quick feedback on 1.12 version - tomtomwatch.properties: "https://mail.hieroglyphe.net/agps/agps/f2p{DAYS}enc.ee" throws out some error on my PC. http://mail.hieroglyphe.net/agps/f2p{DAYS}enc.ee is working fine with 14 days. http instead of https.

That is odd... I just tried https://mail.hieroglyphe.net/agps/agps/f2p7enc.ee in the browser and it works. I also tried your link: http://mail.hieroglyphe.net/agps/f2p7enc.ee. It works but it redirects to https.

However, the https links don't work in TomTomWatch. I have to look into it. Maybe it has to do that I use a http request, maybe the problem is that the https sites appear not to have a valid certificate.

Yup! Checked: The message on https is: java.security.cert.CertificateException: No subject alternative DNS name matching mail.hieroglyphe.net found. It is odd that if you use http, you get redirected to https, and then it doesnt check the certificate??!!

scubajorgen commented 1 month ago

Just a quick feedback on 1.12 version - tomtomwatch.properties: "https://mail.hieroglyphe.net/agps/agps/f2p{DAYS}enc.ee" throws out some error on my PC. http://mail.hieroglyphe.net/agps/f2p{DAYS}enc.ee is working fine with 14 days. http instead of https.

You can now use the setting tlsCertificateCheckDisable=true to disable TLS Certificate Validation. USE AT OWN RISK. IS BAD PRACTICE. But the https links will work then.

ZX-Commodore-ST commented 1 month ago

Some browsers change http to https without asking. I have no problem with http or disabled check for TSL certificate as we only download public available data without logging or sending anything other than request for a file.