search

prichterich / compactgnss

A compact format for GNSS data
MIT License
0 stars 0 forks source link

Format description #2

Open prichterich opened 1 year ago

prichterich commented 1 year ago

Frames for minimal content used by the ESP logger need to be defined. This issue summarizes the current suggestion.

Minimal GNSS Frame (uncompressed)

Name Type Units Scaling Description
id u1 - - Type identifier (xE0)
flags u1 - - Flags (default 0)
hdop u2 - 2 x 10-2 HDOP
datetime u8 ms - Unix time
speed u4 m/s 2 x 10-3 Speed (over ground) in mm/s
serr u4 m/s 2 x 10-3 Speed error (SDOP/sAcc)
lat i4 deg 2 x 10-7 Latitude
long i4 deg 2 x 10-7 Longitude
cog i4 deg 2 x 10-5 Course over ground
sats u1 - - Number of satellites
fix u1 - - GPS fix (3 = 3D fix)
cksum u2 - - Checksum (ubx Fletcher)

TOTAL = 36 bytes

Minimal GNSS Frame (compressed)

Name Type Units Scaling Description
id u1 - - Type identifier (xD0)
flags u1 - - Flags (default 0)
hdop u2 - 2 x 10-2 HDOP
timedelta i2 ms - Delta time
speeddelta i2 m/s 2 x 10-3 Speed (over ground) in mm/s
serrdelta i2 m/s 2 x 10-3 Speed error (SDOP/sAcc)
latdelta i2 deg 2 x 10-7 Latitude
longdelta i2 deg 2 x 10-7 Longitude
cogdelta i2 deg 2 x 10-2 Course over ground scaled delta
sats u1 - - Number of satellites
fix u1 - - GPS fix (3 = 3D fix)
cksum u2 - - Checksum (ubx Fletcher)

TOTAL = 20 bytes

See https://github.com/Logiqx/open-gnss/discussions/22 for description of basic compression ideas.

Based on discussions on Seabreeze, https://raw.githubusercontent.com/Logiqx/open-gnss/main/docs/thoughts.md, and discussions at https://github.com/Logiqx/open-gnss/discussions

Notes: Type identifiers define the record type (also called frame type), with reserved ranges:

Flags are intended for "post-logger" use. For example, software like GPS Speedreader may interpolate missing records that will be saved to files, and the flags can be used to identify such records. Other bits could be used to identify smoothed data, average data (e.g. when down-sampling), or whether records have been/should be used for speed results (i.e. filter flags). The meaning of these flags has to be defined. For straight logging of data received from a GPS chip, all flags are set to 0.

Checksums are 16-bit Fletcher checksums (https://en.wikipedia.org/wiki/Fletcher%27s_checksum) with a modulus of 256. This is different from the more commonly used modulus of 255, but it is identical to how the checksum in u-blox (.ubx) files is calculated. The advantage of using a 256 modulus is that simplifies the calculation by avoiding divisions, instead allow the use of unsigned 8-bit integers (in C) or bit operators and cast operations (in Java), this allowing for simpler and faster code.

Additional descriptions and explanations to come, either here or in the discussions.

prichterich commented 1 year ago

Correction Dec 6, 2022 5:54 pm EST: lat, long, and cog are signed ints (i4) in the u-blox documentation, not unsigned.

RP6conrad commented 1 year ago

I assume the checksum is over all the bytes, except the two last checksum bytes (byte 0 to 33) ? For The Fletcher function, if I use a uint8_t for sum1 and sum2, the %256 is not needed anymore ?

uint16_t sum1,sum2; for ( int i = 4; i<(count-2); ++i ) { sum1 = (sum1 + data[i]) % 256;//divide by 256 in stead of 255 !! sum2 = (sum2 + sum1) % 256; }

Greetings, Jan.

Verzonden vanuit Mail voor Windows

Van: prichterich Verzonden: dinsdag 6 december 2022 23:56 Aan: prichterich/compactgnss CC: RP6conrad; Manual Onderwerp: Re: [prichterich/compactgnss] Format description (Issue #2)

Correction Dec 6, 2022 5:54 pm EST: lat, long, and cog are signed ints (i4) in the u-blox documentation, not unsigned. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

prichterich commented 1 year ago

I assume the checksum is over all the bytes, except the two last checksum bytes (byte 0 to 33) ?

Correct. Check the u-blox receiver protocol description for an example. Here's what it looks like in my Java implementation, using 4-byte ints for the 2 checksum bytes: image I have a writer and reader for the new format in Java (as part of GPS Speedreader). In first limited tests, they seem to work. File size can be indeed reduced to 20% of ubx (for 5 Hz data, 1 Hz should be 20.3%).

I need to clean up some things (e.g. handling checksum violations). I will also add a detailed point-to-point comparison of two files, so that it is possible to compare a ubx file and a gpy file so verify that all data in the gpy file are saved and read correctly. Once I got that, I can send you a pre-release version, so you can use it to check files you generate.

The header in Mike's original description is to close to the FIT file format, which adds unwanted complexity. The real format does not matter much to me, but there are changes to the start of the file header that do matter:

  1. First (type) byte is xF0
  2. Second byte is 0 (flags, unused so far)
  3. Next two bytes are uint2 and give the length of the header record, including everything from first byte to checksum bytes (included!).
  4. Checksum bytes are at the end of the record (like in all other records).

The length of the header record may be subject to change in the future, so having the length in bytes 3 and 4 will allow reading it even if something changed. The header length (with checksum bytes) should be a multiple of 4.