Anomalies in UBX packets RXM-RAWX and NAV-PVT causing issue in file processing

[cturvey]

Hi Tim,

Ok, so had a report this morning of RINEX generation failures, where it appeared the Count in RXM-RAWX and Used in NAV-PVT had the high order bit (7) set. Not sure which NEO-M8P HPG firmware this is, but there were several anomalies in the file provided, whilst checksums, etc, suggested they were intact, and from the device. The NAV-PVT is probably not an issue for you, I need to check if any other messages are impacted

https://portal.u-blox.com/s/question/0D5Oj00000Ux8tRKAR/conversion-of-ubx-to-rinex-in-rtklib-fails-sometimes-only-converts-a-few-samples-rawx-and-sfrbx-messages-are-recorded

https://github.com/rtklibexplorer/RTKLIB/blob/16ea4949e31c45661a869f5aa471b378b5dad7f1/src/rcv/ublox.c#L373 nmeas=U1(p+11); / numMeas /

nmeas &= 0x7F; // mask the satellite count, pending an understanding of what bit 7 communicates

---UBX-----------------------------------------------------------------------
0000 : B5 62 02 15 10 01 89 41-60 E5 37 83 01 41 1F 09 .b.....A`.7..A..
0010 : 12 88 01 01 02 05 9C A0-BA 9C 73 2F 6F 41 D0 49 ..........s/oA.I  <<< The 0x88 here should be interpreted as 0x08
0020 : 46 46 2A 7C 94 41 64 AA-A6 C4 00 0F 00 00 28 00 FF*|.Ad.......(.
0030 : 1E 09 07 0A 01 00 C9 92-07 49 4C A5 70 41 15 80 .........IL.pA..
0040 : FD 75 51 DE 95 41 18 0F-0D C5 00 16 00 00 98 35 .uQ..A.........5
0050 : 2B 06 01 08 0F 00 6C 40-48 35 8F 3B 70 41 5E 2C +.....l@H5.;pA^,
0060 : 2E 4C 67 53 95 41 1A 1D-2F C5 00 0D 00 00 70 92 .LgS.A../.....p.
0070 : 25 08 03 08 0F 00 92 A7-8C AE A8 D4 70 41 CD 60 %...........pA.`
0080 : 50 EF 89 1C 96 41 B0 74-DF 44 00 18 00 00 00 00 P....A.t.D......
0090 : 24 0C 0F 0C 01 00 CE F7-5A A5 E4 7C 71 41 65 DE $.......Z..|qAe.
00A0 : 9E D8 8E F9 96 41 0E F0-48 C5 00 0E 00 00 60 04 .....A..H.....`.
00B0 : 30 07 02 08 03 00 AA FB-BC F0 3D E5 71 41 19 F4 0.........=.qA..
00C0 : 29 F1 A5 82 97 41 D4 9A-4A 44 00 11 00 00 AC 08 )....A..JD......
00D0 : 2E 07 01 08 03 00 E2 44-8C 3C 6F 2A 72 41 57 BA .......D.<o*rAW.
00E0 : 50 B1 8C DD 97 41 00 AA-46 C3 00 17 00 00 00 00 P....A..F.......
00F0 : 23 0E 0F 0E 01 00 18 42-D0 A4 FC C9 72 41 6D 90 #......B....rAm.
0100 : F0 89 29 AF 98 41 2A 67-1D 45 00 13 00 00 04 33 ..)..A*g.E.....3
0110 : 1D 08 04 08 0F 00 DF C0-                        ........

02 15 RXM-RAWX        - DF C0 : DF C0 280
!!! RXM RAWX COUNT HIGH BIT SET !!!
RXM-RAWX - TOW:143462.987 WEEK:2335 COUNT: 8 STAMP:24 10  7 15 51  2.9870000
Leap Valid 18
Reserved 01 02 05  1282
280 280
M8 Version
GPS      15 L1CA       85920401.569    16350108.898       -1333.325  30      P        40
GPS      22 L1CA       91722845.498    17454276.564       -2256.943  43 HALF P C H 13720
GPS      13 L1CA       89446867.045    17021171.330       -2801.819  37 HALF P C H 37488
GPS      24 L1CA       92742267.828    17648266.909        1787.646  36      P         0
GPS      14 L1CA       96363446.155    18337354.335       -3215.003  48      P C    1120
GPS      17 L1CA       98609532.291    18764767.046         810.419  46      P C    2220
GPS      23 L1CA      100098860.329    19048179.784        -198.664  35      P         0
GPS      19 L1CA      103533154.485    19701706.301        2518.448  29 HALF P C H 13060

End user UBX log file https://portal.u-blox.com/s/contentdocument/069Oj00000Dh4FNIAZ

My analysis / RINEX, this is a .RAR file, uploaded as a .DOCX so uBlox forum would host https://portal.u-blox.com/s/contentdocument/069Oj00000DhBqRIAV

I'm going to run this to ground over at uBlox

Regards, -Clive

[cturvey]

Here's an offending NAV-PVT packet for context, I don't think RTKLIB is implicitly using it. I've use on occasion to establish ECEF location for Rover RINEX antenna estimate in header. It might be flagging the drop-out of PVT reporting, I'm observing a 4 second absence of the message immediately prior.

---UBX-----------------------------------------------------------------------
0000 : B5 62 01 07 5C 00 50 66-9B 08 E8 07 0A 07 10 06 .b..\.Pf........
0010 : 18 B7 06 00 00 00 F3 DD-03 00 03 01 EA 8A 08 3F ...............?  <<< probably this 0x8A should be interpreted as 0x0A
0020 : 43 06 4F 2E A3 1C 67 02-0A 00 2B 4A 09 00 0A 07 C.O...g...+J....
0030 : 00 00 3B 0B 00 00 00 00-00 00 FD FF FF FF FA FF ..;.............
0040 : FF FF 03 00 00 00 25 77-C3 01 93 00 00 00 28 1F ......%w......(.
0050 : 44 00 B3 00 00 00 12 19-26 3B 00 00 00 00 00 00 D.......&;......
0060 : 00 00 B8 24                                     ...$

01 07 NAV-PVT         - B8 24 : B8 24 100
PVT DELTA TOW 4000 ??
NAV-PVT  144402000     253427          6 3 01 EA
   105070344   480456271      655975      608811        1802       2875
           0          -3          -6           3    29587237        147    4464424
 1.79 138
TOW:144402.000253427  1 16:06:42.000
Valid Fix
PSM is not active
3D Fix
!!! NAV PVT USED HIGH BIT SET !!!
LLH:   48.0456271   10.5070344  655.975  608.811   47.164
ACC:  3.39  HACC:  1.80  VACC:  2.88  PDOP:  1.79  USED: 10

[cturvey]

NEO-M8P-2 EXT CORE 3.01 (db0c89) ROM BASE 3.01 (107888) FWVER=HPG 1.40REF PROTVER=20.30

[cturvey]

Honestly, the more I look at this the more suspicious I'm getting with the SparkFun Artemis OpenLog implementation, or I2C transport.

The NEO-M8P firmware doesn't look to use these high bits, but there is a persistent presence in the OP's data logging. Like a timing, or pull-up issue on the I2C bus, or how the transfer deblocks over the bus. I can't currently explain the Fletcher checksum's being good, unless it's pattern specific, and it's dropping all the packets with bad checksums from the logging entirely. Which might be the case as the UBX-RXM-RAWX packets are relatively large, and the OP's complaining about packet loss too, as well as the inability to process the logged data.

@PaulZC

[Edit:OpenLog Artemis GNSS Data Logging links removed for clarity and reduced confusion]

[ourairquality]

There are other indications in the stream of data corruption, in the high bit. The svid numbers have occasional high bit set, the svid for system GPS having the high bit set, the sigid having the high bit set, the gnss value having the high bit set. But not a single CRC error, and the CRC code does appear capable of detection this, it is working when high bit errors are injected, so it would not appear to be data corruption in the pipeline unless the data is being re-encoded.

Is there a vendor that is responsive to RTKLib level issues, and more so on what might be a legacy product? Shall be interesting to see if PocketSDR can be a substitute.

[cturvey]

Is there a vendor that is responsive to RTKLib level issues, and more so on what might be a legacy product?

Why would any be responsive? I'm not aware any selling off the back of RTKLIB The M9, M10 and F10 all have viability if there's some quantifiable value to implementing them. Similarly fractional CNO can be recovered.

CRC code does appear capable of detection this

The Fletcher sum doesn't really have the rigor of a CRC, I didn't try flipping bits to confirm, but I am suspicious that only packets with a viable sum are logged, my tools didn't flag any errors in the two exemplar files, but would also explain the missing packets. The speed of the logger could be helped significantly if aligned 8KB or 32KB were flushed instead of the tetris method employed. Basically reading the available() bytes on the receiver's I2C interface, and dumping to the file in a more direct / minimal step process rather than parsing / de-muxing packets. This would make a lot more sense, especially at 5, 8 or 10 Hz rates, when it really just needs to fire-hose the data on to the card.

[PaulZC]

Hi Clive (@cturvey ),

Please delete the two links in your post above. I believe they're not relevant and will just confuse anyone following them.

I believe the user is using the SparkFun OpenLog_Artemis_GNSS_Logger firmware. I have opened an issue here:

https://github.com/sparkfun/OpenLog_Artemis_GNSS_Logger/issues/34

Best, Paul