search

semuconsulting / pygnssutils

Python GNSS CLI utility library for reading, parsing and broadcasting NMEA, UBX, RTCM3, NTRIP and SPARTN protocols
BSD 3-Clause "New" or "Revised" License
84 stars 25 forks source link

gnssntripclient occasionally omits certain messages from certain casters #85

Closed semuadmin closed 1 month ago

semuadmin commented 1 month ago

pygnssutils Bug Report Template

Describe the bug

On enabling logging (--verbosity 3), it appears the 'missing' RTCM3 messages are being received but are failing the CRC check in pyubx2, which suggests that the data stream is being garbled or prematurely truncated. May be a low-level TCP socket handling issue.

gnssntripclient --server euref-ip.net --port 443 --https 1 --mountpoint DARE00GBR0 --ntripuser myuser --mypassword yourpassword --verbosity 2
2024-08-15 19:20:06.308 - INFO - pygnssutils.gnssntripclient - Streaming RTCM data from euref-ip.net:443/DARE00GBR0 ...
2024-08-15 19:20:06.309 - ERROR - pyubx2.ubxreader - Unknown protocol header b'$_'.
2024-08-15 19:20:06.309 - ERROR - pyubx2.ubxreader - Unknown protocol header b'\xd3\xcc'.
2024-08-15 19:20:06.311 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'Cs\x04'
2024-08-15 19:20:06.312 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'D\xec\x8b'
2024-08-15 19:20:06.313 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'v\xde\x1d'
2024-08-15 19:20:06.314 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1019
2024-08-15 19:20:06.315 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'\xff\xfb\xe9'
2024-08-15 19:20:06.315 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'\xfc!\x00'
2024-08-15 19:20:06.316 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1020
2024-08-15 19:20:07.436 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'@\x9c\x01'
2024-08-15 19:20:07.459 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'\xc3F\x06'
2024-08-15 19:20:07.460 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'E\x8c\x8e'
2024-08-15 19:20:07.460 - ERROR - pyubx2.ubxreader - Unknown protocol header b'$\xd3'.
2024-08-15 19:20:07.460 - ERROR - pyubx2.ubxreader - Unknown protocol header b'$\xa9'.
2024-08-15 19:20:07.460 - ERROR - pyubx2.ubxreader - Unknown protocol header b'\xd3\x88'.
2024-08-15 19:20:07.461 - ERROR - pyubx2.ubxreader - RTCM3 message invalid - failed CRC: b'p\x03?'
^C2024-08-15 19:20:10.866 - INFO - pygnssutils.gnssntripclient - Streaming terminated.

On examining the incoming NTRIP HTTP responses and data streams using Wireshark, the only significant difference I can see is that the euref-ip.net caster explicitly states it's using a 'chunked' Transfer-Encoding and the Win size is 65024, while for rtk2go.com it's 2097664.

euref-ip.net:

HTTP/1.1 200 OK
Ntrip-Version: Ntrip/2.0
Server: NTRIP BKG Caster/2.0.45
Date: Thu, 15 Aug 2024 18:56:51 GMT
Cache-Control: no-store,no-cache,max-age=0
Pragma: no-cache
Connection: close
Content-Type: gnss/data
Transfer-Encoding: chunked
euref_wireshark

rtk2go.com:

HTTP/1.1 200 OK
Date: Thu, 15 August 2024 18:55:19 UTC
Server: SubCarrier Systems Corp SNIP simpleNTRIP_Caster_[3.15.12]RwPRO
Ntrip-Version: Ntrip/2.0
Cache-Control: no-store, no-cache, max-age=0
Pragma: no-cache
Connection: close
Content-Type: gnss/data
rtk2go_wireshark

To Reproduce

Steps to reproduce the behaviour:

  1. Run gnssntripclient with an igs caster e.g. euref-ip.net using NTRIP 2.0.

Expected Behaviour

All RTCM3 data from the designated caster should be received and parsed OK.

semuadmin commented 1 month ago

Hi,

Unfortunately gnssntripclient doesn't currently support Transfer-Encoding: chunked, which is only available in NTRIP 2.0 (since HTTP/1.1). A fix is in hand in RC 1.1.0, but as a temporary workaround, select NTRIP version 1.0 instead (--ntripversion 1.0). The IGS casters like euref-ip.net, igs-ip.net etc. still support NTRIP 1.0 (as do many others) which does not use chunked encoding - you should find the stream works fine then (you may see a handful of Unknown protocol header messages at the beginning of a verbose log - this is just residual data being flushed from the buffer):

gnssntripclient --server euref-ip.net --port 443 --https 1 --mountpoint DARE00GBR0 --ntripversion 1.0 --ntripuser youruser --ntrippassword yourpassword --verbosity 2
2024-08-18 08:15:03.618 - INFO - pygnssutils.gnssntripclient - Streaming RTCM data from euref-ip.net:443/DARE00GBR0 ...
Unknown protocol header b'\xd3p'.
Unknown protocol header b'\xb5\xde'.
Unknown protocol header b'\xb5k'.
Unknown protocol header b'$\xf8'.
Unknown protocol header b'\xd3\xf3'.
2024-08-18 08:15:03.622 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1127
2024-08-18 08:15:03.624 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1097
2024-08-18 08:15:03.626 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1087
2024-08-18 08:15:03.628 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1077
2024-08-18 08:15:03.628 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1006
2024-08-18 08:15:03.628 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1230
2024-08-18 08:15:03.628 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 4094
2024-08-18 08:15:03.628 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1019
2024-08-18 08:15:03.629 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1042
2024-08-18 08:15:03.629 - INFO - pygnssutils.gnssntripclient - RTCMMessage received: 1046
...