search

semuconsulting / pyubx2

Python library for parsing and generating UBX GPS/GNSS protocol messages.
BSD 3-Clause "New" or "Revised" License
173 stars 67 forks source link

Is my ubx file compatible? #79

Closed jgguitare closed 2 years ago

jgguitare commented 2 years ago

Describe the bug Cannot run a simple import of my test.ubx file. pyubx2.version: 1.2.14 uCenter software used to generate the test.ubx file: v21.09

Error message: [some valid parsed data from (1)... to (114) with parsed string] Traceback (most recent call last): File "C:\Users\jgorisse\Documents_FFANT\SW\GNSS_analyzer\parser.py", line 9, in print(f'({str(i)})\t{parsed_data}') File "C:\Asulab\Python310\lib\site-packages\pyubx2\ubxmessage.py", line 848, in str val = ubt.UBX_MSGIDS[clsid + msgid] KeyError: b'\x13`'

python code: from pyubx2 import UBXReader stream = open('ubx_file.txt', 'rb') ubr = UBXReader(stream, protfilter=2) i = 0 for (raw_data, parsed_data) in ubr: i += 1 print(f'({str(i)})\t{parsed_data}')

ubx file (ubx_file.ubx, extension simply changed to .txt): ubx_file.txt

To Reproduce

  1. save the python code to parser.py
  2. run the parser.py python code

Expected Behaviour

A string with parsed data.

Desktop (please complete the following information):

Additional context

uCenter has no problem to read back the above mentioned .ubx file. What I really want to do is to directly read the .ubx files (I have many of those files), parse the messages and build some numpy tables with C/N0, used of each satellites and then utc, lat, lon, alt, psm and fix that I can post-process with some already operational python codes. Now I have to extract those data from the table view of uCenter and save it in .csv files which is very time-consuming; this is why I am interested in this library to process the data with minimal human interaction.

semuadmin commented 2 years ago

Hi @jgguitare ,

The issue you're getting is essentially because the .ubx file created by u-center v21.09 includes by default a bunch of diagnostic and debugging message classes which - until the most recent v1.2.15 - were not handled by pyubx2 as a) they're not publicly documented in any interface specification, and b) they're really only of relevance to u-blox support technicians. As it happens I've recently responded to a similar issue - see #78.

You can usually avoid this issue by selecting 'No' when prompted by u-center to specify whether you want to 'Add Receiver Configuration' to the recorded .ubx file, so I recommend you try this first:

u-center debug

As of v1.2.15, pyubx2 can now 'handle' these undocumented debugging messages, at least to the extent of parsing them to a nominal payload definition (the actual payload definitions are not publicly available), so I also recommend you update to this latest version.

However, having examined your .ubx file I've noticed another problem. There are CFG POLL requests for 3 legacy UBX message classes (CFG-PM, CFG-TMODE and CFG-TP) which are not relevant to NEO-M8N devices but u-center has polled them anyway. Ironically, pyubx2 had previously supported these message classes but they were removed some time ago as I understood them to be obsolete. I've now restored these three message classes and will upload the revised code shortly but - as I say - you should be able to avoid the problem in the first place by selecting 'No' to 'Add Receiver Configuration' in u-center when you record your .ubx file.

I've run your script using my updated version of pyubx2 and it appears to parse OK:

from pyubx2 import UBXReader
stream = open("/Users/semuadmin/quarantine/ubx_file.txt", "rb")
ubr = UBXReader(stream, protfilter=2)
i = 0
for (raw_data, parsed_data) in ubr:
    i += 1
    print(f"({i:03d})\t{parsed_data}")
(001)   <UBX(MON-VER)>
(002)   <UBX(MON-VER, swVersion=b'EXT CORE 3.01 (107900)\x00\x00\x00\x00\x00\x00\x00\x00', hwVersion=b'00080000\x00\x00', extension_01=b'ROM BASE 2.01 (75331)\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_02=b'FWVER=SPG 3.01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_03=b'PROTVER=18.00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_04=b'MOD=NEO-M8N-0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_05=b'FIS=0xEF4015 (100111)\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_06=b'GPS;GLO;GAL;BDS\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', extension_07=b'SBAS;IMES;QZSS\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')>
(003)   <UBX(CFG-ANT)>
(004)   <UBX(CFG-ANT, svcs=1, scd=1, ocd=0, pdwnOnSCD=1, recovery=1, pinSwitch=16, pinSCD=15, pinOCD=14, reconfig=1)>
...
(515)   <UBX(ACK-ACK, clsID=CFG, msgID=CFG-TP5)>
(516)   <UBX(ACK-NAK, clsID=CFG, msgID=CFG-TXSLOT)>
(517)   <UBX(CFG-USB)>
(518)   <UBX(CFG-USB, vendorID=5446, productID=424, reserved0=0, reserved1=0, powerConsumption=100, reEnum=0, powerMode=1, vendorString=b'u-blox AG - www.u-blox.com\x00\x00\x00\x00\x00\x00', productString=b'u-blox GNSS receiver\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', serialNumber=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')>
(519)   <UBX(ACK-ACK, clsID=CFG, msgID=CFG-USB)>

Hope this helps.

semuadmin commented 2 years ago

I've uploaded an updated version of pyubx2. Unfortunately there appears to be (yet another) temporary glitch with GitHub Actions workflow which is preventing me from generating a new release, but you can download and install the code manually if you like. But, as I say, selecting 'No' to 'Add Receiver Configuration' in u-center should avoid the problem so try that and let me know how you get on.

jgguitare commented 2 years ago

Thank you very much for your help.

I have upgrade pyubx2 up to the v1.2.16 and everything operates as expected. This entirely solves my problem to parse already collected .ubx log files. I still have to do some tests with uCenter, saving a file without the configuration messages. But as far as I have looked it, I could not reopen a .ubx file and save it without the configuration messages which means all my already stored measurements would be lost (hopefully not with the v1.2.16 of pyubx2). I will investigate with a uBlox dev-kit connected to my PC to investigate how uCenter really operates... This is more a uCenter issue now (or how I use it).

May I suggest for the undocumented configuration messages to just indicate them as "unsupported content" when parsing but avoiding the parser to crash. Or just adding a "try" in front of the search in your dictionary to catch cases where the configuration message is unknown and skip those messages. Because in practice the interest of your code - for me - is mainly to retrieve the NMEA messages from an *.ubx file; configuration messages are not very interesting in that case. But keep the ability to parse those files even if there are some configuration messages in it is of a major interest. Just an idea if it becomes too complicate to support obsolete, and undocumented configurations messages...

Thanks again for your quick support.

semuadmin commented 2 years ago

Hi @jgguitare

As it happens, I have just added a further enhancement to pyubx2 to allow it to parse ANY unrecognised (i.e. undocumented) UBX message class to a nominal payload definition UBX-NOMINAL (which is a simple byte array) and add the term 'NOMINAL' to the message identity - provided the message is well-formed (i.e. the length and checksum are valid).

For example, if it encounters unknown message class b"\xb5b\x55\x66\x02\x00\x33\x44\x34\xae", it will parse it as: 

<UBX(UNKNOWN-5566-NOMINAL, data_01=b'3', data_02=b'D')>

FYI if you want to filter unwanted message protocols or specific message types from your u-center .ubx file, you could use something like the following script:

# temp4.py sample code to strip unwanted config messages from *.ubx file

from pyubx2 import (
    UBXReader,
    protocol,
    UBX_PROTOCOL,
    NMEA_PROTOCOL,
    RTCM3_PROTOCOL,
)

INFILE = "ubx_file.txt"
OUTFILE = "ubx_filtered.ubx"

with open(OUTFILE, "wb") as output:
    with open(INFILE, "rb") as stream:
        ubr = UBXReader(stream, protfilter=7, quitonerror=2)
        i = 0
        n = 0
        for (raw_data, parsed_data) in ubr:
            i += 1
            # if you want to filter specific message types, use the identity function:
            # if parsed_data.identity in ["GNGSV", "GNRMC", "NAV-PVT"]:
            # if you want to filter specific protocol(s), use the protocol function:
            if protocol(raw_data) == NMEA_PROTOCOL:

                n += 1
                output.write(raw_data)

print(f"{i} messages read from {INFILE}")
print(f"{n} messages written to {OUTFILE}, {i-n} messages filtered")
➜  python3 temp4.py
6604 messages read from ubx_file.txt
6085 messages written to ubx_filtered.ubx, 519 messages filtered

Hope this helps