geospace-code / georinex

Python RINEX 2 / 3 NAV / OBS / sp3 reader & batch convert to HDF5 with C-like speed
MIT License
220 stars 91 forks source link

Error loading BRDC00IGS_R_20240840000_01D_MN.rnx #105

Open kstamatiou opened 6 months ago

kstamatiou commented 6 months ago
  1. Said Rinex file is the daily IGS multi-constellation broadcast ephemeris in rinex v3 format. File format is as follows:
    
     3.04           N: GNSS NAV DATA    M: MIXED            RINEX VERSION / TYPE
    MergeMNfile.tcl     IGS                 20240325 193506 GMT PGM / RUN BY / DATE
    BDSA   3.8184E-08  4.4703E-07 -3.8743E-06  7.0930E-06 Q 02  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 05  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 06  IONOSPHERIC CORR
    BDSA   3.6322E-08  5.3644E-07 -4.2915E-06  7.5102E-06 G 07  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 Q 08  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 09  IONOSPHERIC CORR
    BDSA   3.6322E-08  5.5879E-07 -4.2319E-06  6.9737E-06 K 10  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 U 11  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 S 12  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 S 13  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 14  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 16  IONOSPHERIC CORR
    BDSA   3.5390E-08  5.9605E-07 -4.3511E-06  7.1526E-06 O 19  IONOSPHERIC CORR
    BDSA   3.2596E-08  5.5879E-07 -3.9935E-06  6.3181E-06 A 20  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 S 21  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 Q 22  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 23  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 24  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 25  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 26  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 27  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 28  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 X 29  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 30  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 32  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 33  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 S 34  IONOSPHERIC CORR
    BDSA   4.0978E-08  2.6077E-07 -2.1458E-06  3.1590E-06 Q 35  IONOSPHERIC CORR
    BDSA   3.6322E-08  5.5879E-07 -4.2319E-06  6.9737E-06 K 36  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 S 37  IONOSPHERIC CORR
    BDSA   3.5390E-08  5.9605E-07 -4.3511E-06  7.1526E-06 O 38  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 39  IONOSPHERIC CORR
    BDSA   3.6322E-08  5.3644E-07 -4.2915E-06  7.5102E-06 G 40  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 41  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 42  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.0350E-07 -4.4703E-06  7.3314E-06 U 43  IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 Q 44  IONOSPHERIC CORR
    BDSA   3.6322E-08  5.6624E-07 -4.2319E-06  6.8545E-06 M 45  IONOSPHERIC CORR
    BDSA   3.6322E-08  5.3644E-07 -4.2915E-06  7.5102E-06 G 46  IONOSPHERIC CORR
    BDSA  -9.3132E-09 -6.8545E-07  1.1921E-07  3.3379E-06 D 50  IONOSPHERIC CORR
    BDSB   8.8064E+04  7.2090E+05 -5.3084E+06  5.8327E+06 Q 02  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 05  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 06  IONOSPHERIC CORR
    BDSB   9.2160E+04  1.0322E+06 -7.7332E+06  7.9299E+06 G 07  IONOSPHERIC CORR
    BDSB   1.1674E+05  5.5706E+05 -5.6361E+06  6.6191E+06 Q 08  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 09  IONOSPHERIC CORR
    BDSB   8.8064E+04  1.1305E+06 -8.3231E+06  8.3231E+06 K 10  IONOSPHERIC CORR
    BDSB   1.1674E+05  5.2429E+05 -5.2429E+06  6.0948E+06 U 11  IONOSPHERIC CORR
    BDSB   1.1059E+05  6.5536E+05 -6.0293E+06  6.8157E+06 S 12  IONOSPHERIC CORR
    BDSB   1.1059E+05  6.5536E+05 -6.0293E+06  6.8157E+06 S 13  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 14  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 16  IONOSPHERIC CORR
    BDSB   1.0445E+05  7.7005E+05 -6.8157E+06  7.5366E+06 O 19  IONOSPHERIC CORR
    BDSB   9.2160E+04  1.1469E+06 -8.3231E+06  8.3231E+06 A 20  IONOSPHERIC CORR
    BDSB   1.1059E+05  6.5536E+05 -6.0293E+06  6.8157E+06 S 21  IONOSPHERIC CORR
    BDSB   1.1674E+05  5.5706E+05 -5.6361E+06  6.6191E+06 Q 22  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 23  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 24  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 25  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 26  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 27  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 28  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 X 29  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 30  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 32  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 33  IONOSPHERIC CORR
    BDSB   1.1059E+05  6.5536E+05 -6.0293E+06  6.8157E+06 S 34  IONOSPHERIC CORR
    BDSB   1.2902E+05 -3.1130E+05  1.3107E+06 -3.2768E+05 Q 35  IONOSPHERIC CORR
    BDSB   8.8064E+04  1.1305E+06 -8.3231E+06  8.3231E+06 K 36  IONOSPHERIC CORR
    BDSB   1.1059E+05  6.5536E+05 -6.0293E+06  6.8157E+06 S 37  IONOSPHERIC CORR
    BDSB   1.0445E+05  7.7005E+05 -6.8157E+06  7.5366E+06 O 38  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 39  IONOSPHERIC CORR
    BDSB   9.2160E+04  1.0322E+06 -7.7332E+06  7.9299E+06 G 40  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 41  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 42  IONOSPHERIC CORR
    BDSB   1.1674E+05  5.2429E+05 -5.2429E+06  6.0948E+06 U 43  IONOSPHERIC CORR
    BDSB   1.1674E+05  5.5706E+05 -5.6361E+06  6.6191E+06 Q 44  IONOSPHERIC CORR
    BDSB   8.8064E+04  1.1305E+06 -8.3231E+06  8.3231E+06 M 45  IONOSPHERIC CORR
    BDSB   9.2160E+04  1.0322E+06 -7.7332E+06  7.9299E+06 G 46  IONOSPHERIC CORR
    BDSB  -1.8842E+05 -7.5366E+05 -7.3400E+06  4.1288E+06 D 50  IONOSPHERIC CORR
    GAGP  4.8312358558E-09-2.664535259E-15      0 2307          TIME SYSTEM CORR
    GAL    1.6725E+02 -1.0547E-01  7.2937E-03                   IONOSPHERIC CORR
    GPSA   3.9116E-08  1.4901E-08 -1.7881E-07  0.0000E+00       IONOSPHERIC CORR
    GPSB   1.4336E+05  0.0000E+00 -3.2768E+05  2.6214E+05       IONOSPHERIC CORR
    QZSA   1.0245E-08 -1.4901E-08 -4.1723E-07  1.1921E-06       IONOSPHERIC CORR
    QZSB   1.0240E+05  0.0000E+00 -8.5197E+05  8.3231E+06       IONOSPHERIC CORR
    BDSA   3.4459E-08  6.1095E-07 -4.4703E-06  7.3910E-06 W 60  IONOSPHERIC CORR
    BDSB   1.1469E+05  5.2429E+05 -5.1773E+06  6.0293E+06 W 60  IONOSPHERIC CORR
    GAUT  1.8626451492E-09-8.881784197E-16 518400 2306          TIME SYSTEM CORR
    GPUT -2.7939677238E-09-1.776356839E-15 233472 2307          TIME SYSTEM CORR
    GLUT  3.2596290112E-09 0.000000000E+00      0    0          TIME SYSTEM CORR
    QZUT -1.8626451492E-09 0.000000000E+00 356352 2158          TIME SYSTEM CORR
    BDUT -9.3132257462E-10 2.042810365E-14 604745  950          TIME SYSTEM CORR
    18    18  1929     7                                    LEAP SECONDS
                                                            END OF HEADER
    G01 2024 03 23 23 59 44 1.756171695888E-04 2.160049916711E-12 0.000000000000E+00
     2.000000000000E+01 4.296875000000E+01 3.556219559390E-09 2.959257184757E+00
     2.056360244751E-06 1.293790643103E-02 1.289322972298E-05 5.154013608932E+03
     6.047840000000E+05 4.656612873077E-08-2.990394506620E+00-1.918524503708E-07
     9.909930100366E-01 1.444062500000E+02 1.002905023843E+00-7.299589771500E-09
     1.003613233091E-10 1.000000000000E+00 2.306000000000E+03 0.000000000000E+00
     2.000000000000E+00 6.300000000000E+01 5.122274160385E-09 2.000000000000E+01
     6.012180000000E+05 4.000000000000E+00

2. The error that `georinex.load()` returns is:

ValueError Traceback (most recent call last) Cell In[2], line 2 1 filepath = "/home/kostas/Downloads/BRDC00IGS_R_20240840000_01D_MN.rnx" ----> 2 dataset = gr.load(filepath)

File ~/zephr-xyz/zephyrus/.venv/lib/python3.10/site-packages/georinex/base.py:65, in load(rinexfn, out, use, tlim, useindicators, meas, verbose, overwrite, fast, interval) 62 info = rinexinfo(rinexfn) 64 if info["rinextype"] == "nav": ---> 65 return rinexnav(rinexfn, outfn, use=use, tlim=tlim, overwrite=overwrite) 66 elif info["rinextype"] == "obs": 67 return rinexobs( 68 rinexfn, 69 outfn, (...) 77 interval=interval, 78 )

File ~/zephr-xyz/zephyrus/.venv/lib/python3.10/site-packages/georinex/base.py:166, in rinexnav(fn, outfn, use, group, tlim, overwrite) 164 nav = rinexnav2(fn, tlim=tlim) 165 elif int(info["version"]) == 3: --> 166 nav = rinexnav3(fn, use=use, tlim=tlim) 167 else: 168 raise LookupError(f"unknown RINEX {info} {fn}")

File ~/zephr-xyz/zephyrus/.venv/lib/python3.10/site-packages/georinex/nav3.py:98, in rinexnav3(fn, use, tlim) 95 duplicate = tu.size != t[svi][check].size 97 full_fields = fields[sv[0]] ---> 98 compact_fields = _sparefields(full_fields, sys=sv[0], N=_num_fields(raws[svi[0]])) 99 compact_index = [i for i, c in enumerate(full_fields) if c in compact_fields] 101 darr = np.empty((svi.size, len(full_fields)))

File ~/zephr-xyz/zephyrus/.venv/lib/python3.10/site-packages/georinex/nav3.py:234, in _sparefields(cf, sys, N) 231 cf = cf[:28] 233 if N != len(cf): --> 234 raise ValueError(f"System {sys} NAV data is not the same length as the number of fields.") 236 return cf

ValueError: System I NAV data is not the same length as the number of fields.



3. Operating system is Ubuntu 22.04.4. LTS, Python version is 3.10.10, and I have installed the latest georinex release.
betaBison commented 6 months ago

This is representative of a larger issue.

  1. For a temporary fix @kstamatiou, you could clone the repository and install as a development version. Then, change these lines to be
    elif sys == "I":
    if N < 28 and len(cf) >= 28:
        N = 28
        cf = cf[:28]
    elif N == 28:
        cf = cf[:28]

    Also, the cropped BRDC00IGS_R_20240840000_01D_MN.rnx file you posted in this issue should work with or without this fix since the issue is with IRNSS (system I) and not GPS (system G). I downloaded the full rinex file from IGS and have attached here a cropped version of the rinex that includes an "I02" entry.

BRDC00IGS_R_20240840000_01D_MN.zip

  1. I think the larger issue at hand is that #98 changed the logic of the _sparefields check. Now there are many possibilities of N being smaller than the standard value for its constellation since missing values that are treated as zero may be present at the end of the entry. In a previous commit, I made a patch to fix the same issue with QZSS, but in this issue the IRNSS entries have a similar problem.

@scivision, I'm happy to create a pull request to fix this specific IRNSS issue, but I don't know if you'd rather have me reevaluate the _sparefields function or even the _num_fields function to fix the overall problem and not just patch by constellation. Let me know what you'd think is best.

kstamatiou commented 6 months ago

Thanks a lot for the reply @betaBison.

Indeed, the problem is due to IRNSS, so the interim solution that I came up with is to simply truncate the file in order to exclude these entries; then load() works just fine.