This PR adresses the current bottleneck of prx's speed, which is observation file parsing, by adding
a benchmark for parsing speed by splitting a 24h, 30s obs interval file into a sequence of smaller files and recording the parsing time
a faster rinex 3.05 obs parser making heavy use of pandas
a unit test comparing georinx parser outputs to those of the new prx parser
Note that the PR adds the new parser, but does not enable it, so prx still uses georinex as its parsing backend.
What about the loss-of-lock indicator
The loss-of-lock indicators are not extracted yet, will deal with that in a follow-up PR.
What about nav files?
We can continue to rely on georinex's nav file parser, since parsing e.g. a 1-day file takes only about 45 seconds (M2 Mac), and most web-prx users will benefit from caching, since it is likely that a previous users has caused prx to download and parse the same nav files.
Testing
Running the benchmark yields
Note that the pandas-based parser's speed is pretty much independent of file size, at least in this benchmark.
jtec
commented
2 months ago
This PR adresses the current bottleneck of prx's speed, which is observation file parsing, by adding
Note that the PR adds the new parser, but does not enable it, so prx still uses georinex as its parsing backend.
What about the loss-of-lock indicator
The loss-of-lock indicators are not extracted yet, will deal with that in a follow-up PR.
What about nav files?
We can continue to rely on georinex's nav file parser, since parsing e.g. a 1-day file takes only about 45 seconds (M2 Mac), and most web-prx users will benefit from caching, since it is likely that a previous users has caused prx to download and parse the same nav files.
Testing
Running the benchmark yields
Note that the pandas-based parser's speed is pretty much independent of file size, at least in this benchmark.