berthubert / galmon

galileo open source monitoring
GNU General Public License v3.0
198 stars 53 forks source link

galmon

galileo/GPS/GLONASS/BeiDou open source monitoring. GPL3 licensed. (C) AHU Holding BV - bert@hubertnet.nl - https://berthub.eu/

Live website: https://galmon.eu/

Multi-vendor, with support for U-blox 8 and 9 chipsets and many Septentrio devices. Navilock NL-8012U receiver works really well, as does the U-blox evaluation kit for the 8MT. In addition, many stations have reported success with this very cheap AliExpress sourced device.

For ublox, there is good support for the F9P, several of us use the ArdusimpleRTK2B board. It adds the Galileo E5b band.

Septentrio devices support even more bands.

An annotated presentation about our project aimed at GNSS professionals can be found here.

NOTE: One of our programs is called 'ubxtool'. Sadly, we did not do our research, and there is another 'ubxtool' already, part of gpsd. You might have ended up on our page by mistake. Sorry!

To deliver data to the project, please read The Galmon GNSS Monitoring Project and consult the rules outlined in the operator guidelines.

Highlights

Data is made available as JSON, as a user-friendly website and as a time-series database. This time-series database is easily mated to the industry standard Matplotlib/Pandas/Jupyter combination (details here.

There is also tooling to extract raw frames/strings/words from specific timeframes.

Goals:

1) Support multiple wildly distributed receivers 2) Combine these into a forensic archive of all Galileo/GPS NAV messages 3) Make this archive available, offline and as a stream 4) Consume this stream and turn it into an attractive live website (https://galmon.eu/). As part of this, perform higher-level calculations to determine ephemeris discontinuities, live gst/gps/galileo time offsets, atomic clock jumps etc. 5) Populate an InfluxDB timeseries database with raw measurements and higher order calculations

Works on Linux (including Raspbian Buster on Pi Zero W), OSX and OpenBSD.

Build locally (Linux, Debian, Ubuntu)

To get started, make sure you have a C++17 compiler (like g++ 8 or higher), git, protobuf-compiler. Then run 'make ubxtool navdump' to build the receiver-only tools.

To build everything, including the webserver, try:

apt-get install protobuf-compiler libh2o-dev libcurl4-openssl-dev libssl-dev libprotobuf-dev \
libh2o-evloop-dev libwslay-dev libncurses5-dev libeigen3-dev libzstd-dev g++
git clone https://github.com/ahupowerdns/galmon.git --recursive
cd galmon
make

If this doesn't succeed with an error about h2o, make sure you have this library installed. If you get an error about 'wslay', do the following, and run make again:

echo WSLAY=-lwslay > Makefile.local

Building on OSX

With thanks to a contributor from Prague. First make sure you've installed brew, which you can get here. Then do:

brew install protobuf lzlib zstd h2o eigen

And then:

git clone https://github.com/ahupowerdns/galmon.git --recursive
cd galmon
make

Running in Docker

We publish official Docker images for galmon on docker hub for multiple architectures.

To run a container with a shell in there (this will also expose a port so you can view the UI too and assumes a ublox GPS device too - you may need to tweak as necessary):

docker run -it --rm --device=/dev/ttyACM0 -p 10000:10000 berthubert/galmon

Running a daemonized docker container reporting data to a remote server might look like:

docker run -d --restart=always --device=/dev/ttyACM0 --name=galmon berthubert/galmon ubxtool --wait --port /dev/ttyACM0 --gps --galileo --glonass --destination [server] --station [station-id] --owner [owner]

To make your docker container update automatically you could use a tool such as watchtower.

Running

On u-blox: Once compiled, run for example ./ubxtool --wait --port /dev/ttyACM0 --station 1 --stdout --galileo | ./navparse --bind [::1]:10000

For Septentrio, try: nc 192.168.1.1 29000 | ./septool --station x --stdout | ./navparse --bind [::1]:10000, assuming your Septentrio can be reached on 192.168.1.1.1 and you have defined an SBF stream on port 29000. For more details, please see below.

Next up, browse to http://[::1]:10000 (or try http://localhost:10000/ and you should be in business. ubxtool changes (non-permanently) the configuration of your u-blox receiver so it emits the required frames for GPS and Galileo. If you have a u-blox timing receiver it will also enable the doppler frames.

By default the ublox receiver module will be configured to use the USB port, if you want to use a different interface port on the ublox module then add the --ubxport <id> option using one of the following numeric IDs:

0 : DDC (aka. I2C)
1 : UART[1]
2 : UART2
3 : USB (default)
4 : SPI

To see what is going on, try:

./ubxtool --wait --port /dev/ttyACM0 --station 1 --stdout --galileo | ./navdump

To distribute data to a remote navrecv, use:

./ubxtool --wait --port /dev/ttyACM0 --galileo --station 255 --destination 127.0.0.1

This will send protobuf to 127.0.0.1:29603. You can add as many destinations as you want, they will buffer and automatically reconnect. To also send data to stdout, add --stdout.

Tooling:

Linux Systemd

First make sure 'ubxtool' has been compiled (run: make ubxtool). Then, as root:

mkdir /usr/local/ubxtool
cp ubxtool ubxtool.sh /usr/local/ubxtool/
cp ubxtool.service /etc/systemd/system/

Then please reach out as indicated in Operator.md to obtain your station ID and the receiver hostname and run:

echo RECEIVER-NAME > /usr/local/ubxtool/destination
echo STATION-NUMBER > /usr/local/ubxtool/station

Then start up the service (as root):

systemctl enable ubxtool
systemctl start ubxtool

To check if it is all working, do 'service ubxtool status'.

NOTE! If you don't use one of the AliExpress or Navilock devices, it may be that your U-blox is not connected to the USB-port of the U-blox chip but to the UART1 or UART2 port. If so, you'll need to edit the script so it finds your USB-to-serial adapter. At the very least you'll have to update the DEVICE line. You'll likely also have to add --ubxport 1 at the end, and likely also the baudrate (-b) and/or --rtscts=0.

To change the default constellations, create a file called /usr/local/ubxtool/constellations and set your favorites. To set all four constellations (which only F9-receivers support), do as root:

echo --gps --glonass --beidou --galileo > /usr/local/ubxtool/constellations

And then 'service ubxtool restart'.

Distributed setup

Run navrecv -b :: --storage ./storage to receive frames on port 29603 of ::, aka all your IPv6 addresses (and IPv4 too on Linux). This allows anyone to send you frames, so be aware.

Next up, run navnexus --storage ./storage -b ::, which will serve your recorded data from port 29601. It will merge messages coming in from all sources and serve them in time order.

Finally, you can do nc 127.0.0.1 29601 | ./navdump, which will give you all messages over the past 24 hours, and stream you more. This also works for navparse for the pretty website and influx storage, nc 127.0.0.1 29601 | ./navparse --influxdb=galileo, if you have an influxdb running on localhost with a galileo database in there. The default URL is http://127.0.0.1:29599/

Septentrio specifics

Unlike ubxtool, our septool does not (re)configure your Septentrio device. Instead, the tool expects Septentrio Binary Format (SBF) on input, and that this stream includes at least the following messages:

We currently parse and understand:

Support will be added soon for:

A typical invocation of septool looks like this:

nc 192.168.1.1 29000 | ./septool --station x --destination galmon-eu-server.example.com

Or to test, try:

nc 192.168.1.1 29000 | ./septool --station x --stdout | ./navdump

This is assuming that you can reach your Septentrio on 192.168.1.1 and that you have defined a TCP server stream on port 29000.

Septool will also accept input from a serial port or basically anything that can provide SBF. Please let us know if our tooling can make your life easier.

Internals

The transport format consists of repeats of:

1) Four byte magic value 2) Two-byte frame length 3) A protobuf frame

The magic value is there to help us resync from partially written data.

The whole goal is that we can continue to rebuild the database by rerunning 'navstore' and 'navinflux'.

Documents

Data sources

The software can interpret SP3 files, good sources:

To get SP3 GBM from GFZ Potsdam for GPS week number 2111:

WN=2111
lftp -c "mget ftp://ftp.gfz-potsdam.de/GNSS/products/mgnss/${WN}/gbm*sp3.Z"
gunzip gbm*sp3.Z

To feed data, use:

./sp3feed --sp3src=gbm --influxdb=galileo gbm*sp3

This will populate the sp3 tables in the database. A subsequent run of reporter, while setting the --sp3src parameter, will provide SP3 deviation statistics, and fill out the sp3delta table in there, which stores deviations from the SP3 provided position, per sp3src.

Further interesting (ephemeris) data is on http://mgex.igs.org/IGS_MGEX_Products.php

RTCM

RTCM is the Radio Technical Commission for Maritime Services, and confusingly, also the name of a protocol.

This protocol is proprietary, but search for a file called RTCM3.2.pdf or 104-2013-SC104-STD - Vers. 3.2.docx and you might find a copy.

This project can parse RTCM 10403.1 messages, and currently processes State Space Representation (SSR) messages, specifically types 1057/1240 (GPS/Galileo Orbit corrections to broadcast ephemeris) and 1058/1241 (GPS/Galileo Clock corrections to broadcast ephemeris).

RTCM messages need to be converted to protobuf format, and the rtcmtool is provided for this purpose.

RTCM is frequently transmitted over the internet using 'ntrip', a typical commandline to process RTCM in our project is:

$ ntripclient ntrip:CLKA0_DEU1/user:password@navcast.spaceopal.com:2101 | ./rtcmtool --station x --destination y

User and password can be obtained from https://spaceopal.com/navcast/ - the Galileo operating company.

The IGS also offers excellent streams, but without Galileo. Information is here. A typical commandline is:

$ ntripclient ntrip:IGS01/user:password@products.igs-ip.net:2101 | ./rtcmtool --station x --destination y

User and password can be requested through http://www.igs.org/rts/access

An interesting list is here: http://products.igs-ip.net/

There are many other sources of RTCM but currently not many offer the SSR messages we can use.

Tooling

Sample command lines

Look at old data:

$ ./navcat storage "2020-01-01 00:00" "2020-01-02 00:00" | ./navdump  

Global coverage (via volunteers)

In alphabetical order:

Additional sites are welcome (and encouraged) as the more data receiving sites that exist, then more accurate data and absolute coverage of each constellation can be had.

The galmon project is very grateful to all its volunteering receiving stations.

ubxtool