search

coverclock / com-diag-hazer

Parse NMEA sentences and other typical output from GNSS devices.
GNU Lesser General Public License v2.1
14 stars 6 forks source link

readme

com-diag-hazer

Parse NMEA strings and other typical output from GNSS devices.

Copyright

Copyright 2017-2024 by the Digital Aggregates Corporation, Colorado, USA.

Except where noted, this software is an original work of its author.

Trademarks

"Digital Aggregates Corporation" is a registered trademark of the Digital Aggregates Corporation, Arvada, Colorado, USA.

"Chip Overclock" is a registered trademark of John Sloan, Arvada, Colorado, USA.

License

Licensed under the terms in LICENSE.txt.

Contact

Chip Overclock
Digital Aggregates Corporation
3440 Youngfield Street, Suite 209
Wheat Ridge CO 80033 USA
http://www.diag.com
mailto:coverclock@diag.com

Disclaimer

I am a software developer. I have been writing code since circa 1970 and started working full time in the field in 1976. I have an M.S. (1983) and a B.S. (1980) in computer science from an accredited program at a university. What I am not is a hardware or radio engineer, nor am I an expert in global navigation satellite systems. Behind all the software work I present here is a vast collection of hardware, firmware, and radio frequency expertise contributed by the developers of the GNSS products I use.

If you're wondering why I don't use the excellent open source GPS Daemon (gpsd) and its GPS Monitor (gpsmon), the answer is: I have, in many projects, typically in conjunction with the open source NTPsec Daemon (ntpd). Hazer was developed as an excuse for me to learn in detail more about how GPS works and how NMEA and UBX sentences are formatted (because I only learn by doing), and also to develop an NMEA and UBX parsing library that I can incorporate into the kinds of embedded systems I am frequently called to work upon. Hazer and gpstool have also turned out to be really useful tools for testing and evaluating GNSS devices. Finally, Hazer's gpstool utility serves as a useful testbed for some of the underlying Diminuto library's more complex features like thread handling and General Purpose Input/Output.

I also make use of Lady Heather, a terrific real-time GNSS monitoring program. I run it, along with my own software, and a script that periodically runs NTP query (ntpq), 24x7 on a Raspberry Pi with a touch-sensitive LCD display. This makes it easy for me to keep an eye on GPS and other GNSS constellations and on my various NTP micro-servers that depend upon GNSS for their clock discipline.

Abstract

This file is part of the Digital Aggregates Corporation Hazer project. Hazer is an open source toolkit for interpreting data streams typically generated by GPS and other GNSS receivers and emitted from serial(ish) ports, including USB and Bluetooth.

Hazer has a layered architecture that allows it to be used in a variety of ways. You can use the gpstool application in full-screen mode, watching its output in real-time. The app can be run in the background, for example as a daemon, in "headless" mode, and you can display its full-screen output in real-time at will in one or more windows. You can run the app in report mode, in which it emits the same lines of output, in the same fixed format, to standard output, and your own application can parse the results, for example via a shell pipeline or even via a socket. Or you can write your own code and call the C functions in the underlying library directly.

While Hazer is the name of the repository, it incorporates features in support of several different related projects described below, some of which are works in progress, and which may be found in this or other Digital Aggregates repositories:

The Hazer repo includes parsers for four different input formats - NMEA, UBX (U-blox), RTCM, and CPO (Garmin) - and interprets many of the data records represented in those formats into C structures. For purposes of commenting in my C code, I find it useful to refer to NMEA records as "sentences" (terminology consistent with the NMEA standard), UBX and CPO records as "packets", and RTCM records as "messages".

NMEA sentences are printable ASCII messages that conform to the National Marine Electronics Association (NMEA) 0183 specification. NMEA 0183 describes the output produced by most receiving devices of Global Navigation Satellite Systems (GNSS), of which the U.S. Global Positioning System (GPS), formerly known as Navstar, is an example. The NMEA software stack is named Hazer (same as the Git repository).

UBX packets are in a proprietary binary data format generated by devices made by U-Blox, a Swiss designer and manufacturer of GNSS receivers. UBX packets may include not only data generated by the GNSS receiver, but also data generated by an Inertial Measurement Unit (IMU), part of a Inertial Navigation System (INS). UBX is supported by a parallel stack called Yodel.

RTCM messages are in a binary data format used in Differential GNSS (DGNSS) that makes use of Real-Time Kinematics (RTK). This allows communicating GNSS receivers to compare notes and potentially, over time, achieve very high degrees of precision down to centimeters. RTCM was developed by the Radio Technical Commision for Maritime services (RTCM). RTCM is supported by a parallel stack called Tumbleweed.

CPO packets are in a proprietary binary data format generated by some devices made by Garmin. CPO (no clue what this stands for, but Garmin uses it in their scant documentation) is supported by a parallel stack called Calico. The documentation for the Garmin CPO binary output format is slim enough that a lot of reverse engineering and, frankly, guesswork went into its implementation. It's highly experimental.

More broadly: Hazer uses GNSS to do geolocation for applications like moving map displays; Yodel uses GNSS with high precision geolocation and integrated inertial measurement features of specific u-blox devices; and Tumbleweed uses Differental GNSS with specific u-blox devices. All three projects reside in the Hazer repository.

When I use the term "Hazer", I am not very good about distinguishing whether I am talking about the software stack, the project, or the entire repo.

Hazer includes a gpstool application as a kind of Swiss Army knife for dealing with various GNSS devices. gpstool accepts data streams from standard input, from serial-ish devices including serial USB, from files or anything that looks like a file e.g. a FIFO (a.k.a. a named pipe), or from a UDP socket, and can send validated data to a remote UDP socket, write it to a device, and display the interpreted information.

Hazer also includes an rtktool application that is a point-to-multipoint RTCM router that can receive data via a UDP socket from a remote gpstool and forward it to one or more remote gpstools via UDP. This is used along with gpstool to implement a DGNSS system using a stationary base station in survey mode to provide real-time corrections to one or more mobile rovers. This project, called Tumbleweed (same as the RTCM software stack), does not use the Networked Transport of RTCM via Internet Protocol (Ntrip), but instead uses its own trivial data format consisting of raw RTCM messages preceeded by a four-byte sequence number carried over UDP datagrams.

Pro Tips

When you link against the library or use any of the binaries or scripts that are artifacts of the build process, the linker and the shell have to know where to find those artifacts. Furthermore, some of the binaries or scripts may depend upon values in your environment to work correctly.

You need to set the LANG (Language) environmental variable to set your locale to use the U.S. version of UTF-8. This allows applications to correctly display Unicode symbols like the degree symbol and the plus/minus symbol. If you use the bash shell (as I do), you can put the following line in your .profile in your home directory so that it is set everytime you log in (as I do). Or you can just set it when you need to.

export LANG=en_US.UTF-8

On some platforms, you may need to install or generate the Unicode locales for this to work. On Microsoft's Windows Subsystem for Linux (WSL) version 2, I did something like this.

sudo apt-get install locales-all
sudo update-locale LANG=${LANG}
sudo locale-gen $LANG

If you don't install libraries, binaries, and scripts in one of the usual system locations like /usr/local/lib and /usr/local/bin (I typically don't), you can temporarily modify your environment so that the linker and your shell can find them. This bash sourcing script is an artifact of the build process and sets the PATH and LD_LIBRARY_PATH environmental variables and exports them.

. ~/src/com-diag-hazer/Hazer/out/host/bin/setup

Some scripts in the bin and fun directories save stdout, stderr, CSV, or other data in files. By default, most scripts save such output files in the out/${TARGET}/tmp subdirectory of the code base, where TARGET is set to the name of the Makefile configuration file (typically "host"). This location can be overridden if the environmental variable below is set.

export COM_DIAG_HAZER_SAVDIR="${HOME}/save"

The libraries, binaries, and scripts make use of the Diminuto logging system. The importance of log messages is classified into eight severity levels, ranging from DEBUG (log mask 0x01, which may emit a firehose of information) to EMERGENCY (log mask 0x80, in which case your system is probably in deep trouble). You can control which level of messages are emitted, either to standard error (if your application has a controlling terminal), or to the system log (if your application, like a daemon, does not). One way to control this is to set the log mask in your environment.

export COM_DIAG_DIMINUTO_LOG_MASK=0xfe

The log mask value is an eight-bit number in decimal, hexadecimal, or even octal. In addition, the string ~0 can be used to enable all log levels, equivalent to 255, 0xff, or 0377. (Generally I find 0xfe to be a good starting point.)

You can alter the log mask in real-time for gpstool by creating a file named com_diag_hazer_log_mask.msk in the current working directory in which gpstool is running, and containing just the log mask numeric string in the same format as the value of the environmental variable above. gpstool imports the log mask from this file, if it exists (it's not an error if it doesn't), at start up, and whenever the process receives a SIGHUP signal.

If the environmental variable is not defined, and the file does not exist, the default log mask value for the Diminuto Log feature is used.

Manual Pages and Reference Manual

These PDFs of the manual pages and associated reference manual were built from Hazer's embedded Doxygen comments on 2024-03-25 using version 61.0.1 . They will not reflect changes made since then.

The unit tests (Hazer/tst), functional tests (Hazer/fun), and command line uilities (Hazer/bin) contain useful and non-trivial examples of how to use the Hazer library.

Dependencies

Diminuto

The Hazer library and its utilities depend on my Diminuto library. Diminuto is a general purpose C-based systems programming library that supports serial port configuration, socket-based communication, and a passle of other useful stuff. gpstool and rtktool are excellent examples of how to leverage Diminuto to get a lot done in not so much C code. I use Diminuto in virtually all of my C-based projects, and sometimes in other languages too that support C-linkage. Portions of Diminuto have also shipped in products from several of my clients. (Details of building Diminuto are shown below.)

Tool Chain

These days most Linux distributions do not include the basic tools to build C and C++ software (despite having themselves been developed using them).

sudo apt-get install gcc
sudo apt-get install g++
sudo apt-get install make

Documentation

If you want to make documentation, doxygen and related tools will need to be installed.

sudo apt-get install doxygen
sudo apt-get install texlive-base
sudo apt-get install texlive-fonts-recommended
sudo apt-get install texlive-fonts-extra
sudo apt-get install texlive-science

Utilities

Some of the Hazer scripts use command line utilities that not all Linux/GNU distros install by default.

sudo apt-get install bc
sudo apt-get install inotify-tools
sudo apt-get install socat
sudo apt-get install expect
sudo apt-get install sharutils

Workflow

For my own development workflow, I have also found it useful to install other packages that are often optional in various Linux/GNU distros. Your mileage may vary.

sudo apt-get install openssh-server
sudo apt-get install git
sudo apt-get install vim
sudo apt-get install screen

LFS

The dat directory is where I collect data from tests of Hazer (moving map), Tumblweeed (differential GNSS), and Yodel (integrated intertial measurement). Some of the comma separated value (CSV) files are too large (over a hundred megabytes uncompressed) to be pushed to GitHub in the normal way. These files use GitHub's Large File Storage (LFS).

sudo apt-get install git-lfs

Unless you install GitHub's LFS feature (which manifests as the "git lfs" command), files stored in LFS will appear to have contents that look something like this.

version https://git-lfs.github.com/spec/v1
oid sha256:bcb222ac76e51cad8010086754d4cadeeebd0161b51029d6870a0d806db5f42f
size 137136500

You have to initialize LFS once per machine for every repo it is used in.

git lfs install

You can also safely skip installing LFS, and ignore the large data files, to use this repo.

geodesic

The geodesic application is based on algorithms described in

Charles F. F. Karney, "Algorithms for geodesics", Journal for Geodesy, 2013-01, 87.1, pp. 43..55

and uses one .c file and one .h file, included in this repository, that were written by Mr. Karney. The original source files can be found at

https://geographiclib.sourceforge.io

and are licensed under the MIT license.

locale

The gpstool, rtktool, and mapstool programs depend on running in a POSIX locale that allows the use of Unicode characters like the degree symbol. Locales like "POSIX" and "C" don't support this, at least not on the systems I have. Locales like "en_US.UTF-8" work okay. Your mileage may vary.

dialout

GNSS modules typically express a serial port - either an actual serial port, a USB serial port emulation ('''ttyUSB'''), or a modem emulation ('''ttyACM'''). The easiest way to get read/write access to such a port as a non-root user is by adding that user to the '''dialout''' group. The user may have to log out and back in again to have this take effect.

sudo adduser pi dialout

Reverse Dependencies

The Tesoro project integrates Hazer with an OpenStreetMaps (OSM) tile server to create a moving map display. Tesoro steers the display by taking location updates, in the form of datagrams containing JSON over UDP, either in real-time or from playback, from Hazer using scripts like csvfollow and csvplayback.

Branching

Hazer is big and complex enough that I sometimes move to a "master" and "develop" dual branch model of development. I make and test major changes in the "develop" branch, and when I think I have a stable release, I merge "develop" into the "master" branch. I still make what I consider to be minor changes in the master branch.

Versioning

I tag major releases with a three-number tuple that is defined in the build Makefile for each project. Release numbers, e.g. 22.2.1, consist of a major number (22), a minor number (2), and a build number (1). The major number changes when I've made a change significant enough that I think applications using the library will likely need to be changed, even though they may compile. The minor number changes when I've added new features or functionality, but existing features and functionality haven't changed. The build number changes when I've fixed a bug so that existing features or functionality works as described (even though this may break workarounds in applications).

The major and minor numbers are incorporated into the name of the shared object (dynamic link library) produced by the build.

The vintage is the date and time of the most recent build in UTC and expressed in ISO8601 format.

The revision is the Git commit number.

The release, revision, vintage, and a bunch of other stuff, are embedded as modules in the Hazer and Diminuto libraries. Those modules can be linked into an application. Each project has a binary executable named vintage that displays this information.

The gpstool and rtktool applications can be run with a -V flag to display the release, vintage, and revision of the Hazer library with which they were built.

Repositories

https://github.com/coverclock/com-diag-hazer

https://github.com/coverclock/com-diag-diminuto

Related

https://github.com/coverclock/com-diag-hazer-dat

https://github.com/coverclock/com-diag-tesoro

https://github.com/coverclock/com-diag-codex

https://github.com/coverclock/com-diag-cesium

Sentences

Hazer processes the following standard NMEA sentences.

Hazer processes the following proprietary u-blox NMEA-style sentences.

Hazer recognizes and logs the following proprietary NMEA-style sentences.

Yodel processes or logs the following binary UBX packets.

Calico processes the following binary CPO packets.

Tumbleweed recognizes RTCM binary messages with a valid CRC but does not process their contents. As a special case, an RTCM message with a zero payload length is used by gpstool and rtktool as a keep alive message.

Talkers

Hazer recognizes the following talkers as belonging to the corresponding constellations and systems.

These talkers have been observed in the wild coming from actual GPS receivers.

Support for these talkers, which may not be in the NMEA standard, but which have been defined in the documentation of some devices I have used, has been unit tested, but the talkers have never been observed in the wild.

Identifiers

Receivers that implement the GSA System ID field introduced in NMEA 4.10 may reuse satellite identifiers. For example, I see GPS and Galileo both using identifier 9 on the U-Blox 9 receiver. Despite NMEA and vendor documentation to the contrary, I do not find the satellite identifer convention reliable and only use it as a last resort on older receivers that do not implement the System ID field to identify the constellation. Hazer recognizes the following satellite identifiers in the GSA and GSV messages.

These satellite identifiers have been observed in the wild coming from actual GNSS receivers.

Support for these satellite identifiers has been unit tested but has these identifiers have never been observed in the wild. (Hazer has successfully used GNSS devices that are receiving signals from the BeiDou and Galileo constellations, but those devices used the GSA System ID field so that Hazer does not have to infer the constellation from the satellite identifier.)

Devices

I have successfully tested Hazer with the following GPS chipsets.

I have successfully tested Hazer with the following serial-to-USB chipsets.

I have successfully tested Hazer with the following GNSS devices.

[^1]: A good inexpensive introductory GPS device easily acquired from numerous sources; it is my go-to device to regression test Hazer and gpstool.
[^2]: Emits all sorts of interesting stuff in unsolicited $GPTXT or $GNTXT sentences.
[^3]: Install udev rules in overlay to prevent ModemManager from toying with device.
[^4]: Receives GPS (U.S.) and GLONASS (Russia) constellations concurrently.
[^5]: Receives GPS (U.S.) or GLONASS (Russia) constellations via configuration.
[^6]: I have used the 1PPS indication exported via a digital output pin.
[^7]: I have used the 1PPS indication exported via Data Carrier Detect (DCD).
[^8]: Supports UBX.
[^9]: A software defined radio (SDR).
[^10]: Receives GPS (U.S.), GLONASS (Russia), Galileo (EU), and COMPASS (China) concurrently.
[^11]: Has integrated Inertial Measurement Unit (IMU).
[^12]: Equipped with a CR1220 3V button cell.
[^13]: A tear down of the BU-353W10 reveals it has a button cell for its battery backed RAM.
[^14]: Make sure you mount this component side down, patch antenna side up.
[^15]: Uses Bluetooth (see procedure for pairing in https://github.com/coverclock/com-diag-hazer/blob/master/REMARKS.md).
[^16]: Bluetooth pairs with Raspberry Pi after update to FW 3.0.0. [^17]: This SparkFun board requires opening (cutting) and closing (soldering) traces for dual UART operation.
[^18]: This U-Blox gen 8 receiver is embedded in a U-Blox LTE-M module which required a lot of config via AT commands.
[^19]: 1PPS (a.k.a. TPS in the docs) only has a pulse width of a few microseconds hence LED inoperative (bug).
[^20]: Timing and frequency reference claimed +/- 5ns.
[^21]: Supposed to receive both GPS and GLONASS but my GLO and both of my GLO 2 units just receive GPS.
[^22]: Rumored to differ from the original GLO only by its increased battery life.
[^23]: Receives GPS (U.S.), GLONASS (Russia), Galileo (EU), COMPASS (China), and NavIC (India) concurrently.
[^24]: The labels on the PVT (1pps) and PWR LEDs on this board are reversed.
[^25]: Vendor web site and OEM packaging correctly states GPS only.
[^26]: Emits GSAs with GLONASS id (2) but only emits GSVs with GPS talker (GP).
[^27]: Initially updates at 1Hz until it achieves a fix, than updates at 5Hz.
[^28]: In either NMEA or CPO (binary) mode.
[^29]: Enumerates multiple USB devices: Ethernet port with router and web server, serial ports, storage device. [^30]: Consider using a powered USB hub if EOFs typically indicating a USB disconnect are observed.

Note that when sending commands to and parsing output from GNSS devices, Postel's Law - "Be conservative in what you send, be liberal in what you accept." - applies. Also, there is plenty of ambiguity and underspecification to be found in standards like NMEA 0183.

Notes

GlobalSat is also known as GlobalSat Technology, USGlobalSat, US GlobalSat Technology, or GlobalSat Worldcom, but is apparently not the same as the Globalsat mobile satellite service provider.

GlobalTop Technology is also known as GTop, and is part of Sierra Wireless.

SiRF is part of Qualcom.

Devices (Other)

In the context of this repository, I also deal with devices that are not GNSS receivers but are related to navigation or timing, like intertial measurement units (IMUs). Output from these devices isn't processed by gpstool but by other means. The utility serialtool from the Diminuto repository is particularly useful in this respect.

Note that some devices are both GNSS receivers and IMUs, and these may be processed by gpstool.

Platforms

I routinely test Hazer and gpstool (and therefore, many of the features of the Diminuto library on which they depend) on the following development platforms, in no particular order.

Intel NUC7i7BNH
Intel Core i7-7567U x86_64 x2 x2
Ubuntu 22.04.2 LTS "Jammy Jellyfish"
Linux 5.19.0
GNU 11.3.0

Raspberry Pi 4 Model B BCM2835
ARM Cortex-A72 x4
Ubuntu 22.04.2 LTS "Jammy Jellyfish"
Linux 5.15.0
GNU 11.3.0

Raspberry Pi 4 Model B BCM2835
ARM Cortex-A72 x4
Raspbian 11 "bullseye"
Linux 6.1.19
GNU 10.2.1

I run very recent versions of Hazer and gpstool on the following test platforms, which run 24x7.

Raspberry Pi 3 Model B+ BCM2835
ARM Cortex-A72 x4
Raspbian 10 "buster"
Linux 5.10.103
GNU 8.3.0
(Differential GNSS Base Station)

Raspberry Pi 3 Model B+ BCM2835
ARM Cortex-A72 x4
Raspbian 9 "stretch"
Linux 4.19.66
GNU 6.3.0
(Differential GNSS Stationary Rover)

Raspberry Pi 3 Model B+ BCM2835
ARM Cortex-A72 x4
Raspbian 10 "buster"
Linux 4.19.50
GNU 8.3.0
(Differential GNSS RTCM Server)

Raspberry Pi 3 Model B+ BCM2835
ARM Cortex-A72 x4
Raspbian 9 "stretch"
Linux 4.14.30
GNU 6.3.0
(NTP Server Monitoring System)

In the past, I have tested various versions of Hazer and gpstool on the following platforms, which give you some idea of the wide variety of computers one may use.

Dell OptiPlex 7040
Intel Core i7-6700T x86_64 x4 x2
Ubuntu 16.04.2 "Xenial"
Linux 4.4.0
GNU 5.4.0

HP Envy x360 Convertible Model 15
AMD Ryzen 7 5825U x86_64 x16
Ubuntu 22.04.2 "Jammy Jellyfish"
Linux 5.15.90 (Microsoft 11 Windows Subsystem for Linux 2)
GNU 11.3.0

HP Mini 110-1100 Netbook
Intel Atom N270 i686 x2
Mint 19.3 "Tricia"
Linux 5.0.0
GNU 7.5.0

GPD Micro PC
Intel Celeron N4100 x86_64 x2 x2
Ubuntu MATE 19.10 "Eoan"
Linux 5.3.0
GNU 9.2.1

OrangePi 5
aarch64 x8
Ubuntu 22.04.2 LTS (Jammy Jellyfish)
Linux 5.10.110
GNU 11.3.0

Pi-Top 3 (Raspberry Pi 3 Model B+)
Broadcom BCM2837B0 Cortex-A53 ARMv7 x4
pi-topOS "Polaris" (Raspbian 9.9 "Stretch")
Linux 4.19.66
GNU 6.3.0

Pi-Top 4 (Raspberry Pi 4 Model B)
Broadcom BCM2711 Cortex-A72 ARMv8 x4
pi-topOS "Sirius" (Raspbian 10 "Buster")
Linux 5.4.79
GNU 8.3.0

Raspberry Pi Zero 2 W Rev 1.0
Broadcom BCM2835 ARMv7 x4
Raspbian 11 "Bullseye"
Linux 6.1.21
GNU 10.2.1

StarFive VisionFive (RISC-V SBC)
sifive u74-mc riscv64 x2
Fedora 33 "Rawhide"
Linux 5.15.10
GNU 10.3.1

StarFive VisionFive 2 (RISC-V SBC)
sifive u74-mc riscv64 x4
Debian 11.3.0
Linux 5.15.10
GNU 11.3.0

VMware Workstation 15 Pro under Windows 10
Intel Core i7-3520M x86_64 x2
Ubuntu 19.10 "Eoan" (Microsoft Common Base Linux "Mariner")
Linux 5.3.0
GNU 9.2.1

Your mileage may vary. Hazer and gpstool are not resource intensive. I routinely run gpstool on an old 32-bit Intel i686 netbook, and run several simultaneous instances of gpstool on a single Rasbperry Pi 4B.

Diagnostics

Errors

Error messages may be logged by gpstool with the following error numbers (errno) for malformed input data. (Errors reported by system calls and library functions may also cause error messages not related to the input data to be logged.)

Each Hazer library function that parses NMEA sentences (hazer), UBX packets (yodel), or RTCM messages (tumbleweed) returns a value >= 0 if the data were valid and any PNT structures passed to the function were updated. If the function returns a value < 0, the data were rejected, and errno is set to a value. If errno > 0, then the data were invalid and the error number, as indicated in the list above, suggests the reason why. If errno == 0, then the data were valid but indicated that the GNSS device did not have a valid solution, so the PNT structures were not updated.

Synchronization

The following log messages regarding input stream synchronization may be reported by gpstool.

Articles

Chip Overclock, "Better Never Than Late", 2017-02, http://coverclock.blogspot.com/2017/02/better-never-than-late.html

Chip Overclock, "Time and Space", 2017-09, https://coverclock.blogspot.com/2017/09/time-space.html

Chip Overclock, "A Menagerie of GPS Devices", 2018-04, https://coverclock.blogspot.com/2018/04/a-menagerie-of-gps-devices-with-usb.html

Chip Overclock, "Practical Geolocation", 2018-08, https://coverclock.blogspot.com/2018/08/practical-geolocation.html

Chip Overclock, "Practical Geolocation II", 2018-08, https://coverclock.blogspot.com/2018/08/practical-geolocation-ii.html

Chip Overclock, "We Have Met the Enemy and He Is Us", 2018-09, https://coverclock.blogspot.com/2018/09/we-have-met-enemy-and-he-is-us.html

Chip Overclock, "GPS Satellite PRN 4", 2018-11, https://coverclock.blogspot.com/2018/11/gps-satellite-prn-4.html

Chip Overclock, "This Is What You Have To Deal With", 2019-06, https://coverclock.blogspot.com/2019/06/this-is-what-you-have-to-deal-with.html

Chip Overclock, "Geolocation While Airborne", 2019-09, https://coverclock.blogspot.com/2019/09/geotagging-while-airborne.html

Chip Overclock, "When Learning By Doing Goes To Eleven", 2020-03, https://coverclock.blogspot.com/2020/03/when-learning-by-doing-goes-to-eleven.html

Chip Overclock, "Improvisational Engineering", 2020-04, https://coverclock.blogspot.com/2020/04/improvisational-engineering.html

Chip Overclock, "Keyhole", 2020-05, https://coverclock.blogspot.com/2020/05/keyhole.html

Chip Overclock, "Pseudorange Multilateration", 2020-05, https://coverclock.blogspot.com/2020/05/pseudorange-multilateration.html

Chip Overclock, "Dilution of Precision", 2020-05, https://coverclock.blogspot.com/2020/05/dilution-of-precision.html

Chip Overclock, "Practical Differential Geolocation", 2020-05, https://coverclock.blogspot.com/2020/05/practical-differential-geolocation.html

Chip Overclock, "Frames of Reference IV", 2020-05, https://coverclock.blogspot.com/2020/05/frames-of-reference-iv.html

Chip Overclock, "Negative Results Are Still Results", 2020-05, https://coverclock.blogspot.com/2020/05/negative-results-are-still-results.html

Chip Overclock, "Location, Location, Location", 2020-06, https://coverclock.blogspot.com/2020/06/location-location-location.html

Chip Overclock, "Headless", 2020-06, https://coverclock.blogspot.com/2020/06/headless.html>

Chip Overclock, "Dead Reckoning", 2020-09, https://coverclock.blogspot.com/2020/09/dead-reckoning.html

Chip Overclock, "A Moving Map Display Using OpenStreetMap", 2021-02, https://coverclock.blogspot.com/2021/02/a-moving-map-display-using.html

Chip Overclock, "The OpenStreetMap Moving Map In Real-Time", 2021-02, https://coverclock.blogspot.com/2021/02/the-openstreetmap-moving-map-in-real.html

Chip Overclock, "The OpenStreetMap Moving Map On Mobile Devices", 2021-03, https://coverclock.blogspot.com/2021/03/the-openstreetmap-moving-map-on-mobile.html

Chip Overclock, "A Static Route Map Display Using OpenStreetMap", 2021-03, https://coverclock.blogspot.com/2021/03/a-static-route-map-display-using.html

Chip Overclock, "Advanced Static Route Maps With OpenStreetMap", 2021-03, https://coverclock.blogspot.com/2021/03/advanced-static-route-maps-with.html

Chip Overclock, "Where the RF Meets the Road", 2021-03, https://coverclock.blogspot.com/2021/03/where-rf-meets-road.html

Chip Overclock, "Solar Power", 2022-03, https://coverclock.blogspot.com/2022/03/solar-power.html

Chip Overclock, "It's About Time", 2022-08, https://coverclock.blogspot.com/2022/08/its-about-time.html

Chip Overclock, "Bob Green Road", 2023-02-12, https://coverclock.blogspot.com/2023/02/it-will-probably-come-as-no-surprise.html

Chip Overclock, "Playing with a RISV-V SBC Running Linux", https://coverclock.blogspot.com/2022/08/playing-with-risc-v-sbc-running-linux.html

Chip Overclock, "Hazer with the U-blox NEO-F10T GNSS Receiver on the Ardusimple SimpleGNSS Board", 2023-07-03, https://coverclock.blogspot.com/2023/07/hazer-with-u-blox-neo-f10t-gnss.html

Media

John Sloan, "BU-353W10 Tear Down", album, https://flic.kr/s/aHsmWxTpfs

John Sloan, "Dead Reckoning", album, https://flic.kr/s/aHsmPsGqjA

John Sloan, "Decimal Degrees", album, https://flic.kr/s/aHsmV3GnBe

John Sloan, "GN803G and Hazer 8.0.0", video, https://youtu.be/ZXT_37PvmhE

John Sloan, "GPS Receiver Gallery", album, https://flic.kr/s/aHsmvoXqSH

John Sloan, "Hazer", album, https://flic.kr/s/aHskRMLrx7

John Sloan, "Hazer Test 2017-02-15 15:45 UTC", video, https://youtu.be/UluGfpqpiQw

John Sloan, "NGS AA7126", album, https://flic.kr/s/aHsmEnM8We

John Sloan, "NGS KK1446", album, https://flic.kr/s/aHsmFKdcgF

John Sloan, "NGS KK1770", album, https://flic.kr/s/aHskTG9K1h

John Sloan, "NTP Clock Gallery, album, https://flic.kr/s/aHsmgrizkL

John Sloan, "Tesoro", album, https://flic.kr/s/aHsmTB3tme

John Sloan, "Time and Space", album, https://flic.kr/s/aHsm3ghwxP

John Sloan, "Time and Space", video, https://youtu.be/szoT23ZBcVU

John Sloan, "Tumbleweed", album, https://flic.kr/s/aHsmcdEgq6

John Sloan, "Screen Recording 2021 02 18 at 8 55 20 AM", video, https://youtu.be/GjR7fPQRCZc

John Sloan, "Screen Recording 2021 02 24 at 10 06 20 AM", video, https://youtu.be/-6p3w1SxW1A

John Sloan, "Wheatstone", album, https://flic.kr/s/aHsmVu1wkK

John Sloan, "Yodel", album, https://flic.kr/s/aHsmNDPmDN

John Sloan, "UBX NEO F10T Video", video, https://youtu.be/gCKnlchwkzU

John Sloan, "UBX NEO F10T Screen", video, https://youtube.com/shorts/nC-NkjRSWRk?feature=share

References

David Burggraf ed., "OGC KML 2.3", Open Geospatial Consortium, 2015-08-04

Paul E. Ceruzzi, GPS, MIT Press, 2018

David Doyle, "NAD-83 vs WGS-84", NGS, 2000-05-10

David Doyle and Ed McKay, "NGS SURVEY MARKER ACCURACY", NGS, 2000-05-04

M. Dunn at al., "Navstar GPS Space Segment/Navigation User Interfaces", IS-GPS-200H, Global Positioning Systems Directorate / Systems Engineering & Integration, 2013-09-24

Dane E. Ericksen, "NAD 83: What Is It And Why You Should Care", 1994 SBE National Convention and World Media Expo, 1994

Amy Fox, "Precision Matters: The Critical Importance of Decimal Places", blis.com, 2017-07-09

Garmin, "GPS 18x TECHNICAL SPECIFICATIONS", 190-00879-08 Rev. D, Garmin International, Inc., 2011-10

Garmin, "Garmin Device Interface Specification", 001-00063-00 Rev. G, Garmin International, Inc., 2020-04-14

Garmin, "Garmin Proprietary NMEA 0183 Sentences TECHNICAL SPECIFICATIONS", 190-00684-00 Rev. C, Garmin International, Inc., 2008-12

Geocaching.com, "Benchmark Hunting", 2019-09-16

Mohinder S. Gerwal et al., Global Navigation Satellite Systems, Inertial Navigation, and Integration, Wiley, 2013

GIS Geography, "Geodetic Datums: NAD 27, NAD 83 and WGS 84", 2019-03-04

GPS NAVSTAR, "Global Positioning System Standard Positioning Service Signal Specification", 2nd Edition, 1995-06-02

John G. Grimes et al., Global Position System Standard Positioning Service Performance Standard, 4th edition, 2008-09

Grosse-Kunstleve RW, Terwilliger TC, Sauter NK, Adams PD, "Automatic Fortran to C++ conversion with FABLE", Source Code for Biology and Medicine 2012, 7:5

GTop, "PMTK command packet", Revision A11, GlobalTop, 2012

Tony Haddrell, Marino Phocas, Nico Ricquier, "Mobile Phone GPS Antennas - Can They Be Better?", GPS World, 2010-02

IGS et al., "RINEX - The Receiver Independent Exchange Format", Version 3.03, RTCM-SC104, 2015-07-14

ISO, "Standard representation of geographic point location by coordinates", Second edition, ISO 6709:2008(E), 2008-07-15

ISO, "Data elements and interchange formats - Information interchange - Representation of dates and times", First edition, ISO8601:1988(E), 1988-06-15

Elliott D. Kaplan ed., Understanding GPS principles and Applications, Artech House, 1996

Charles F. F. Karney, "Algorithms for geodesics", Journal for Geodesy, 2013-01, 87.1, pp. 43..55

G. Klyne, C. Newman, "Date and Time on the Internet: Timestamps", RFC3339, IETF, July 2002

H. Lyons, "Atomic Clocks", Scientific American, 1957-02

Richard Mason et al., Analyzing a More Resilient National Positioning, Navigation, and Timing Capability, Homeland Security Operational Analysis Center, RAND Corporation, 2021

Greg Milner, Pinpoint: How GPS is Changing Technology, Culture, and Our Minds, W. W. Norton, 2017-05-16

NMEA 0183, "Standard for Interfacing Marine Electronic Devices", Version 4.10, National Marine Electronics Association, 2012-06

NMEA 0183, "Standard for Interfacing Marine Electronic Devices", Version 4.11, National Marine Electronics Association, 2018-10

NMEA 0183, "Standard for Interfacing Machine Electronic Devices", Version 4.30, National Marine Electronics Association, 2023-12

NMEA 0183, "Technical Bulletin - NMEA 0183 Errata Version 4.11"", ERRATA #0183 20190507 GSV Sentence

NMEA 0183, "Technical Bulletin - NMEA 0183 Errata Version 4.11"", ERRATA #0183 20190515 GSV Sentence

NovAtel, "An Introduction to GNSS", 2nd ed., NovAtel Inc., 2015

Quectel, "L89 R2.0 GNSS Protocol Specification", L89_R2.0_GNSS_Protocol_Specification_V1.0, (Preliminary), 2020-04-29

Eric S. Raymond, "NMEA Revealed", 2023-05-03, https://gpsd.gitlab.io/gpsd/NMEA.html

RTCM 10403.3, "Differential GNSS (Global Navigation Satellite Systems) Services - Version 3", 141-2016-SC104-STD, 2016-10-07

RTCM 10410.1, "Networked Transport of RTCM via Internet Protocol (NTRIP) - Version 2.0", 111-2009-SC104-STD + 139-2011-SC104-STD, 2011-06-28

SAE 6857, "Requirements for a Terrestrial Based Positioning, Navigation, and Timing (PNT) System to Improve Nagivation Solutions and Ensure Critical Infrastructure Security", SAE6857, 2018-04

Thingstream, "PointPerfect getting started", Thingstream, 2022

Thingstream, "PointPerfect L-band Configuration", Thingstream, 2022

u-blox 7, "Receiver Description Including Protocol Specification V14", GPS.G7-SW-12001-B, ublox, 65525, 2013-02-01

u-blox 8, "Receiver Description Including Protocol Specification", v15-20.30.22-23.01, UBX-13003221-R15, ublox, 26415b7, 2018-03-06

u-blox 9, "ZED-F9P Interface Description", UBX-18010854-R05, ublox, 6cc4473, 2018-12-20

u-blox 9, "ZED-F9P Integration Manual", UBX-18010802-R03, ublox, 2018-12-20

u-blox 9, "ZED-F9T series Product Summary", UBX-18011626-R06, ublox, 2022

u-blox 9, "ZED-F9T Data Sheet", UBX-18053713-R07, ublox, 2022-01-04

u-blox 9, "u-blox ZED-F9T Interface Description", UBX-18053584-R02, ublox, 2019-06-13

u-blox 9, "ZED-F9T Integration Manual", UBX-19005590-R05, ublox, 2020-11-18

u-blox 9, "ZED-F9P-02B High precision GNSS module Professional grade Data sheet", UBX-21023276-R03, ublox, 2023-03-24

u-blox 9, "ZED-F9P u-blox F9 high precision GNSS module Integration Manual", UBX-18010802-R12, ublox, 2022-05-03

u-blox 9, "u-blox F9 HPG 1.32 u-blox F9 high precision GNSS receiver Interface Description", UBX-22008968-R01, ublox, 2022-05-02

u-blox 9, "Product summary ZED-F9P series u-blox F9 high precision GNSS modules", UBX-17005151-R14, ublox, 2023

u-blox 9, "NEO-D9S Correction data receiver Integration manual", UBX-19026111-R08, ublox, 2023-01-05

u-blox 9, "u-blox D9 PMP 1.04 u-blox D9 correction data receiver Interface description", UBX-21040023-R01, ublox, 2021-10-21

u-blox 9, "NEO-D9S and ZED-F9 configuration SPARTN L-band correction data reception Application Note", UBX-22008160-R02, ublox, 2022-07-27

u-blox 9, "Product summary NEO-D9S u-blox D9 correction data receiver", UBX-17010946-R05, ublox, 2021

u-blox 10, "NEO-F10T High precision timing GNSS module Professional Grade Data Sheet", UBX-22022576-R02, ublox, 2023-03-30

u-blox 10, "NEO-F10T u-blox F10 GNSS timing module Integration Manual", UBX-22018271-R01, ublox, 2023-04-05

u-blox 10, "Product Summary NEO-F10T module u-blox F10 high accuracy timing module", UBX-22025534-R02, ublox, 2023

u-blox 10, "u-blox F10 TIM 3.01 - u-blox F10 GNSS timing receiver - Interface Description", UBX-23003447-R01, ublox, 2023-03-21

u-blox, "u-center GNSS evaluation software for Windows User guide", UBX-13005250-R30, ublox, 2022-05-24

Tools

https://www.notams.faa.gov/dinsQueryWeb/

https://pilotweb.nas.faa.gov/PilotWeb/noticesAction.do?queryType=ALLGPS&formatType=ICAO

https://www.navcen.uscg.gov/?pageName=gpsAlmanacs

https://navcen.uscg.gov/?Do=constellationStatus

https://celestrak.com/GPS/NANU/description.php

https://www.gsc-europa.eu/system-status/Constellation-Information

https://www.ngs.noaa.gov/NGSDataExplorer/

https://www.ngs.noaa.gov/NCAT/

https://tagis.dep.wv.gov/convert/

https://www.gpsvisualizer.com/map_input?form=googleearth

https://www.gpsbabel.org

https://github.com/gpsbabel/gpsbabel

https://geodesy.noaa.gov/cgi-bin/HTDP/htdp.prl?f1=4&f2=1

http://www.earthpoint.us/Convert.aspx

https://www.calculator.net/distance-calculator.html

Resources

http://www.catb.org/gpsd/NMEA.txt DEPRECATED

http://www.catb.org/gpsd/NMEA.html DEPRECATED

https://gpsd.gitlab.io/gpsd/NMEA.html

https://support.google.com/earth/answer/148095

http://earth.google.com/intl/ar/userguide/v4/index.htm

http://static.googleusercontent.com/media/earth.google.com/en//userguide/v4/google_earth_user_guide.pdf

https://support.google.com/earth/answer/168344

https://dl.google.com/earth/client/GE7/release_7_1_8/googleearth-pro-7.1.8.3036.dmg

https://support.google.com/maps/answer/18539

https://fossies.org/linux/misc/gpsd-3.17.tar.gz/gpsd-3.17/test/daemon/beidou-gb.log

http://ktuukkan.github.io/marine-api/0.9.0/javadoc/net/sf/marineapi/nmea/sentence/TalkerId.html

https://github.com/mvglasow/satstat/wiki/NMEA-IDs

https://www.rapidtables.com/convert/number/degrees-to-degrees-minutes-seconds.html

https://in-the-sky.org/satmap_radar.php Note: when using the "Live Map of Satellite Positions" page, "Satellites above your horizon" tab, the displayed view is as if you are looking upwards into the sky. This means that the azimuth, which starts at 0 degrees at North, increases in the counter clockwise direction. This is probably obvious to the astronomers in the audience, but it wasn't to me.

https://www.gpsworld.com/the-almanac/

https://en.wikipedia.org/wiki/Global_Positioning_System

https://en.wikipedia.org/wiki/List_of_GPS_satellites

https://gssc.esa.int/navipedia/index.php/GPS_Navigation_Message

ftp://ftp.agi.com/pub/Catalog/Almanacs/SEM/GPSAlmanac.al3

https://www.rtca.org/sites/default/files/intentional_gps_interference_approved.pdf

https://github.com/mvglasow/satstat/wiki/NMEA-IDs

https://web.archive.org/web/20130225182002/http://www.colorado.edu/geography/gcraft/notes/gps/gpseow.htm

https://www.novatel.com/support/knowledge-and-learning/published-papers-and-documents/unit-conversions/

http://geostar-navi.com/files/docs/geos5/GeoS_NMEA_protocol_v4_0_eng.pdf

http://ozzmaker.com/wp-content/uploads/2016/08/M10478-M10578-NMEA_Sentence_Output.pdf

https://drotek.gitbook.io/rtk-f9p-positioning-solutions/tutorials/updating-zed-f9p-firmware

https://www.ardusimple.com/simplertk2b-hookup-guide/

https://www.ardusimple.com/simplertk2b-hack-1-unleash-the-usb-power-of-simplertk2b/

https://www.ardusimple.com/simplertk2b-hack-2/

https://www.ardusimple.com/simplertk2b-hack-3/

https://www.ardusimple.com/question/xbee-usage-documentation-for-simplertk/

https://www.ardusimple.com/question/problem-in-configuring-base-in-time-mode-sending-out-rtcm3-messages-from-base/

https://www.ardusimple.com/question/both-units-now-blink-gps-to-xbee-but-no-xbee-to-gps/

https://www.digi.com/resources/documentation/Digidocs/90001456-13/concepts/c_transparent_mode_detailed.htm?tocpath=XBee%20transparent%20mode%7CXBee%20transparent%20mode%20in%20detail%7C_____0

https://github.com/digidotcom/xbee_ansic_library

https://github.com/ukyg9e5r6k7gubiekd6/gpsd/blob/master/crc24q.c

https://youtu.be/T-g27KS0yiY Adam Hart-Davis, "The Clock That Changed the World", A History of the World, BBC, video

https://www.movable-type.co.uk/scripts/latlong.html

https://en.wikipedia.org/wiki/Haversine_formula

https://en.wikipedia.org/wiki/North_American_Plate

https://cs.nyu.edu/visual/home/proj/tiger/gisfaq.html

https://www.solidsignal.com

https://www.tigersupplies.com

https://www.surveying.com

https://www.sparkfun.com/products/15136

https://register.gotowebinar.com/recording/6016509329100006146

https://www.scientificamerican.com/article/gps-is-easy-to-hack-and-the-u-s-has-no-backup/ Paul Tullis, "GPS Is Easy To Hack, And The U.S. Has No Backup", Scientific American, 2019-12-01 (published in the 2019-12 print edition under the title "GPS Down")

https://www.cbsnews.com/news/global-positioning-system-preparing-the-next-generation-of-gps/ CBS Sunday Morning, "Preparing the next generation of GPS", 2019-12-01, video

http://www.aholme.co.uk/GPS/Main.htm

https://time.gov

https://aerospaceamerica.aiaa.org/features/cosmic-gps/ Adam Hadhazy, "Cosmic GPS", Aerospace America, 2020-05

https://www.gps.gov

https://en.wikipedia.org/wiki/Decimal_degrees

https://jsonlines.org

https://www.faa.gov/about/office_org/headquarters_offices/avs/offices/aam/cami/library/online_libraries/aerospace_medicine/tutorial/media/III.4.1.4_Describing_Orbits.pdf

https://www.gpsworldbuyersguide.com

https://gist.github.com/fnky/458719343aabd01cfb17a3a4f7296797

https://youtu.be/NvpbW7JRu0Q "Philomena Cunk's Moments of Wonder Ep. 1: Time", Charlie Brooker's Weekly Wipe, BBC, video

https://tf.nist.gov/general/pdf/1568.pdf Harold Lyons, "Atomic Clocks", Scientific American, 1957-02 (archived on a NIST website)

http://freenmea.net/docs

https://www.sparkfun.com/datasheets/GPS/NMEA%20Reference%20Manual-Rev2.1-Dec07.pdf

https://www.celestis.com/resources/faq/what-are-the-azimuth-and-elevation-of-a-satellite/

https://ciechanow.ski/gps/

https://learn.sparkfun.com/tutorials/how-to-upgrade-firmware-of-a-u-blox-gnss-receiver/all

https://www.wired.com/story/satellite-time-distribution/ W. Ralston, "When the UK's Timing Systems Fail, This Service Will Save Them", WIRED, 2022-08-26

https://stjarnhimlen.se/comp/time.html Paul Schlyter, "Time Scales"

https://www.gpsrchive.com/Shared/Satellites/GPS%20vs%20GLONASS%20vs%20Galileo.html

https://www.ietf.org/timezones/data/leap-seconds.list

https://geodesy.noaa.gov/CORS/resources/gpscals.shtml

http://leapsecond.com/java/gpsclock.htm

https://youtu.be/FUHkTs-Ipfg?si=BWIJwMdPdK-iGkKA "The SAT Question Everyone Got Wrong"

Soundtrack

https://www.youtube.com/playlist?list=PLd7Yo1333iA9FMHIG_VmuBCVTQUzB-BZF

Build

Diminuto

Clone and build Diminuto, which is used by gpstool although not by libhazer. (Or follow the directions in the Diminuto README.)

cd ~
mkdir -p src
cd src
git clone https://github.com/coverclock/com-diag-diminuto
cd com-diag-diminuto/Diminuto
make pristine
make depend
make all

Hazer

Clone and build Hazer. (If you don't build Diminuto where the Hazer Makefile expects it, some minor Makefile hacking might be required.)

cd ~
mkdir -p src
cd src
git clone https://github.com/coverclock/com-diag-hazer
cd com-diag-hazer/Hazer
make pristine
make depend
make all

Documentation

You can make HTML, man page, and PDF documentation based on the Doxygen comments embedded in the source code by using the following make targets.

make documentation
make manuals

You can find the resulting documentation in the following directories. The variable TARGET will be replaced by "host" unless you are using some special Makefile configuration (for example, for cross-compilation).

out/${TARGET}/doc/html
out/${TARGET}/doc/man
out/${TARGET}/doc/pdf

(Diminuto has a similer documentation build process.)

Administration

The fs directory contains a file system overlay of files that I've found useful to carefully install in system directories like /etc and /lib. It is not installed automatically, because whether they are helpful, or they reduce your target system to a smoking heap of silicon, depends on your exact circumstances: the target system, the version of Linux it runs, etc. Some of the files under fs are new files in their entirety, some just contain lines that need to be added to the indicated files.

Most Linux distros will require the program accessing the GPS/GNSS device to either have root privileges or be in the "dialout" group. The latter is of course preferred.

sudo usermod -a -G dialout ${USER}

${USER} will need to log out and back in for this to take effect.

It is also possible to edit the udev rules to make the device readable and writeable by the ${USER} (okay) or anyone (nope).

Depending on what features of Hazer you choose to use, you will need to define some services in the /etc/services file. The port numbers are your choices to make. Note that the TCP and UDP port numbers for the same service (e.g. tesoro) can be (and typically are) the same number. Port number values may range from 0 to 65353 and must be unique. Linux also assigns "ephemeral" or temporary port numbers which will not appear in the /etc/services file.

tumbleweed 22222/udp  # Tumbleweed RTK source/sink
tesoro     33333/tcp  # Tesoro JSON source
tesoro     33333/udp  # Tesoro JSON sink
hazer      44444/udp  # Hazer NMEA source/sink

Unit Tests

Set up environment and run tests and utilities. (This establishes the paths for both the Hazer and the Diminuto executables so you don't have to install the libraries and binaries in the system directories.)

cd ~/src/com-diag-hazer/Hazer
export LANG=en_US.UTF-8 # You can put this in your .profile.
. out/host/bin/setup
make unit-test

Functional Tests

There are a bunch of scripts that use actual hardware. Some of them require home-built test fixtures and General Purpose Input/Output (GPIO) pins. But there is a small collection of functional tests that do not require any GNSS device at all. Mostly I use these to make sure that gpstool does not throw an assert and core dump.

cd ~/src/com-diag-hazer/Hazer
export LANG=en_US.UTF-8 # You can put this in your .profile.
. out/host/bin/setup
make functional-test

Directories

Artifacts

Bash Sourced Files

Applications

Utilities

General Purpose

Data Analysis

Google Maps Moving Map (DEPRECATED)

Google Earth Keyhole Markup Language (KML)

Intertial Measurement Unit (Yodel)

Differential GNSS (Tumbleweed)

Safe Position Augmentation for Real-Time Navigation (Nicker)

OpenStreetMap Moving Map (Tesoro)

UDP over LTE-M (Wheatstone)

Precision Time and Frequency Reference (Metronome)

Output Control

WT901 IMU (Dally)

Functional Tests

Comma Separated Value (CSV) Output

The -T flag for gpstool will cause the utility to save the current Position, Velocity, Time (PVT) solution once a second to a "trace file" in CSV format as described below. This makes it easy to analyze results using tools like Excel and several tools provided by Hazer itself.

The PVT solution is taken from the high precision u-blox UBX-NAV-HPPOSLLH message if it is available, from the ensemble GNSS solution if it exists, or from one of the four Global Satellite Navigation Systems solutions in this order of preference: GPS, GLONASS, Galileo, BeiDou. The attitude (roll, pitch, yaw) is taken from the UBX-NAV-ATT message from the IMU if it is available.

Fields which are not available or are not supported by the receiver have values coded as "0." instead of an empty string to simplify parsing in post-processing.

I have enough code, both scripts and C code, that depends on the CSV format, not to mention archived CSV files, that it is a big deal to change. I've started encoding special characters at the end of the NAM hostname string instead of adding another field. Expect more of this.

A snippet of an actual CSV file looks like this.

"neon", 116, 3, 0, 12, 1598455524.895094741, 1598455524.000000000, 39.7943026, -105.1533253, 0., 1708.000, 1686.500, 0., 0.071000, 0., 0.00000, 80.98829, 0.00000, 20.00000, 43.29752, 167.44616, 0, 0.
"neon", 117, 3, 0, 12, 1598455525.895126606, 1598455525.000000000, 39.7943030, -105.1533263, 0., 1708.100, 1686.600, 0., 0.017000, 0., 0.00000, 80.98829, 0.00000, 20.00000, 43.29752, 167.44616, 0, 0.
"neon", 118, 3, 0, 12, 1598455526.895211419, 1598455526.000000000, 39.7943030, -105.1533276, 0., 1708.200, 1686.700, 0., 0.012000, 0., 0.00000, 80.98829, 0.00000, 20.00000, 43.29752, 167.44616, 0, 0.
"neon", 119, 3, 0, 12, 1598455527.895183560, 1598455527.000000000, 39.7943031, -105.1533286, 0., 1708.300, 1686.800, 0., 0.016000, 0., 0.00000, 80.98829, 0.00000, 20.00000, 43.29752, 167.44616, 0, 0.
"neon", 120, 3, 0, 12, 1598455528.901673983, 1598455528.000000000, 39.7943035, -105.1533300, 0., 1708.500, 1687.000, 0., 0.039000, 0., 0.00000, 80.98829, 0.00000, 20.00000, 43.29752, 167.44616, 0, 0.

csv2tty

Piping the CSV snippet into the script csv2tty produces readable output in fixed columns that looks like this.

GN 12 3D | 2020-08-26T15:25:24Z |  39°47'39"N, 105°09'11"W |  1708m |    0kn |   0° N   |   0°,  80°,   0°
GN 12 3D | 2020-08-26T15:25:25Z |  39°47'39"N, 105°09'11"W |  1708m |    0kn |   0° N   |   0°,  80°,   0°
GN 12 3D | 2020-08-26T15:25:26Z |  39°47'39"N, 105°09'11"W |  1708m |    0kn |   0° N   |   0°,  80°,   0°
GN 12 3D | 2020-08-26T15:25:27Z |  39°47'39"N, 105°09'11"W |  1708m |    0kn |   0° N   |   0°,  80°,   0°
GN 12 3D | 2020-08-26T15:25:28Z |  39°47'39"N, 105°09'11"W |  1708m |    0kn |   0° N   |   0°,  80°,   0°

These columns contain the following information.

The following command pipeline is useful.

tail -f out/host/tmp/example.csv | csv2tty

The peruse command supports this directly when used with a CSV file.

peruse example csv

csv2dat

Piping the CSV snippet into the script csv2dat produces a different readable output in fixed columns that looks like this.

GN 12 3D |    116 | 2020-08-26T09:25:24J | 2020-08-26T15:25:24Z |  39°47'39.489360"N, 105°09'11.971080"W |  1708m   5603ft |    0kn    0mph     0kph |   0° N   |   0°,  80°,   0°
GN 12 3D |    117 | 2020-08-26T09:25:25J | 2020-08-26T15:25:25Z |  39°47'39.490799"N, 105°09'11.974680"W |  1708m   5604ft |    0kn    0mph     0kph |   0° N   |   0°,  80°,   0°
GN 12 3D |    118 | 2020-08-26T09:25:26J | 2020-08-26T15:25:26Z |  39°47'39.490799"N, 105°09'11.979359"W |  1708m   5604ft |    0kn    0mph     0kph |   0° N   |   0°,  80°,   0°
GN 12 3D |    119 | 2020-08-26T09:25:27J | 2020-08-26T15:25:27Z |  39°47'39.491160"N, 105°09'11.982959"W |  1708m   5604ft |    0kn    0mph     0kph |   0° N   |   0°,  80°,   0°
GN 12 3D |    120 | 2020-08-26T09:25:28J | 2020-08-26T15:25:28Z |  39°47'39.492599"N, 105°09'11.987999"W |  1708m   5605ft |    0kn    0mph     0kph |   0° N   |   0°,  80°,   0°

These columns contain the following information.

I find this format more suitable for viewing the data during post-processing.

csv2dgm

Piping the CSV snippet into the script csv2dgm and providing an IP host address and port number like this

csv2dgm -j -U hostname:5555

sends a subset of the CSV line to an endpoint as a UDP datagram in JSON format. Receiving the datagrams using a command like this

socat -u UDP6-RECV:5555 -

produces output that looks like this.

{ "NAM": "neon", "NUM": 116, "TIM": 1598455524, "LAT": 39.7943026, "LON": -105.1533253, "MSL": 1708.000, "LBL": "2020-08-26T15:25:24Z" }
{ "NAM": "neon", "NUM": 117, "TIM": 1598455525, "LAT": 39.7943030, "LON": -105.1533263, "MSL": 1708.100, "LBL": "2020-08-26T15:25:25Z" }
{ "NAM": "neon", "NUM": 118, "TIM": 1598455526, "LAT": 39.7943030, "LON": -105.1533276, "MSL": 1708.200, "LBL": "2020-08-26T15:25:26Z" }
{ "NAM": "neon", "NUM": 119, "TIM": 1598455527, "LAT": 39.7943031, "LON": -105.1533286, "MSL": 1708.300, "LBL": "2020-08-26T15:25:27Z" }
{ "NAM": "neon", "NUM": 120, "TIM": 1598455528, "LAT": 39.7943035, "LON": -105.1533300, "MSL": 1708.500, "LBL": "2020-08-26T15:25:28Z" }

This output can be processed by the fun/channel script in another project, Tesoro, which uses this JSON and some JavaScript code to drive a moving map display using an OpenStreetMap tile server.

The same utility can be used to send the same subset of the CSV output to a serial port. I use this to transmit the CSV data via an Digi XBEE 3 LTE-M radio. If the XBEE is configured correctly, it can send the same datagram to the same endpoint via LTE-M, yielding the same result as above. I use LTE-M SIMs for AT&T's One Rate plan for this.

csv2dgm -j -D /dev/ttyUSB0 -b 9600 -8 -1 -n

csvfollow and csvplayback

The csvfollow script follows (tail -f) a CSV file being output in real-time by gpstool and uses csv2dgm to forward the corresponding JSON datagrams to (for example) a system that is using them to steer a moving map display.

csvfollow 192.168.1.253:tesoro out/host/tmp/vehicle.csv

The csvplayback script uses csvmeter to meter out an existing CSV file at approximately the same rate as it was originally generated by gpstool and uses csv2dgm to forward the corresponding JSON datagrams to (for example) a system that is using them to steer a moving map display.

csvplayback 192.168.1.253:tesoro dat/yodel/20200917/vehicle.csv

See the README in the Tesoro repository for more information.

csvdataset

The csvdataset script is a filter that reads a CSV file and produces a JSON dataset suitable for importation into the Tesoro choosedataset feature to create a static route map. Multiple routes can be rendered on the same map, the the properties of the route line (color, weight, etc.) can be changed either globally or on a per route basis.

csvdataset < dat/yodel/20200917/vehicle.csv > 20200917.json

The entire JSON dataset has to be processed and stored in memory by Tesoro. (This is a requirement of the underlying Leaflet third-party library.) For very large datasets this can be problematic. One way to handle this is to sample the CSV file. An optional value on the command line can cause csvdataset to sample the input file. The first and last data point is always included regardless of the sampling modulo value.

csvdataset 10 < dat/yodel/20200917/vehicle.csv > 20200917.json

See the README in the Tesoro repository for more information.

Help

gpstool

> gpstool -?
usage: gpstool
               [ -d ] [ -v ] [ -z ]
               [ -D DEVICE [ -b BPS ] [ -7 | -8 ] [ -e | -o | -n ] [ -1 | -2 ] [ -l | -m ] [ -h ] [ -s ] | -S FILE ] [ -B BYTES ]
               [ -R | -E | -H HEADLESS | -P ] [ -F SECONDS ] [ -i SECONDS ] [ -t SECONDS ] [ -a ]
               [ -C FILE ]
               [ -O FILE ]
               [ -L FILE ]
               [ -T FILE [ -f SECONDS ] ]
               [ -N FILE ]
               [ -Q FILE [ -q MASK ] ]
               [ -K [ -k MASK ] ]
               [ -A STRING ... ] [ -U STRING ... ] [ -W STRING ... ] [ -Z STRING ... ] [ -w SECONDS ] [ -x ]
               [ -4 | -6 ]
               [ -G :PORT | -G HOST:PORT [ -g MASK ] ]
               [ -Y :PORT | -Y HOST:PORT [ -y SECONDS ] ]
               [ -I CHIP:LINE | -I NAME | -c ]
               [ -p CHIP:LINE | -p NAME ]
               [ -M ] [ -X MASK ] [ -V ]
       -1              Use one stop bit for DEVICE.
       -2              Use two stop bits for DEVICE.
       -4              Prefer IPv4 for HOST.
       -6              Prefer IPv6 for HOST.
       -7              Use seven data bits for DEVICE.
       -8              Use eight data bits for DEVICE.
       -A STRING       Collapse STRING, append Ubx end matter, write to DEVICE, expect ACK/NAK.
       -A ''           Exit when this empty STRING is processed.
       -B BYTES        Set the input Buffer size to BYTES bytes.
       -C FILE         Catenate input to FILE or named pipe.
       -D DEVICE       Use DEVICE for input or output.
       -E              Like -R but use ANSI Escape sequences.
       -F SECONDS      Update report no more than every SECONDS seconds, 0 always, <0 never.
       -G HOST:PORT    Use remote HOST and PORT as dataGram sink.
       -G :PORT        Use local PORT as dataGram source.
       -H HEADLESS     Like -R but writes each iteration to HEADLESS file.
       -I CHIP:LINE    Take 1PPS from GPIO CHIP LINE (requires -D) (LINE<0 active low).
       -I NAME         Take 1PPS from GPIO NAME (requires -D) (-NAME active low).
       -K              Write input to DEVICE sinK from datagram source.
       -L FILE         Write pretty-printed input to Listing FILE.
       -M              Run in the background as a daeMon.
       -N FILE         Use fix FILE to save ARP LLH for subsequeNt fixed mode.
       -O FILE         Save process identifier in FILE.
       -P              Process incoming data even if no report is being generated.
       -Q FILE         Write validated input to FILE or named pipe.
       -R              Print a Report on standard output.
       -S FILE         Use source FILE or named pipe for input.
       -T FILE         Save the PVT CSV Trace to FILE.
       -U STRING       Collapse STRING, append Ubx end matter, write to DEVICE.
       -U ''           Exit when this empty STRING is processed.
       -V              Log Version in the form of release, vintage, and revision.
       -W STRING       Collapse STRING, append NMEA end matter, Write to DEVICE.
       -W ''           Exit when this empty STRING is processed.
       -X MASK         Enable special test modes via MASK.
       -Y HOST:PORT    Use remote HOST and PORT as keepalive sink and surveYor source.
       -Y :PORT        Use local PORT as surveYor source.
       -Z STRING       Collapse STRING, write to DEVICE.
       -Z ''           Exit when this empty STRING is processed.
       -a              Display Active satellite views first.
       -b BPS          Use BPS bits per second for DEVICE.
       -c              Take 1PPS from DCD (requires -D and implies -m).
       -d              Display Debug output on standard error.
       -e              Use Even parity for DEVICE.
       -f SECONDS      Set trace Frequency to 1/SECONDS.
       -g MASK         Set dataGram sink mask (NMEA=1, UBX=2, RTCM=4, CPO=8, default=15).
       -h              Use RTS/CTS Hardware flow control for DEVICE.
       -i SECONDS      Bypass input check every SECONDS seconds, 0 always, <0 never.
       -k MASK         Set device sinK mask (NMEA=1, UBX=2, RTCM=4, CPO=8, default=15).
       -l              Use Local control for DEVICE.
       -m              Use Modem control for DEVICE.
       -n              Use No parity for DEVICE.
       -o              Use Odd parity for DEVICE.
       -p CHIP:LINE    Assert GPIO outPut CHIP LINE with 1PPS (requires -D and -I or -c) (LINE<0 active low).
       -p NAME         Assert GPIO outPut NAME with 1PPS (requires -D and -I or -c) (-NAME active low).
       -q MASK         Set Queue mask (NMEA=1, UBX=2, RTCM=4, CPO=8, default=15).
       -s              Use XON/XOFF (c-Q/c-S) Software flow control for DEVICE.
       -t SECONDS      Timeout GNSS data after SECONDS seconds [0..255].
       -u CCM          Use CCM for convergence threshold in centicentimeters.
       -v              Display Verbose output on standard error.
       -w SECONDS      Write STRING to DEVICE no more than every SECONDS seconds, 0 always, <0 never.
       -x              EXit if a NAK is received.
       -y SECONDS      Send surveYor a keep alive every SECONDS seconds, 0 always, <0 never.
       -z              Exit if all state machines stop.

rtktool

> rtktool -?
usage: rtktool [ -d ] [ -v ] [ -M ] [ -V ] [ -M ] [ -p :PORT ] [ -t SECONDS ]
       -M          Run in the background as a daeMon.
       -V          Log Version in the form of release, vintage, and revision.
       -d          Display Debug output on standard error.
       -p :PORT    Use PORT as the RTCM source and sink port.
       -t SECONDS  Set the client timeout to SECONDS seconds.
       -v          Display Verbose output on standard error.

csv2dgm

> csv2dgm -?
usage: csv2dgm [ -d ] [ -v ] [ -c | -h | -j | | -q | -s | -x | -y ] [ -t ] [ -D DEVICE [ -b BPS ] [ -7 | -8 ] [ -1 | -2 ] [ -e | -o | -n ] [ -m ] [ -r ] ] [ -F FILE ] [ -M MODE ] [ -U HOST:PORT ]
       -1              Set DEVICE to 1 stop bit.
       -2              Set DEVICE to 2 stop bits.
       -7              Set DEVICE to 7 data bits.
       -8              Set DEVICE to 8 data bits.
       -D DEVICE       Write datagram to DEVICE.
       -F FILE         Save latest datagram in observation FILE.
       -M MODE         Set FILE mode to MODE.
       -U HOST:PORT    Forward datagrams to HOST:PORT.
       -b BPS          Set DEVICE to BPS bits per second.
       -c              Emit CSV.
       -d              Enable debug output.
       -e              Set DEVICE to even parity.
       -h              Emit HTML.
       -j              Emit JSON.
       -o              Set DEVICE to odd parity.
       -m              Set DEVICE to use modem control.
       -n              Set DEVICE to no parity.
       -q              Emit URL Query.
       -r              Set DEVICE to use hardware flow control.
       -s              Emit Shell commands.
       -t              Write to standard output.
       -v              Enable verbose output.
       -x              Emit XML.
       -y              Emit YAML.

Memory Leaks

When testing for memory leaks, I strongly recommend using valgrind, specfically with the full leak check option.

valgrind --leak-check=full gpstool ...

I was surprised to find it called out a twelve byte memory leak in the C library's locale handling, which gpstool requires to correctly display Unicode symbols like the degree symbol. Perhaps this will be fixed in a future GNU update (or it's a bug in my code that I can't see).

For More Information

Acknowledgements

Thanks to Charles F. F. Karney for his MIT licensed geographiclib.

Thank you to Kathy Kadnuck of the Valley Water District, who, thanks to her collection of infrastructure survey maps, helped me to determine the location of an old NGS survey marker in my neighborhood which, apparently, had been buried under a sidewalk as part of a development in the past few decades. (Remarkably, it is not illegal to landscape over an NGS survey marker. It's merely illegal to remove or damage them. If a municipality needs to find the marker for official business, be prepared to have your landscaping back-hoe'ed up. And be billed for it.)

A big thank you to Brad Gabbard, a professional surveyor with Flatirons, Inc., a surveying, engineering, and geomatics firm located in Boulder Colorado. Mr. Gabbard generously shared some results off his professional Trimble GPS rover taken at NGS survey marker KK1446 that I have used to test this code.

Thanks to several manufacturers of GNSS devices, or of products that have GNSS devices embedded within them, who, because of design flaws and bugs in their products, led to vast improvements in this code. No, I'm not kidding: almost every time I test a new GNSS-based product I find the opportunity for improvement.

Special thanks to Mrs. Overclock for her assistance in road testing (literally) this software and for her understanding regarding having GPS equipment littered all around the house both inside and outside.

Footnotes