Setup Trouble on Pi Zero 2 W with MAX-M8Q GNSS HAT

[AlexLandherr]

Hi,

I tried setting up this solution for my NTP server running on a Pi Zero 2 W and using the MAX-M8Q GNSS HAT. I am following this guide: NEO-M8T_GNSS_TIMING_HAT, NTP_Server

The setup shell script runs fine but when I run: watch -n1 chronyc sourcestats -v

I get this:

What could I be doing wrong?

[beta-tester]

which ntpdate packet did you installed - just the depricated ntpdate package or the newer ntpsec-ntpdate ? both ntpdate versions have similar options. -t for timeout, -u direct mode for better chance to come through firewalls. -b to force step the clock instead of slew. sudo ntpdate -buv -t 5 192.168.1.12

[beta-tester]

EDIT: Nevermind, I got the sudo ntpdate ip_address_of_server working now. As long as the Pi Zero 2 W that is acting as the server is within 1 microsecond of UTC I'm very happy considering the price of the components.

now you can make an accurate happy new year count down clock ... or an automated firework rocket launcher... :clock12: :fireworks: :tada: :smile:

[beta-tester]

if you install the package gpsd-client to the other computers, you also can use and visualize the GPS coorninates e.g. xgps 192.168.1.12 for desktop or cgps 192.168.1.12 for console (where the ip of the RPi zero with the GPS must be)

[AlexLandherr]

Could I use a Python Library to "listen" on the GPIO pin used for PPS while it's disciplining the Pi's time and use the PPS to trigger some event like the tick of a counter?

My idea is detailed in this forum post of mine: Using PPS Signal from GNSS Receiver to Trigger Camera

[ComputerSmiths]

If the kernel has the pin, I'd doubt you can read it from Python at the same time. You could probably jumper it over to another GPIO with no pullups or pulldowns and read that, though the kernel interrupt is going to take priority.

And I don't think you are going to get the kind of precision you think you are, while the PPS is within nanoseconds of realtime, the capture routines take several seconds, the OS is going to have higher-priority stuff running, and I'd have to guess the jitter in "absolute time within ps of realtime that a particular pixel is captured" is going to be on the order of ms, so you are probably better off with a timer interrupt.

And what's the actual requirement? To capture a pixel within picoseconds of when you think you did, or to capture a frame every minute?

Willie

On Dec 29, 2021, at 6:38 AM, Alex Landherr @.***> wrote:

Could I use a Python Library to "listen" on the GPIO pin used for PPS while it's disciplining the Pi's time and use the PPS to trigger some event like the tick of a counter?

My idea is detailed in this forum post of mine: Using PPS Signal from GNSS Receiver to Trigger Camera

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[AlexLandherr]

The intent as described in the post is to have the image interval be as precise as possible.

Alternatively could I have it so that the GNSS receiver sets the time as usual on the Pi4B but I use my regular code to capture the images and rely on the system clock being within +/- 1 millisecond of true time? If so the image captures would be within acceptable limits for me.

Or is there a possibility that I could use 2 GNSS receivers of which one sets the time for the Pi4B whilst the other one triggers the camera? The second receiver would use another GPIO pin on the Pi4B for PPS and have Python "listen" on that pin which is not used for anything else.

[ComputerSmiths]

On Dec 29, 2021, at 7:43 AM, Alex Landherr @.***> wrote:

The intent as described in the post is to have the image interval be as precise as possible. Alternatively could I have it so that the GNSS receiver sets the time as usual on the Pi4B but I use my regular code to capture the images and rely on the system clock being within +/- 1 millisecond of true time?

Right, but is the "as precise as possible" picoseconds or milliseconds? What's the precision requirement for the actual application?

If I run:

time /usr/bin/raspistill --awb sun --quality 25 --width 1640 --height 1232 -o /root/foo.jpg

I get:

real 0m5.717s user 0m0.016s sys 0m0.066s

[And it varies by well over 100 ms on subsequent runs]

I see that the capture takes almost 6 seconds, where the image is actually taken in that period is anyone's guess.

I've got a script that runs from crontab every minute to capture a still frame (as above), time-stamp it, and save it for making time-lapse movies, and the seconds in the time-stamp bounces back and forth between 6 and 7 seconds after the minute. Which is fine for this application.

See http://dotnetdotcom.net/GeekHo/TimeLapse/SunRisePi/Timelapse_2021-12-23.mp4 for an example.

If you are making a speed-trap camera, then you obviously care more about the temporal and spatial resolution of the system (the front of the car was on this pixel at this time, and at this other pixel at this other time), but you aren't clear on why you need "as precise as possible", what that means, and what the application is.

Willie

[AlexLandherr]

My application is simply put a time lapse camera where the start of the capture sequence is precise to +/- 1 millisecond.

Suppose I want to capture a sequence of 3840x2160 PNG images every 60 seconds starting on 12:00:00 UTC and ending on 13:00:00 UTC.

That would translate to the first image being captured (i.e. the capture sequence/all steps to record the data for a single image) at 12:00:00 UTC and the final image being captured at 12:59:00 UTC for a total of 60 images.

I already have a working program that does this without any active disciplining of the system clock. The critical part of the code is a while loop that as fast as the CPU will allow checks the system time and if it is equal to the time of the next image an image is captured, all within a window of time given by the user as say:

Start time (in UTC): 2021-12-01 00:00:00

Stop time (in UTC): 2021-12-02 00:00:00

Interval (in seconds, as an integer): 60

Using this info the total number of images is calculated and used to check if it's time to exit the while loop.

My thought was that by either having a GNSS receiver constantly disciplining the Pi4B whilst the program was running or having 2 receivers where one sets the time and one triggers the capture would aid in having the interval and time stamp of the images be as accurate as +/- 1 millisecond.

My understanding of how an image is captured tells me that only the start of the capture sequence needs to be known well. As long as the image interval is longer than the steps the electronics goes through to capture and save the image data I don't mind that this image took x fractions of second to capture and save.

As for your "bouncing" time stamp I would suggest reducing the image resolution in pixels, I ran into this problem early on when I first got a Pi4B and using the picamera library. My time stamps were fluctuating like yours by about 1 second.

My hypothesis is that the buffer for the data gets saturated and needs more time to store the data but since the program wants to save new data before all of the previous data has been written to storage the interval necessarily goes up.

I set a minimum allowable interval of 11 seconds which really means that an integer value of 10 or lower will not be used and the program asks the user to enter another value.

So far I've written 3840x2160 PNG images of about 10 Mebibytes each at intervals of 20 seconds and not gotten the issue visible in your film.

My solution however causes the system time to drift somewhat over longer time lapse captures. That's why I need constant disciplining of the system time so I can trust the time stamps on the images.

[AlexLandherr]

Hi again,

How do I create a discussion thread on your repository so that this thread doesn't unnecessarily expand and I can ask you again in the future?

[beta-tester]

there is no "private message" available at github. so best is to keep asking on this thread. do not open new issues as long there is no new serious issue with my script.

[AlexLandherr]

Is there a way via Python to get the UTC offset for the system time like when I run that chronyc (or corresponding command)?

I plan to use a GNSS HAT and a SparkFun 20x4 Character LCD to make a UTC clock that along with the date and time displays the offset something like this: 2022-01-14 12:42:58 UTC Offset: -0.000000032 sec

[beta-tester]

i would try to see what i get:

• run chronyc tracking inside python and parse the output for the required values. (gives the time offset of the local system).
• enable logging in chronyd and watch to the offset values in the /var/log/chrony/tracking.log (gives the time offset of the local system).
• run ppstest /dev/pps0 and parse the assert value the first part should be the seconds since 1970-01-01 00:00:00 and the value of interest is the ony behind the dot (only able on the system where GPS is attached).
• run gpspipe -w <gpsd-server> and parse its PPS output clock_nsec. same as before but you can run it from any computer where you have a gpsd-client installed. (but it gives you the time offset of the time server where GPS is attached itself, not from the local system).

something like this:

import subprocess
...
with subprocess.Popen("sudo chronyc tracking", shell=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) as proc:
  out = proc.stdout
  error_code = proc.poll()
  if not error_code:
    str = out.readline().decode()
    index = str.find('System time     : ')
    ...

maybe run it in a loop in a separate thread.

[AlexLandherr]

New problem: For some reason the PPS LED on the HAT I'm using has turned off or isn't blinking anymore. The physical connections all seem okay and the antenna has a clear view of the sky.

When I run the chronyc sourcestats (or similar) to see the offset from the different time sources on PPS0 it gives me an offset of ~+365 ms.

Could it be that I'm not getting a PPS signal through?

[beta-tester]

yes, that's possible. to me it happens also sometimes, specially when there are less satellites in view or most of them are lower than around 30° above horizont or there are interferences/noise from somewhere else.

the ~365ms are comming from the NMEA output over the serial port of your GPS. if not already done, you can adjust the offset of the GPS0 to under +/- 200ms (see note2 in the readme)

you can also set a higher baud rate for the serial port (e.g. 115200), that sometimes improves the accuracy of the NMEA time codes.

and take a look to the manual of your GPS module if there is a setting where you can choose the mode/behavior when the PPS signal is provided. 3D FIX or if it is already good when there is 1D FIX - i can't remember how the option is called exactly.

i think where you live you will have not so many satellites straight above you. maybe you can enable the triple mode GPS, GLONASS, Galileo - but then you definitely has to speed up your baud rate of your serial port, itherwise there are coming more date then the baud rate allowes.

but as i remember you told, that you removed the buffer battery from your GPS, so in case you change the PPS behavior, this and all other changed setting goes lost as soon your GPS looses its power.

if you have internet, add some stratum one time server close to your localtion to chrony.

to see if you computer gets PPS from the GPS, run sudo ppstest /dev/pps0

to see the sky view of your GPS, run xgps you should have at least 4 satellites in green

[AlexLandherr]

Which command do I use to add a stratum 1 server to chrony? I know of a few good ones in here in Sweden.

[beta-tester]

take a look to the file /etc/chrony/stratum1/20-ntp-servers.conf there are some stratum one servers for germany and belgium (commented out) take this as template for yours and remove the # in front to get used. and comment out the poor pool ntp servers.

[AlexLandherr]

i would try to see what i get:

• run chronyc tracking inside python and parse the output for the required values. (gives the time offset of the local system).

• enable logging in chronyd and watch to the offset values in the /var/log/chrony/tracking.log (gives the time offset of the local system).

• run ppstest /dev/pps0 and parse the assert value the first part should be the seconds since 1970-01-01 00:00:00 and the value of interest is the ony behind the dot (only able on the system where GPS is attached).

• run gpspipe -w <gpsd-server> and parse its PPS output clock_nsec. same as before but you can run it from any computer where you have a gpsd-client installed. (but it gives you the time offset of the time server where GPS is attached itself, not from the local system).

something like this:

import subprocess

...

with subprocess.Popen("sudo chronyc tracking", shell=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) as proc:

  out = proc.stdout

  error_code = proc.poll()

  if not error_code:

    str = out.readline().decode()

    index = str.find('System time     : ')

    ...

maybe run it in a loop in a separate thread.

How to I run it in a separate thread in a loop in Python? I haven't done that before.

[beta-tester]

please read documentation: https://docs.python.org/3/library/threading.html

maybe something like this.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#

import threading
import queue

stop_event = threading.Event()
data_queue = queue.Queue()

def data_collector_loop():
  try:
    while not stop_event.is_set():
      data_item = None

      # TODO: collect a data_item
      ...

      # put data_item to the data_queue to get handled by the other thread
      #   with timeout, to see if stop_event was fired
      if data_item:
        try:
          data_queue.put(data_item, timeout=2)
        except queue.Full as ex:
          print(f"Oops, lost data_item {data_item}, {ex}")
          continue

      # sleep 5s without missing stop_event
      stop_event.wait(timeout=5)
  except KeyboardInterrupt:
    pass
  except SystemExit:
    pass
  finally:
    stop_event.set()

def data_handler_loop():
  try:
    while not stop_event.is_set():
      data_item = None

      # wait for new data_items in data_queue
      #   with timeout, to see if stop_event was fired
      try:
        data_item = data_queue.get(timeout=2)
      except queue.Empty:
        print(f"Nothing to do.")
        continue

      # TODO: do something with the data_item
      ...
  except KeyboardInterrupt:
    pass
  except SystemExit:
    pass
  finally:
    stop_event.set()

def main(args):
  try:
    thread1 = threading.Thread(target=data_collector_loop, daemon=False)
    thread2 = threading.Thread(target=data_handler_loop, daemon=False)

    thread1.start()
    thread2.start()

    # TODO: do something else
    ...

    # block main thread until thread1 and thread2 are done
    thread1.join()
    thread2.join()
  except KeyboardInterrupt:
    pass
  except SystemExit:
    pass
  finally:
    stop_event.set()
  return 0

if __name__ == '__main__':
  import sys
  sys.exit(main(sys.argv))

[AlexLandherr]

New question: Wouldn't Last offset be better than System time?

Or is System time with the seconds fast/slow of NTP time the same as Last offset, i.e. how many (fractions of a) second ahead/behind UTC with respect to what the GPS receiver is telling the Pi?

Would it be possible to just run it as a function and call the function inside the loop where I get the system time? This is my code so far:

from __future__ import print_function
import qwiic_serlcd
from time import sleep
from datetime import datetime, timezone
import subprocess

def get_offset():
    with subprocess.Popen("sudo chronyc tracking", shell=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) as proc:
        out = proc.stdout
        error_code = proc.poll()
        if not error_code:
            str = out.readline().decode()
            out = str[str.find("Last offset     : "):(str.find("Last offset     : ") + 12)] #Slices out the proper value from the str variable.
        return out

try:
    #Setup code for 20x4 LCD.
    lcd = qwiic_serlcd.QwiicSerlcd()

    lcd.setBacklight(255, 255, 255)
    lcd.setContrast(5)
    lcd.begin()
    lcd.disableSystemMessages()
    lcd.clearScreen()
    sleep(1)
    lcd.print("UTC Time:")
    lcd.setCursor(0, 2)
    lcd.print("Offset from UTC:")

    while True:
        lcd.setCursor(0, 1)
        lcd.print(datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S"))
        lcd.setCursor(0, 3)
        lcd.print(get_offset())
        sleep(0.1)

except KeyboardInterrupt:
    print("\nExited 'UTC_Clock.py'.")
    lcd.clearScreen()

My primary problem now is that I get a OSError: [Errno 121] Remote I/O error. According to SparkFun this is because the cable to the LCD isn't properly connected. This seems unlikely as I have checked several times now: Troubleshooting Guide for LCD

[beta-tester]

i gave only an example. you can take what ever you want and need. i can't tell you what's best for you. if you don't need to run other stuff in parallel, then you can do everything without threads.

[AlexLandherr]

Ok, though I'm having trouble slicing out the relevant part of the chronyc tracking.

Regarding the offset; is the System time output from chronyc tracking the preferred value if my intent is to obtain the offset from true UTC time?

[beta-tester]

Last offset from system time or UTC - where is the different. if chrony tells the clock is 100ns off from the time source and chrony keeps the clock in sync, then the clock is 100ns off from system time and UTC in the same way... isn't it.

but please read the documentation of chrony and gpsd if you need the exact definition you are looking for.

[AlexLandherr]

Last offset from system time or UTC - where is the different. if chrony tells the clock is 100ns off from the time source and chrony keeps the clock in sync, then the clock is 100ns off from system time and UTC in the same way... isn't it.

but please read the documentation of chrony and gpsd if you need the exact definition you are looking for.

I checked the chrony documentation and it seems the "Last offset" section is the proper value. So if my understanding of your answer and the documentation is correct the preferred value is Last offset then?

If I seem slow to understand slight differences I apologize for not making it clear enough.

[AlexLandherr]

I now have a (mostly) working program:

from __future__ import print_function
import qwiic_serlcd
from time import sleep
from datetime import datetime, timezone
import subprocess
import re

def get_offset():
    p = subprocess.Popen("sudo chronyc tracking | grep -i  Last ", stdout=subprocess.PIPE, shell=True)
    (output, err) = p.communicate()
    status = p.wait()
    offset_str = re.findall(r"[-+]?\d*\.\d+|\d+", str(output))[0]
    return offset_str

try:
    #Setup code for 20x4 LCD.
    lcd = qwiic_serlcd.QwiicSerlcd()

    lcd.setBacklight(255, 255, 255)
    lcd.setContrast(5)
    lcd.begin()
    lcd.disableSystemMessages()
    lcd.clearScreen()
    sleep(1)
    lcd.print("UTC Time:")
    lcd.setCursor(0, 2)
    lcd.print("Offset from UTC:")

    while True:
        lcd.setCursor(0, 1)
        lcd.print(datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S"))
        lcd.setCursor(0, 3)
        lcd.print(get_offset() + " seconds")
        sleep(0.2)

except BaseException:
    print("\nExited 'UTC_Clock.py'.")

finally:
    lcd.clearScreen()

My question is whether there is a faster way to access the Last offset value from Python than issuing a shell command and waiting for the output?

[beta-tester]

not that i know

[AlexLandherr]

Ok, thanks.

With the method used in my above posted code could I obtain the offset value of PPS0 from watch -n1 chronyc sourcestats -v?

And if not do you have any tips for how it could be done?

[beta-tester]

for other suggestions, please see my older comment: https://github.com/beta-tester/RPi-GPS-PPS-StratumOne/issues/10#issuecomment-1016727180

i would try to see what i get:

• run ppstest /dev/pps0 and parse the assert value the first part should be the seconds since 1970-01-01 00:00:00 and the value of interest is the ony behind the dot (only able on the system where GPS is attached).
• run gpspipe -w <gpsd-server> and parse its PPS output clock_nsec. same as before but you can run it from any computer where you have a gpsd-client installed. (but it gives you the time offset of the time server where GPS is attached itself, not from the local system).

another way could be the statistics of PPS0 (/var/log/chrony/...), when logging and statistics of chrony are enabled (it should if /etc/chrony/stratum1/30-logging.conf is untouched).

[AlexLandherr]

I may come of as slow to understand but how do I get this to work?:

run gpspipe -w and parse its PPS output clock_nsec. same as before but you can run it from any computer where you have a gpsd-client installed. (but it gives you the time offset of the time server where GPS is attached itself, not from the local system).

I've tried these commands but get these errors: pi@raspberrypi:~/Python_Code $ gpspipe -w clock_nsec gpspipe: could not connect to gpsd clock_nsec:2947, can't get host entry(-2) pi@raspberrypi:~/Python_Code $ sudo gpspipe -w clock_nsec gpspipe: could not connect to gpsd clock_nsec:2947, can't get host entry(-2) pi@raspberrypi:~/Python_Code $ sudo gpspipe -w <gpsd-server> -bash: syntax error near unexpected token 'newline'

I wan't to print the clock_nsec value to an LCD and if it as I understand it represents the system time offset in nanoseconds from UTC it would be the most desirable option. I don't need the offset displayed on another computer on the same network, just on the same Pi that is running the server and is attached to the GPS.

Based on the errors what am I doing wrong here?

[beta-tester]

please read the document ! https://gpsd.gitlab.io/gpsd/gpspipe.html

just execute gpspipe -w from the command line on the computer where gpsd is running and see what you get. in case it is on another computer, then you have to give the name of the computer or ip like gpspipe -w my_gpsd_server or gpspipe -w 192.168.1.111

[AlexLandherr]

please read the document !

https://gpsd.gitlab.io/gpsd/gpspipe.html

just execute gpspipe -w from the command line on the computer where gpsd is running and see what you get.

in case it is on another computer, then you have to give the name of the computer or ip like gpspipe -w my_gpsd_server or gpspipe -w 192.168.1.111

Thanks! Sorry for the somewhat stupid question.