Idea : Improve accuracy aiming sub millisecond

[jcdevaux]

Hi Rop ! First, I'd like to thank you for that nice library.

Millis() and micros()

As far as I can see in sources, you use millis() to compute a timestamp offset from last NTP sync to produce an NTP corrected accurate date aiming ms accuracy (ezTime.cpp:103, 384, ...).

First, why not using micros() instead of millis() as even on smallest Arduino boards @16 MHz (@8 MHz) we got 4 µs (8µs) resolution ?

Local crystal drift compensation

Second, when you use your millis() values, you logically divide by 1000 (which we can call CONSTANT_MS_TO_S).

Couldn't we compensate crystal drift altering that CONSTANT_MS_TO_S value using two (or more) distant NTP sync ? Something like (in µs) : (micros_sync2-micros_sync1)/((ntp2-ntp1)*1000000) or even with millis(). Kind of temperature compensated crystal but using NTP.

Do you think this will allow to acheive sub millisecond accuracy time sync in local networks ?

Looking forward to hearing from you !

[alwynallan]

I have modified ezTime with microsecond resolution, and linear interpolation of a set (32) of recent NTP sync events. It also gradually increases the sync interval until it's only checking every 8 hours. The result is accuracy better than 1ms and compensation for crystal drift. It's only tested on ESP32. If there's interest I could clean it up and document it and submit it for merging here. This is an example of the performance (I have a Stratum 0 NTP server on my LAN, but its good on pool.ntp.org too).

[ropg]

Yay, nice!

Yes please.

[jcdevaux]

Hi there ! That's nice !

Your linear regression of NTP offset describes a constant drift of you ESP32. That's why you can increase delay between NTP sync. This is not true when you have some temperature variations for example if your crystal is not temperature compensated.

[ropg]

Naturally: keeping a constant eye on it is much better...

[jcdevaux]

How did you mesure your overall accuracy @alwynallan ?

[alwynallan]

Thanks for encouragement. I'll have it cleaned up and submitted in a week.

Regarding accuracy, I've been using this approach in an ESP8266 application for years. There are statistical techniques to put confidence bands on the locus of the regression line, and reasonable to assume that the accuracy is the half-width of the band at the current time. In my current application the accuracy is limited by millisecond resolution of the time, but usually settles around 2ms as seen below. With microsecond resolution I'm sure it'll be better that 1ms. Below is a typical chart from my ESP8266 application.

Regarding quartz crystals they are very strange when studied closely over a long period of time. Identical parts can have positive and negative temperature coefficients, and they can have different natural modes that they stay in for weeks, and then suddenly change, with slightly different frequencies. The also change frequency slightly as they age. The consumer-grade parts are usually "binned" so they'll be within spec (say +-20ppm) but seldom in the middle of the range. I have some that slowly change frequency in a repeating sinusoidal pattern, but the period is 2.5 days, while the temperature variation has a 1-day period. Regression does a good job smoothing all this out and improving on both the crystals and NTP.

[jcdevaux]

An idea about the source of the 2.5 days period ? I agree that temperature is around 1-day period if it's due to ambiant temperature. Is your MCU core load constant ? What about supply voltage fluctuation ?

For our readers, here 2 interesting links about crystal drift for Arduino and ESP32 :

• Arduino clock drift analysis
• ESP32 NTP clock drift correction

[alwynallan]

I have the library in good shape with the new features, but it needs testing. It should not break existing applications that use it. I have only tested it on ESP32. I went to great lengths to do all the regular calculations with integers for speed and wide compatibility, but the regressionGraphData() function does use floats. I'm quite pleased with the algorithm I came up with for rejecting NTP syncs that are outliers. I want to add a regression confidence intervals in the regressionGraphData(), but I'm out of time and off to ski for a week.

Here's the code and documentation: https://github.com/alwynallan/ezTime

The changes are in the src/ezTime.cpp, src/ezTime.h, images/regression-graph.png (new), and documentation in README.md.

There are 4 new public functions: micros64(), queryNTPu(), regressionDebugData(), and regressionGraphData().

All the existing functions use the time obtained from extending the regression line to the current value of micros64(), but it's transparent to the user.

Let me know what you think. With a good crystal (consistent not accurate), relatively constant temperatures, a local NTP server (many routers have one) accuracy < 1ms is achievable (RMSE aka stdev of residuals), and with the default options 20ms accuracy is possible with only 3 NTP syncs per day, vs 48 for the current version.

On Fri, Mar 18, 2022 at 6:06 AM jcdevaux @.***> wrote:

An idea about the source of the 2.5 days period ? I agree that temperature is around 1-day period if it's due to ambiant temperature. Is your MCU core load constant ? What about supply voltage fluctuation ?

For our readers, here 2 interesting links about crystal drift for Arduino and ESP32 :

• Arduino clock drift analysis http://jorisvr.nl/article/arduino-frequency
• ESP32 NTP clock drift correction https://climbers.net/sbc/esp32-accurate-clock-sleep-ntp/

— Reply to this email directly, view it on GitHub https://github.com/ropg/ezTime/issues/144#issuecomment-1072259431, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNKNTDSPBAZOC6S2WXVQSDVARIRXANCNFSM5OVH57OQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[jcdevaux]

Hey I will give a try with an arduino nano and wiznet ethernet module. Have a nice week !

Le jeu. 24 mars 2022 à 02:06, Peter Allan @.***> a écrit :

I have the library in good shape with the new features, but it needs testing. It should not break existing applications that use it. I have only tested it on ESP32. I went to great lengths to do all the regular calculations with integers for speed and wide compatibility, but the regressionGraphData() function does use floats. I'm quite pleased with the algorithm I came up with for rejecting NTP syncs that are outliers. I want to add a regression confidence intervals in the regressionGraphData(), but I'm out of time and off to ski for a week.

Here's the code and documentation: https://github.com/alwynallan/ezTime

The changes are in the src/ezTime.cpp, src/ezTime.h, images/regression-graph.png (new), and documentation in README.md.

There are 4 new public functions: micros64(), queryNTPu(), regressionDebugData(), and regressionGraphData().

All the existing functions use the time obtained from extending the regression line to the current value of micros64(), but it's transparent to the user.

Let me know what you think. With a good crystal (consistent not accurate), relatively constant temperatures, a local NTP server (many routers have one) accuracy < 1ms is achievable (RMSE aka stdev of residuals), and with the default options 20ms accuracy is possible with only 3 NTP syncs per day, vs 48 for the current version.

On Fri, Mar 18, 2022 at 6:06 AM jcdevaux @.***> wrote:

An idea about the source of the 2.5 days period ? I agree that temperature is around 1-day period if it's due to ambiant temperature. Is your MCU core load constant ? What about supply voltage fluctuation ?

For our readers, here 2 interesting links about crystal drift for Arduino and ESP32 :

• Arduino clock drift analysis http://jorisvr.nl/article/arduino-frequency
• ESP32 NTP clock drift correction https://climbers.net/sbc/esp32-accurate-clock-sleep-ntp/

— Reply to this email directly, view it on GitHub https://github.com/ropg/ezTime/issues/144#issuecomment-1072259431, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACNKNTDSPBAZOC6S2WXVQSDVARIRXANCNFSM5OVH57OQ

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/ropg/ezTime/issues/144#issuecomment-1076966918, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA52LSOTMLBMAWJSTCAEEEDVBO5YVANCNFSM5OVH57OQ . You are receiving this because you authored the thread.Message ID: @.***>