Redesign NTP refresh based on uptime

arendst commented 4 years ago

With the ever growing amount of Tasmota IoT devices the ntp.org sites get flooded with NTP requests around NTP update time. Currently this is around 2 minutes after the hour skewed by some MAC address seconds.

I'll change the update time and make it dependant on uptime and MAC address second.

Also future NTP servers will need to be changed to 0.tasmota.pool.ntp.org once released.

jziolkowski commented 4 years ago

Hmm if I could obtain a GPS time source, I could add my server to the pool. At 1gbit symmetric fiber available, I wouldn't really notice the NTP traffic

jziolkowski commented 4 years ago

Apparently it's not required to have such source as long as my node is synced to the rest of the pool. I'll see about configuring my device to join the pool.

arendst commented 4 years ago

No it's not an issue of providng a NTP server. It's mere that they want to be able to manage NTP over their different servers.

They provide vendor services (https://www.pool.ntp.org/vendors.html) explaining the situation. I already applied for a tasmota vendor service. Just waiting for my acceptance.

jziolkowski commented 4 years ago

Yes, but that page mentions that as a vendor you can encourage users to join the pool. The generic pool, or tasmota pool?

arendst commented 4 years ago

The upcoming tasmota pool

jziolkowski commented 4 years ago

And that is the plan :)

arendst commented 4 years ago

Yes. I'll change the current default ntp pool servers in my_user_config.h to the new ones once accepted. It will be anounced in the README or wiki or Doc and over time users will switch to one of \<number>.tasmota.pool.ntp.org

jziolkowski commented 4 years ago

https://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html want a Stratum-1 at home for cheap? :)

arendst commented 4 years ago

Some routers also provide NTP services. Even cheaper.

Th0maz commented 4 years ago

Some routers also provide NTP services. Even cheaper.

That's what I chose as best practice for my local network . The router (AVM Fritzbox) gets the time from an internet ntp server and serves as an ntp server itself. All my tasmota and network devices are configured to contact the local router for ntp service.

jziolkowski commented 4 years ago

Yes, I have Mikrotiks here, will explore that option as well.

arendst commented 4 years ago

Reconsidering the issue I think we do not have that big issue here. If you ping the default tasmota pool.ntp.org you will likely get another server than me:

root@domus1:~# ping pool.ntp.org
PING pool.ntp.org (83.98.155.30) 56(84) bytes of data.
64 bytes from ntp1.trans-ix.nl (83.98.155.30): icmp_seq=1 ttl=54 time=22.6 ms
64 bytes from ntp1.trans-ix.nl (83.98.155.30): icmp_seq=2 ttl=54 time=24.4 ms

jziolkowski commented 4 years ago

Well yes, that's the idea behind the pools after all.

Jason2866 commented 4 years ago

Yes, you will get normally a server from your country.

meingraham commented 4 years ago

Is there still an issue with NTP being served from the router serving up DHCP?

arendst commented 4 years ago

In Tasmota DHCP will not provide the NTP server.

The user will have to configure the correct (local) NTP server. I'm not aware of issues where NTP would fail on DHCP providing routers.

meingraham commented 4 years ago

Was referring to this - https://github.com/arendst/Tasmota/issues/5283#issuecomment-466888846

Does the newer SDK address this?

arendst commented 4 years ago

Good question. I haven't tested it.

mweinelt commented 4 years ago

Just tested it against 6.5.0.10 and it does.

Packet capture

14:11:50.851084 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 336)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 5c:cf:7f:9f:41:8f, length 308, xid 0x2fe46609, Flags [none] (0x0000)
      Client-Ethernet-Address 5c:cf:7f:9f:41:8f
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Discover
        MSZ Option 57, length 2: 1500
        Parameter-Request Option 55, length 5: 
          Subnet-Mask, Default-Gateway, BR, Domain-Name-Server
          NTP
        END Option 255, length 0
        PAD Option 0, length 0, occurs 53
14:11:51.853473 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    192.168.120.254.67 > 192.168.120.4.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x2fe46609, Flags [none] (0x0000)
      Your-IP 192.168.120.4
      Client-Ethernet-Address 5c:cf:7f:9f:41:8f
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Offer
        Server-ID Option 54, length 4: 192.168.120.254
        Lease-Time Option 51, length 4: 1440
        Subnet-Mask Option 1, length 4: 255.255.255.0
        Default-Gateway Option 3, length 4: 192.168.120.254
        Domain-Name-Server Option 6, length 4: 192.168.120.254
        NTP Option 42, length 4: 192.168.120.254
        END Option 255, length 0
        PAD Option 0, length 0, occurs 20
14:11:51.863038 IP (tos 0x0, ttl 255, id 1, offset 0, flags [none], proto UDP (17), length 336)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 5c:cf:7f:9f:41:8f, length 308, xid 0x2fe46609, Flags [none] (0x0000)
      Client-Ethernet-Address 5c:cf:7f:9f:41:8f
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: Request
        MSZ Option 57, length 2: 1500
        Requested-IP Option 50, length 4: 192.168.120.4
        Server-ID Option 54, length 4: 192.168.120.254
        Parameter-Request Option 55, length 5: 
          Subnet-Mask, Default-Gateway, BR, Domain-Name-Server
          NTP
        Hostname Option 12, length 18: "tasmota-1073682605"
        END Option 255, length 0
        PAD Option 0, length 0, occurs 21
14:11:51.870232 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    192.168.120.254.67 > 192.168.120.4.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x2fe46609, Flags [none] (0x0000)
      Your-IP 192.168.120.4
      Client-Ethernet-Address 5c:cf:7f:9f:41:8f
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message Option 53, length 1: ACK
        Server-ID Option 54, length 4: 192.168.120.254
        Lease-Time Option 51, length 4: 1440
        Subnet-Mask Option 1, length 4: 255.255.255.0
        Default-Gateway Option 3, length 4: 192.168.120.254
        Domain-Name-Server Option 6, length 4: 192.168.120.254
        NTP Option 42, length 4: 192.168.120.254
        END Option 255, length 0
        PAD Option 0, length 0, occurs 20
14:11:51.878131 IP (tos 0x0, ttl 255, id 2, offset 0, flags [none], proto UDP (17), length 76)
    192.168.120.4.52582 > 192.168.120.254.123: [udp sum ok] NTPv4, length 48
    Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), precision 0
    Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec)
      Reference Timestamp:  0.000000000
      Originator Timestamp: 0.000000000
      Receive Timestamp:    0.000000000
      Transmit Timestamp:   0.000000000
        Originator - Receive Timestamp:  0.000000000
        Originator - Transmit Timestamp: 0.000000000
14:11:51.878742 IP (tos 0x0, ttl 64, id 60080, offset 0, flags [DF], proto UDP (17), length 76)
    192.168.120.254.123 > 192.168.120.4.52582: [bad udp cksum 0x729d -> 0x2f0a!] NTPv4, length 48
    Server, Leap indicator:  (0), Stratum 4 (secondary reference), poll 0 (1s), precision -23
    Root Delay: 0.022155, Root dispersion: 0.000640, Reference-ID: 162.159.200.1
      Reference Timestamp:  3783849109.592920426 (2019/11/27 14:11:49)
      Originator Timestamp: 0.000000000
      Receive Timestamp:    3783849111.877376588 (2019/11/27 14:11:51)
      Transmit Timestamp:   3783849111.877864642 (2019/11/27 14:11:51)
        Originator - Receive Timestamp:  3783849111.877376588 (2019/11/27 14:11:51)
        Originator - Transmit Timestamp: 3783849111.877864642 (2019/11/27 14:11:51)

nagyrobi commented 4 years ago

Would be nice to implement option 42 in the DHCP client: http://www.networksorcery.com/enp/protocol/bootp/option042.htm This would simply allow us to automatically set the NTP server in the clients using the DHCP protocol.

In our case, we use pfSense as DHCP servers, and we hand out NTP server addresses very easily with it. But Microtik and others also offer this possibility.

There should be a way to check if that succeeded on Tasmota, say "ntpserver" command should return the NTP serrver address received from DHCP. Currently, it reports pool.ntp.org. I have to set it manually.

mweinelt commented 4 years ago

Would be nice to implement option 42 in the DHCP client:

This is already the case, see https://github.com/arendst/Tasmota/issues/6995#issuecomment-559082156.

nagyrobi commented 4 years ago

Really? Well it never worked for me. I'm using Tasmota since around version 5.1 and my DHCP server is always handing over the NTP server address in option 42, it never worked. That's why I thought it was never implemented.

PapaSpik3 commented 4 years ago

Really? Well it never worked for me. I'm using Tasmota since around version 5.1 and my DHCP server is always handing over the NTP server address in option 42, it never worked. That's why I thought it was never implemented.

For me neither... #6304 But would be nice to have this feature.

meingraham commented 4 years ago

You have to be running Tasmota 6.5.0.10 or greater.

arendst commented 4 years ago

haven't heard back from the ntp.pool.org so I'll close this issue as

I changed the NTP polling based on clock time to uptime allowing random NTP requests leveraging the hourly peak load of public NTP servers.
Users are aware of using their local NTP servers also leveraging public NTP servers.

Thx for all your input

arendst / Tasmota

Redesign NTP refresh based on uptime #6995