[Feature Request] SNTP Support

[brandond]

Now that I start looking at using Lambda code for more things, I'm realizing it might be nice to have the local RTC set properly.

It appears that both ESP32 and ESP8266 have support for SNTP via the configTime() function:

• https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/esp32-hal-time.c#L46
• https://github.com/esp8266/Arduino/blob/master/cores/esp8266/time.c#L58

As far as exposing this through esphomeyaml, it seems like there are only three things that can be configured:

• GMT Offset (time zone)
• DST offset (if DST is to be observed)
• Server list (1-3 servers)

It would probably be reasonable to have GMT and DST offset default to zero, which effectively puts the RTC at UTC. Similarly, it could default to three pool servers (n.pool.ntp.org)

[OttoWinter]

Oh... I thought NTP was super hard to get working on the ESP... Apparently I was wrong 😬

About the esphomelib side:

• It would need to be a separate component and opt-in. I'm just guessing here but I think these methods have pretty big dependencies (for code-size).
• Another question is how these should be exposed in lambdas. Should we just have a get_time() method on the component that returns the current time. Also, in what format? Is tm_info easy enough to use or should we convert it to our own struct with better attribute names...
• So DST offset is applied to all values - even when it's not daylight savings time? I mean in that case the attribute is a bit useless (but probably also necessary for some users)

For the esphomeyaml side:

• It would be great if we could use the system's timezone as the default.
• Maybe (if we want to get advanced), esphomeyaml could also tell esphomelib in which days of the year to apply the DST offset. Probably quite hard to do though and could be done at a later point.

[brandond]

I'd be glad to take a shot at this. A few additional notes:

• LWIP has an option to configure NTP from DHCP, but this appears to be disabled in the Arduino core.
• When the RTC is set, time() returns an epoch timestamp instead of just time since boot. You can also do getLocalTime() if you want a time struct. Would you want to wrap these somehow, or just let folks call them as a standard Arduino lib?
• esp32 has configTzTime, which takes a TZ string instead of fixed DST and UTC offsets, which seems much more reasonable than simply applying DST corrections at all times. I might see if the code in that function also works on ESP8266.

[OttoWinter]

Great! So about your responses:

• I mean it would be good if we could infer the NTP directly from DHCP, but IMO DHCP exists in so many variations that this would ultimately end up in users asking why it doesn't work.
  • -> Setting the timezone and NTP servers is a one time thing, so I don't think it's a problem if users need to specify these things themselves.
  • Time since boot is I think pretty useless for most use cases... So we don't have to support that. Besides, users can just call millis() to get the current time since boot - rollovers shouldn't be that big of a problem since it only happens once every 50 days...
  • configTzTime - sounds great! Only thing I would test with that is how much the code size increases when that is included in the binary... If the ESP stores a table of all timezones and the daylight savings data that's going to be a lot of added code size...
  • Also, I have no problem with keeping the daylight savings time feature to the ESP32 only for now. Many parts of esphomelib are kind of made with the "goal" of making people switch to the ESP32, it's just so much better in many respects.
  • Also, I think we should just define that this is all using the gregorian calendar.
  • Ideally, a configuration would look something like this (of course nothing final, just some thoughts):

# Other names 'time', 'real_time_clock', ...
local_time:
  # Required, on ESP32 would also apply DST
  timezone: Europe/Vienna
  # Let's give it an ID so that it has a familiar interface for lambda users
  id: my_local_time

  # Optional: 
  # Could also be a ntp_server_1, ntp_server_2, ...  ¯\_(ツ)_/¯, probably doesn't really matter
  ntp_server:
    - 1.pool.ntp.org
    - 2.pool.ntp.org
    - 3.pool.ntp.org

some_component:
  - platform: some_platform
    on_something:
      then:
        - lambda: >-
            # get_time() should return a struct that has the individual time components as
            # .year, .month, ...
            auto time = id(my_local_time).get_time();
            # Use plural for all these names?
            ESP_LOGD("main", "Year is %d", time.year);
            ESP_LOGD("main", "Month is %d", time.month); // 1-12
            ESP_LOGD("main", "Day is %d", time.day); // 1-31
            ESP_LOGD("main", "Hour is %d", time.hour); // 0-23 - no AM/PM bs :P
            ESP_LOGD("main", "Minute is %d", time.minute); // 0-59
            ESP_LOGD("main", "Second is %d", time.second); // 0-59
            ESP_LOGD("main", "Millisecond is %d", time.millisecond); // 0-999

Some of that could maybe be possible to implement... so above format is nothing near final.

[brandond]

It looks like it's using the glibc TZ stuff behind the scenes, and poorly. Check out the goofy munging it's doing here: https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/esp32-hal-time.c#L18

Since all it's really doing is enabling LWIP's SNTP stuff and then setting a time zone, that bit of logic would be easy to replace with a properly formatted POSIX TZ string for the specified zone name, as resolved on the local host at build time.

I'm not sure that this increases the code size any since the SNTP stuff is built into LWIP and the rest of the time logic is built into newlib. It is in fact perfectly valid to configure the TZ and call time functions without setting up NTP, it's just that the ESPs do not have a battery-backed RTC so time resets to unix epoch when it reboots.

At some point someone's probably going to come along and ask for external RTC support, so it might be forward-thinking to handle the RTC as a modular platform, with the built-in RTC as the default and only currently supported option.

So, maybe something like this:

rtc:
  platform: (hardware|pcf8523|ds3231|etc)
  timezone: UTC
  id: local_clock
ntp:
  rtc: local_clock
  servers:
    - 0.pool.ntp.org
    - 1.pool.ntp.org
    - 2.pool.ntp.org

[brandond]

So, I did some searching around and it does not appear that there is any well-maintained mapping of tzinfo zone names to POSIX TZ strings. Boost kinda comes close, but even they use a mapping table that's woefully out of date: https://github.com/boostorg/date_time/blob/master/data/

Probably the best thing to do would be to document the limitation, and simply pass through whatever the user specifies. If they want to figure out a proper TZ string for the current rules in their region, they're welcome to do so.

[brandond]

Early results; initial PR incoming.

[08:32:22][D][debug:136]: Reset Reason: Power on
[08:32:22][D][debug:137]: Reset Info: flag: 0
[08:32:22][C][sntp:024]: Setting up SNTP...
[08:32:22][I][application:071]: Running through first loop()
[08:32:22][I][application:082]: First loop finished successfully!
[08:32:22][C][main:045]: RTC time is 0 (year=70 month=0 day=1)
[08:32:22][C][main:046]: Formatted time is 1970-01-01T00:00:00Z
[08:33:00][C][main:045]: RTC time is 1528561980 (year=118 month=5 day=9)
[08:33:00][C][main:046]: Formatted time is 2018-06-09T16:33:00Z

[OttoWinter]

OMG YES!!! 😺 Absolutely awesome! Also about the tzinfo to TZ mapping: If we provide a link to a table in the docs where people can look the TZ string up themselves that would already be fine.

[brandond]

Sorry for leaving this unfinished for so long - day job getting busy plus summer family vacation has reduced the time I've had to work on this over the last few months.