time() does not work without active NTP server

[av1024]

Basic Infos

Hardware

Hardware: ESP-12E Core Version: 2.1.0-rc2

Description

"Internal" time.c functions always return 0 /1970-01-01/ if not synced via NTP.

I want to initialize internal clock via constant or from RTC for log timestamping (so I don't need precise time on startup, but valid measured intervals)

Settings in IDE

Module: Generic ESP8266 Module Flash Size: 4MB CPU Frequency: 80Mhz Flash Mode: qio Flash Frequency: 80Mhz Upload Using: SERIAL Reset Method: ck

Sketch

#include <Arduino.h>
#include <time.h>

void setup() {
 // WiFi.begin(); // wifi disabled for example
 // configTime(3*3600, 0, "not.avail.ntp"); 
 Serial.begin(115200);
}

void loop() {
  time_t now = time(nullptr);
 Serial.println(ctime(&now));
 delay(1000);
}

Debug Messages

Thu Jan 01 00:00:00 1970

Thu Jan 01 00:00:00 1970

...

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[d-a-v]

already in progress, commit is coming :)

[d-a-v]

Complete commit log:

moved time functions from lwip2 to core

• sntp_set_system_time() is not anymore used by lwip2 nor callable, settimeofday() is the entry point
• ntp timestamps or settimeofday() now reset time functions, and give (sec,usec) instead of (sec)
• update esp8266/NTP-TZ-DST example accordingly
• new core files: coredecls.h and sntp-lwip2.c
• update lwip2 commit ref and lib accordingly

further work has to be done in settimeofday() to properly take tv_usec into account

I will let you fix tv->tv_usec in settimeofday(), I don't feel I'd be good at it :)

[d-a-v]

master's core has been updated

• including active time without NTP
• with settimeofday() receiving but not honouring tv_usec (sntp or user calling)
• without adjtime()

@igrr thanks !

[5chufti]

Hi, I think a little more info is needed:

what is the result of "time();"? a) an internal timer based value or b) a real ntp-server answer? if a) how often/when is this internal timer synced with a ntp-server (if configured and active)? can this be configured/forced (how)?

bg: I only need a longterm accuracy of appr. timestamp resolution (s), and I don't want to flood the ntp-server, so internal timer would suffice for a certain periode but eventual resync is necessary.

thanks for your patience

[d-a-v]

Without NTP, it starts from 1st january 1970 8am (it's EPOCH in Espressif's HQ). It can be forced with settimeofday().

Time is re-set / adjusted by sntp driver (edit: via NTP)

• every time the ESP is receiving a new DHCP update (which happens regularly) containing a NTP server address (depends on your server)
• every hour (3600000us == SNTP_UPDATE_DELAY) by default

edit2: you can see time functions running with a new sketch example esp8266/NTP-TZ-DST

[5chufti]

thank you, I'm working my way through the "time&TZ" functions of newlib at the moment; looks good so far. If I'm using static IP and disable WiFi for energy efficiency there is no way to force "sync" during a WiFi enabled periode?

I did see the example but missed something more enduser oriented like the following on how to get a "real" time with "onboard" features.

#include <ESP8266WiFi.h>
#include <sys/time.h>                   // struct timeval
#include <time.h>                       // time() ctime()

#define ssid "yourSSID"
#define wpwd "yourPASSPHRASE"

#define RTC_TEST 1510592825 // = Monday 13 November 2017 17:07:05 UTC
// #define RTC_TEST 1499893466 // = Wed 12 July 2017 21:04:26 UTC

void setup() {
  Serial.begin(74880);
  WiFi.persistent(false);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, wpwd);

  timeval tv = { RTC_TEST, 0 };
  timezone tz = { 0 , 0 };
  settimeofday(&tv, &tz);
  configTime(0, 0, "pool.ntp.org");
  setenv("TZ", "CET-1CEST,M3.5.0,M10.5.0/3", 3);
  tzset();
  // don't wait, observe time changing when ntp timestamp is received
}

void loop() {
  time_t tnow = time(nullptr);
  Serial.println(ctime(&tnow));

  if (millis() > 30000 and millis() < 31000) {
    WiFi.disconnect();
    WiFi.mode(WIFI_OFF);
    Serial.println("WiFi off");
  }
  delay(1000);
}

[d-a-v]

Do you want to avoid using an external RTC ? Do you want to get a new NTP time stamp, and put Wifi off once received until next time you decide to resynchronize time ? In you sketch above, use configTime to set TZ, DST and default NTP server.

sntp_request(NULL) is what is internally achieved every hour. It can be called by the sketch. Once NTP time stamp is received, settimeofday() is called by SNTP, then you can put Wifi off again.

There is curently no way to know when settimeofday() is called. It can be discussed later if it is needed, in the meantime you can wait a few seconds before going to sleep again and report if it fits your needs.

[5chufti]

If I set the offset in configTime to 0,0 I know what I get: UTC. If I set anything else the only thing I know is: its wrong 50% of the year (you cant get correct TZ rules from two plain offsets) So I prefer to get plain UTC and have newlib take care of the TZ thing ...

And if it takes "several seconds" as I observe at the moment it is around 20!, this is not energy efficient. I speculate it is due to the 15s default of SNTP_UPDATE_DELAY that also applies to the first sync (why the "update" in the name?). It should be zero for start and some default after first successful sync. ... so no game for me, back to "manually" fetching sntp_timestamps.

[d-a-v]

SNTP_UPDATE_DELAY is the internal interval between two NTP requests in the sntp driver in a world where network is always on and available. In your case, if you do your request by hand with sntp_request(NULL) once the network becomes available, you should get your answer right away. Did you try that ?

[5chufti]

unfortunately: "sntp_request" was not declared in this scope even if I declare it: undefined reference to `sntp_request'

[everslick]

extern "C" ?

[5chufti]

what else ...

[d-a-v]

@5chufti I have updated lwip2 to expose sntp_request(NULL) which is static with a new function sntp_force_request(). I have patched arduino core to add a callback when settimeofday() is called. I have updated the NTP_TZ_DST.ino example that makes use of them. Please try and report back. You need to pull my branch goodies from scratch.

I did so because I believe in this use case :)

[5chufti]

thank you very much, but it is not about me and my projects. I prefer a solution that will work in every upcoming release without special adaption; be it in the core or my code. I understand this platform as part of IoT and therefor energy efficiency is relevant, especially as in several fora I encounter requests for battery / solar powered gadgets.

[5chufti]

thank you for your effort, your implementation and example seem to work. BUT: a) with all the headers to pull in and undocumented functions it is not very "arduinish" to use.

b) I think it should be emphasized in the example that there is no automatic DST switchover in core functions. So maybe you would like to add few lines to setup() to show how easy real TZ setup can be achieved?

#else // ntp

  configTime(0, 0, "pool.ntp.org");
  settimeofday(0, nullptr);
  setenv("TZ", "CET-1CEST,M3.5.0,M10.5.0/3", 3);
  tzset();
  WiFi.mode(WIFI_STA);
  WiFi.begin(SSID, SSIDPWD);
  // don't wait, observe time changing when ntp timestamp is received

#endif // ntp

I'm not belitteling your effort but I think the general arduino user is not concerned about monotonic system timestamps, they more likely are searching for local realtime. IMHO People concerned about monotonic µs timestamps should be using sdk ... have system allways return UTC (be ntp compatible, all un*x systems I know have hw/system clock on UTC) and forget about tz and dst-offset in core functions, they are more confusing than helpfull.

[d-a-v]

It would be nice to have such an official example sketch with energy saving and NTP updates when coming up alive to keep sensor logs accurate.

Instead of adding your not-arduinish-too few lines, why wouldn't you write something like this function and propose it as a PR along with a sketch using it (a new one or the current esp8266/NTP-TZ-DST)

#define TZ_CHINA_MAINLAND "xxxyyy"
#define TZ_GERMANY "CET-1CEST,M3.5.0,M10.5.0/3"
#define TZ_...
...

void configTime (const char* TZ, const char* server1, const char* server2, const char* server3)
{
    setenv("TZ", TZ, ...);
    tzset()
    ...
}

@igrr @devyte what would you think of such an overload of configTime() ?

[5chufti]

Hi d-a-v, sorry if I just seem to be wasting your time. I am not asking anybody to do something on my behalf, after 30+years in SW development I find my way around (except where problems hide in binary blobs - as is often the case with ESP). I am just around to remind some of the devs that arduino is about "k.i.s.s." and I couldn't see it here. Having system ignorant about TZ and DST (UTC only) is fine, so no overload is necessary, just remove the confusing remains also and give a decent example of how - even w/o additional libs - local time is achievable via ntp or rtc - easy as that. If thinking about overload, I would plead for an initial timestamp!

[devyte]

Alright, I've been keeping out of this discussion on purpose, mostly because I'm not using this time functionality, and because I currently don't have the details loaded into my brain of what the "standard" or "posixy" way of doing things are.

What I am using is TimeLib, from which I derived my own slightly modified version, and it has been working without issues for months now. Here are my thoughts from a top-level point of view, which completely disregard the current state of this implementation. To be specific, that means that I haven't looked at this implementation (apologies), and this is how I would approach implementing the whole thing:

1. I would break this functionality into 3 parts: time keeping, time providing, time adjusting.

   • Time keeping means just maintain time on the ESP, as in adjust a seconds counter or something every so-and-so ticks.
   • Time providing means where does the time get set from after bootup. There are three possibilities: manual setting, NTP, or external RTC. In addition, for the last two cases at least, time should be synced every so often from the provider. I would suggest a callback method here, similar to what TimeLib does, simply because it is so easy. You pass as argument a function that when called somehow returns the current time, whether it comes from an NTP request or from an external RTC comm.
   • Time adjusting means compensate for time zone and DST. Time keeping should be in UTC, but when requesting the time from the system, these adjustments should be taken into account. Something most implementations forget is that there are non-integer timezones. Each of these is a separate concept, and should be more or less separate of the other, which means function calls that are specific to each, and don't mix arguments. So, about the configTime overload, I would prefer to not mix a TZ setting with NTP server names.

2. There are several implementations of time and zones and so on out there, one of which is TimeLib. We probably don't want to reinvent the wheel. That means that, when in doubt, look at what the other libs are doing. TimeLib does have at least one flaw: the timezone is an integer.

3. DST switches over on specific dates, and these dates are different for different countries. Not only that, but some countries change the official dates every so often (example in point: my own country, just ask any IT here how much they love the government). This seriously complicates trying to implement automatic changeover of DST to give correct time. As a result, most embedded devices don't implement this.

4. Implementing the time zone strings with #defines of const char strings will increase mem usage. I would suggest not showing that as an example. Or to anyone. The strings should always be in flash. I consider the whole string and offset thing to be complex enough that I would implement a table of some kind that can be queried (what is the time offset for this string? what is the string for this time offset? what is the time offset for this index? etc). Personally, I would put the whole thing in a file on SPIFFS like a nano database. Foremost, I am thinking of the memory reduction project, so I would like to reduce keeping any strings in memory where at all possible. If a time string is needed, then build it on the fly when requested, and return it, but don't keep any pieces in mem.

Do you want me to spend some time looking into the implementation, so that I can provide more detailed feedback? Maybe we can look into incorporating some of the TimeLib stuff here, making that lib superflous.

[5chufti]

As we are in the "core", it should only provide 1a & 1b. As you say, 1c - 4 are "next layer" but - fortunately - allready provided within newlib, so even more "arduinish" --> user don't have to find the "most compatible" TZlib, the core provides full standards compatible functions; so why hide it?

[devyte]

@5chufti our core can provide 1a and 1b in favor of arduino compatibility, but we can also provide 1c as our own lib, similar to our other ESP8266 libs. Please don't confuse arduino compatibility with arduino style. We are striving for compatibility, so that apps based on arduino libs can be ported to the ESP easily. We are not striving for arduino style. In general terms, there are so many things wrong with the arduino usage design, that I sometimes seriously think about taking a trip to their HQ just to yell at their devs and slap them in the face. Robin: "hey I just implemented this new global singleton..." Batman: "NO!" Slap!*

[devyte]

@d-a-v I'll try to take a look at the current time/ntp code, contrast against the arduino docs and against what I'm using, and provide feedback at that point. I'll try to come up with a way to get the best of all worlds. Please don't spend more time on this until then. In the meantime, I could use your help elsewhere, to move towards 2.4 final!

[d-a-v]

I will not enter into details. What I proposed was quite simple and, I believe, coherent.

• we already have configTime(tz, dst, ntpsservers)
• we also have newlib and its powerful tzset taking into acount tz, dst
• we can overload configTime as follow (in a separate file.cpp so not everything is linked if not used)

  void configTime (const char* tz, const char* ntp_server)
  {
  setenv("TZ", tz, 1/*overwrite*/);
  tzset();
  configTime(0, 0, ntp_server);
  }

  with some defines which are not stored into ram/flash like the TZ_*country above, only one would be used in sketches (maybe they are already defined somewhere, I did not check). Maybe they would have to change from time to time because of specific politics per country, but they remain only defines. They can be living in their separate .h file, PRs can change them if needed, they are just harmless helpers.

We would then have one arduinish simple line per sketch that would configure in the most simple way accurate local time including DST for those who would like it, with no harm for others.

That's all what I suggested :) (I was not aware of newlib and these powerful tzset settings before this thread, I'm glad we have it)

[d-a-v]

I need to know, for low powered sketches wanting to keep accurate NTP time, if the sntp_force_request() proposed above, along with a settimeofday() callback (example) can be useful to go back to low-power mode as soon as correct time is set and logs stored.

[5chufti]

Hi, thanks again for your interest. As you will see from following data, these functions are very helpfull for energy efficiency. The tests were performed with following sketch and either LwIP V2 or precompiled_gcc (1.4?)

#define VER 1                    // old=0 new=1
#define PER 1                    // persistent n=0 y=1

#include <ESP8266WiFi.h>
#include <time.h>
#if VER
#include <coredecls.h>                  // settimeofday_cb() / sntp_force_request()
#endif
// to get sntp
extern "C"
{
#include "sntp.h"
}

#define SSID "open"
#define WPWD ""
IPAddress ip(192, 168, 22, 5);
IPAddress gw(192, 168, 22, 123);
IPAddress nm(255, 255, 255, 0);
IPAddress mc(255, 255, 255, 255);

time_t t = 0;

void time_is_set (void)
{
  time(&t);
  Serial.println("got time (@" + String(millis()) + "ms) " + ctime(&t));
  ESP.deepSleep(15000000);
  delay(10);
}

void setup() {
  Serial.begin(74880);
  Serial.println("setup(): " + String(millis()));
#if VER
  settimeofday_cb(time_is_set);
  configTime(0, 0, "at.pool.ntp.org");
#else
  sntp_stop();
  sntp_setservername(0, "at.pool.ntp.org");
  sntp_setservername(1, "de.pool.ntp.org");
  sntp_set_timezone(0);
  sntp_init();
#endif
  WiFi.persistent(PER);
  WiFi.mode(WIFI_STA);
  WiFi.config(ip, gw, nm, gw, gw);
  WiFi.begin(SSID, WPWD);
  //  WiFi.begin();
  while (WiFi.status() != WL_CONNECTED) delay(1);
  Serial.println("WiFi connected: " + String(millis()));
#if !PER
  ESP.eraseConfig();
#endif
#if VER
  sntp_force_request();
}
#else
  while (t == 0) {
    t = sntp_get_current_timestamp();
    delay(1);
  }
  Serial.println("got time (@" + String(millis()) + "ms) " + ctime(&t));
  Serial.println(ctime(&t));
  ESP.deepSleep(15000000);
  delay(10);
}
#endif

void loop() {
  yield();
}

The results show that with LwIP V2 time until first valid timestamp was considerably longer than with old version.

#################### LwIP2, persisted WiFi conn. #############

setup(): 197
WiFi connected: 343
got time: 6414

setup(): 206
WiFi connected: 348
got time: 6441

setup(): 203
WiFi connected: 357
got time: 6449

setup(): 194
WiFi connected: 339
got time: 6365

#################### LwIP2, w/o persisted WiFi conn. ##############

setup(): 303
WiFi connected: 3344
got time: 7518

setup(): 307
WiFi connected: 3349
got time: 7369

setup(): 302
WiFi connected: 3351
got time: 7465

setup(): 305
WiFi connected: 3349
got time: 7491

#################### LwIP, persisted WiFi conn. ###############

setup(): 191
WiFi connected: 338
got time: 1123

setup(): 209
WiFi connected: 351
got time: 1055

setup(): 197
WiFi connected: 341
got time: 1118

setup(): 193
WiFi connected: 338
got time: 1137

#################### LwIP, w/o persisted WiFi conn. ###############

setup(): 307
WiFi connected: 3348
got time: 4568

setup(): 293
WiFi connected: 3334
got time: 4346

setup(): 306
WiFi connected: 3345
got time: 4463

setup(): 301
WiFi connected: 3377
got time: 4397

Using "sntp_force_request()" this is now considerably shortened, especially with persistent WiFi connection !!!

#################### LwIP2, force, persisted WiFi conn. #############

setup(): 206
WiFi connected: 351
got time (@397ms) Sat Nov 25 19:57:36 2017

setup(): 197
WiFi connected: 343
got time (@379ms) Sat Nov 25 19:57:51 2017

setup(): 199
WiFi connected: 347
got time (@369ms) Sat Nov 25 19:58:06 2017

setup(): 203
WiFi connected: 352
got time (@398ms) Sat Nov 25 19:58:21 2017

#################### LwIP2, force, w/o persisted WiFi conn. #############

setup(): 304
WiFi connected: 3348
got time (@3533ms) Sat Nov 25 19:53:08 2017

setup(): 306
WiFi connected: 3346
got time (@3537ms) Sat Nov 25 19:54:03 2017

setup(): 307
WiFi connected: 3345
got time (@3535ms) Sat Nov 25 19:54:21 2017

setup(): 314
WiFi connected: 3355
got time (@3538ms) Sat Nov 25 19:54:39 2017

so, again, thank you for your effort

[d-a-v]

Thanks for testing ! I'll try to have a look into lwip2's sntp init(boot time)/reinit(dhcp) to get a faster ntp update once wifi link is up. For coherency, lwip2 should also be configured to allow 3 ntp servers like lwip1.4 (instead of one).

[d-a-v]

3889 @BrandonLWhite

Do you think is is easy to take into account settimeofday()'s Usec and integrate it in system time ?

[BrandonLWhite]

@d-a-v , yes for sure. I believe what should happen is to remove setting s_bootTimeSet in settimeofday (that can go back to non-extern I think), and call a new function in time.c for setting the s_bootTime_us. So something like this:

void setBootTime(uint64_t now_us)
{
     s_bootTime_us = now_us - micros64();
     s_bootTimeSet = true;
}

ensureBootTimeIsSet then becomes (to stay DRY)

static void ensureBootTimeIsSet()
{
....
        if (now_s)
        {
            setBootTime(now_s * 1000000ULL);
        }
    }
}

then over in settimeofday

if (tv) /* after*/
    {
        sntp_set_system_time(tv->tv_sec);
        // reset time subsystem
        setBootTime(tv->tv_sec * 1000000ULL | tv->tv_usec);
    }

Another thing I will throw out there, is that I think "boot time" should be renamed. It is really just an offset from micros64 that gets you the current REALTIME.

[5chufti]

... Another thing I will throw out there, is that I think "boot time" should be renamed. It is really just an offset from micros64 that gets you the current REALTIME. ...

:) my words ...

[d-a-v]

@BrandonLWhite @5chufti can you please check #4001 ?

[5chufti]

as soon as I'm back from holidays...