timeapi.org is no longer a source for date/time

[craftyguy]

Looks like this site went down (and my alarm stopped working as a result!).

I found the source code for it, here., and tried unsuccessfully to get it to run. I was going to host a mirror, but I keep fighting to get it to run in Ruby. Anyways, I've been trying to find an alternative time source to use..

[Jeija]

Oh yes, mine stopped working too :( But I couldn't be bothered to fix that yet due to other work.

The ideal solution would be to use NTP instead, that would be the standard method, but it's also more difficult to implement. An easier option would be to just use HTTP. Actually, most web servers return the current time and date in their HTTP response headers. Just try

curl -X HEAD -i http://google.com

on a system with cURL and you should see the reponse contains something like

Date: Mon, 06 Feb 2017 06:14:25 GMT

The bad news is that this is a little harder to parse and this doesn't easily take timezones into account, so we have to handle that on the ESP. The good news is there are plenty of reliable HTTP servers out there, so we could just use Google, GitHub, DuckDuckGo or some other server we trust to be reliable.

[craftyguy]

I may rewrite the timeapi system in python so I (and others) can host it. Seems like a lot of folks used timeapi so it's a shame to see it gone.

NTP is ideal, but I agree that it would be more difficult to implement.. Sigh.

I looked at the source code you posted here, and I'm not totally sure I understand the syntax you are using in your GET request, and what you expect as a response. Looks like 'ESP' is in both? If a write a new time server, I would want to make it drop-in compatible :)

On February 5, 2017 10:18:48 PM PST, Florian Euchner notifications@github.com wrote:

Oh yes, mine stopped working too :( But I couldn't be bothered to fix that yet due to other work.

The ideal solution would be to use NTP instead, that would be the standard method, but it's also more difficult to implement. An easier option would be to just use HTTP. Actually, most web servers return the current time and date in their HTTP response headers. Just try

curl -X HEAD -i http://google.com

on a system with cURL and you should see the reponse contains something like

Date: Mon, 06 Feb 2017 06:14:25 GMT

The bad news is that this is a little harder to parse and this doesn't easily take timezones into account, so we have to handle that on the ESP. The good news is there are plenty of reliable HTTP servers out there, so we could just use Google, GitHub, DuckDuckGo or some other server we trust to be reliable.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/Jeija/esp8266-light-alarmclock/issues/3#issuecomment-277595168

[Jeija]

Yes, when sending arbitrary strings to timeapi, it just returned them along with the request. So the alarm clock sends an ESP string along with the actual request, just to make sure we get an actual answer and not some error response.

The alarm clock basically requested http://www.timeapi.org/cet?ESP \u \H \M \S (urlencoded) which returned a string with format ESP DAYOFWEEK HOURS MINUTES SECONDS.

Now I'd prefer to not have to rely on a custom time server once again and you propably don't want to get in a position where you are running your own time server and are responsible for many clients that rely on your service. So I'd say getting the time over HTTP from Google or something is propably the best fix / hack for this issue.

[craftyguy]

Here's a simple ntp client for the esp8266, distributed under MIT license.

https://github.com/raburton/esp8266/tree/master/ntp

Tomorrow I can try to integrate it and see if it'll work.

On February 5, 2017 10:45:34 PM PST, Florian Euchner notifications@github.com wrote:

Yes, when sending arbitrary strings to timeapi, it just returned them along with the request. So the alarm clock sends an ESP string along with the actual request, just to make sure we get an actual answer and not some error response.

The alarm clock basically requested http://www.timeapi.org/cet?ESP \u \H \M \S (urlencoded) which returned a string with format ESP DAYOFWEEK HOURS MINUTES SECONDS.

Now I'd prefer to not have to rely on a custom time server once again and you propably don't want to get in a position where you are running your own time server and are responsible for many clients that rely on your service. So I'd say getting the time over HTTP from Google or something is propably the best fix / hack for this issue.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/Jeija/esp8266-light-alarmclock/issues/3#issuecomment-277598335

[craftyguy]

I'm not having much luck finding a solution using NTP, mainly because I don't want to get stuck trying to account for various (dynamic) time offsets like DST, which are (seemingly) arbitrary in how they are defined based on location and specific time zones.

I'm not sure if you've had any luck either, but I may end up just writing a service that runs on my server and serves date/time to the ESP in the same format that timeapi.org was providing. At least until I have more time to implement a better solution..

[Jeija]

What about the solution using HTTP I mentioned earlier?

[craftyguy]

That would require implementing a http client and parsing the output, and still adjusting offset for timezones and DST, right?

[Jeija]

Yes, but we already have an HTTP client (for timeapi.org) and parsing the date format shouldn't be that hard, since the format is always the same. The tricky part really is adjusting for DST offset which is a nightmare. So we could either make it so that you can set your time offset in the web GUI, or put a TZ database on the ESP8266 (which is not a good idea).

Since both of these solutions are not ideal, that leaves us with switching to a different time API provider or running our own time API servers. I'd say the best solution would still be to switch to some sort of official API… there has to be some API out there that at least returns the local time as a by-product (maybe some weather service or something).

[craftyguy]

How about timezonedb.com? It seems reasonably easy to use. You have to sign up for a free account to get an api key though. Looks like it supports json and XML replies. Know of any good parsers for either of those formats for the ESP?

On February 28, 2017 9:39:49 AM PST, Florian Euchner notifications@github.com wrote:

Yes, but we already have an HTTP client (for timeapi.org) and parsing the date format shouldn't be that hard, since the format is always the same. The tricky part really is adjusting for DST offset which is a nightmare. So we could either make it so that you can set your time offset in the web GUI, or put a TZ database on the ESP8266 (which is not a good idea).

Since both of these solutions are not ideal, that leaves us with switching to a different time API provider or running our own time API servers. I'd say the best solution would still be to switch to some sort of official API… there has to be some API out there that at least returns the local time as a by-product (maybe some weather service or something).

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/Jeija/esp8266-light-alarmclock/issues/3#issuecomment-283110862

[Jeija]

timezonedb.com looks fine, maybe we could also include an API key in this repo for testing purposes but urge people to create their own keys. JSON would be great, since Espressif provides a JSON parser for the ESP8266, refer to Section 6.2. Related JSON APIs in the official SDK API Guide.

[craftyguy]

Thank you, I will take a look at the official API guide. I don't know what timezonedb's official stance is on a generally available API key is, that might be the only kink in this plan.. but in any case this project could refer them to the timezonedb site to register for a free (for now) account? On Tue, Feb 28, 2017 at 10:55:08PM -0800, Florian Euchner wrote:

timezonedb.com looks fine, maybe we could also include an API key in this repo for testing purposes but urge people to create their own keys. JSON would be great, since Espressif provides a JSON parser for the ESP8266, refer to Section 6.2. Related JSON APIs in the official SDK API Guide.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/Jeija/esp8266-light-alarmclock/issues/3#issuecomment-283260561

[Jeija]

Technically there are already several timezonedb API keys out there on GitHub, e.g. here or here. I think we should be fine after maybe obfuscating the API key a little, so that people can't just copy-paste our key.

[craftyguy]

Alright, so I kind of gave up on this specific implementation of a light alarm clock. After trying to decode the espressif documentation for json parsing, among other issues, I decided to scrap it and take a different approach.

What I ended up doing was implementing a MQTT client using micropython on the esp8266 that receives a command to "start ramping the light and hold for X amount of time." I'm integrating this into openHab so all of the actual alarm handling while be on my openHab server, which presumably always knows the correct time (and time zone! and DST!), so it just uses the esp8266 as a switch for "ringing" the alarm when it is time.

No more trying to keep up with time synchronization on the esp, and I got to learn quite a bit about micropython!

As an added bonus, there's virtually no flickering of the light in this implementation since there's no webserver running on the esp!

Here's my project if you're interested. It's drop-in compatible with the existing device I had assembled that ran off your firmware.

[Jeija]

Sorry for taking so much time, but I have now added support for TimeZoneDB: https://github.com/Jeija/esp8266-light-alarmclock/commit/91302b2b1813b908ebb04bfbb1c6b4d17ce4da6c

Unfortunately this means that you will have to register with TimeZoneDB and provide your own API key, but at least this should make the alarm clock work again.