Use timezonedb.com API instead of hosting the PHP yourself

trunet commented 6 years ago

This is an enhancement request to use https://timezonedb.com/ API that is free for non-commercial use.

Example:

$ curl 'https://api.timezonedb.com/v2/get-time-zone?key=[YOUR_API_KEY]&format=json&by=zone&zone=Europe/Amsterdam'
{"status":"OK","message":"","countryCode":"NL","countryName":"Netherlands","zoneName":"Europe\/Amsterdam","abbreviation":"CEST","gmtOffset":7200,"dst":"1","dstStart":1521939600,"dstEnd":1540688399,"nextAbbreviation":"CET","timestamp":1522768088,"formatted":"2018-04-03 15:08:08"}

rwpalmer commented 6 years ago

Thank you for the enhancement request. I REALLY like the idea, and I've already registered for an API-key. I should have time to scope the effort next week.

rwpalmer commented 6 years ago

I have a rough prototype working ...

Two queries are required to gather the required data. timezonedb only allows one query/sec with a free license, but the second query has not been blocked in the initial tests with a free license.
Will look at what it would take to enable the GPS and IP options as well.

trunet commented 6 years ago

hello rwpalmer... why do you need two queries?

rwpalmer commented 6 years ago

Sorry, this explanation got a bit long, but timezone calculations are not as straight forward as one might think.

This library is designed to set time for the Particle MPUs. To do this properly, we need to set two data elements.

The "Standard UTC Offset" (in hours). This is an integer ranging from -12 to +12, where 0 = UTC (aka GMT).
The "DST Offset" (in hours). This is zero for time zones that do not observe DST. For time zones that do observe DST, the number is usually -1 or +1, but some fractional offsets do exist. This is therefore a floating point number.

For Particle MPUs, DST is toggled on and off by two functions beginDST( ) and endDST( ). When DST is toggled on, the Particle MPU adds both Standard UTC Offset and DST Offset to UTC time to calculate the local time. When DST is toggled off, the Particle MPU only adds the Standard Offset to UTC time.

timezonedb only provides one bit of data directly, and that is {"dst":"0"} or {"dst":"1"}.

When timezonedb returns a JSON with {"dst":"0"}, we can compute: Standard UTC Offset = gmtOffset/3600.

When timezonedb returns JSON with {"dst":"1"}, we do not have enough information to compute the Standard UTC Offset, but dstEnd provides the information we need to run a second query. That second query just adds "&time=nnn" to the first query, where nnn represents (dstEnd + 1). This second query returns JSON with data that represents settings as they will be after the next DST transition, where {"dst":"0"}, and that gives us the data required to compute the Standard UTC Offset.

The only way to compute DST Offset is to get gmtOffset from both {"dst":"0"} and {"dst":"1"} JSON. We can then compute: DST Offset = ((gmtOffset(1) - (gmtOffset(0))/3600).

Only one query is needed for time zones that to not use DST. We can recognize them because the JSON will include {"dst":"0"} and {"dstEnd":0}. This tells us that the JSON reflects the Standard UTC Offset and that this setting has no end.

trunet commented 6 years ago

Thanks for the explanation.

You should build the library to have 3 working methods that the user can choose: offline (eeprom), online (using the API) and gps connected. (As you said before).

I'm not using tzLib for now, what I did was to write a Thread* calling the API on 00 minute of every hour that set Time.zone(gmtOffset/3600) if it received an HTTP/200. I didn't use beginDST() / endDST(). This also avoided the transitions calculation. I preferred the simple way, but more dependent as this needs to be called every hour.

rwpalmer commented 6 years ago

I now have an early pre-alpha with two methods for setting the local time zone.

setZoneByID("");
setZoneByGPS(, );

These appear to functioning properly, and I'm looking into the third method:

setZoneByIP("");

The only timezonedb option for IP-lookup that I find is one where my library would need to provide the IP address. This is a for-fee option ($5 USD per month) ... I presume because it is designed for web sites to track where their customers are. I've sent an email to the timezonedb developer to ask if he has (or can add) a way to provide time zone information based on the query's source IP address ... preferably a free option. To date, I have not heard back.

If the existing option is the only option that is available:

Particle has a wifi function I can call to obtain the device's IP address, but many/most IOT devices would be assigned a non-routable 10.xxx.xxx.xxx or 192.168.xxx.xxx address, so I don't think this gets me anywhere.
My library could query ipify: http://api.ipify.org/?format=json to obtain the IP address of the device's router. This would give us an IP address that could be used for geolocation, but it adds yet another query to the solution.
I'm going to continue to wait for a response from the timezonedb developer for a while. I'm also going to contact one of the Particle guys to see if they have any ideas. Let me know if you have any thoughts on this topic.

trunet commented 6 years ago

IP Geolocation is a really good idea, although on my case, I would just set the timezone as I'm programming my own devices and I know where they're physically located. Can you explain what is the TimeZoneByID(), is this the one that will load from EEPROM? What about a setZoneByTimezone() that will query timezonedb?

I don't think they will provide IP lookup for free as they have to pay for the database, operational procedure to update it, so on, but worth to try. I think $5 is too expensive.

You could think about building the service yourself and charge the people to use it (a cheaper option, like $5 a year or so). You buy the geoip city database from maxmind (or pay per API call, they have an online API as well), query the source IP to get the city and than you match on https://www.iana.org/time-zones and answer with what you need for the library.

rwpalmer commented 6 years ago

I forgot I was writing to you with MarkDown ... which screwed up the command syntax in my previous post. The commands are:

    setZoneByID("<timezoneID>");
    setZoneByGPS(<latitude>,<longitude>);
    setZoneByIP();

When the device boots, the library will automatically set the timezone to:

the timezoneID stored in EEPROM when there is a timezoneID stored in EEPROM.
the "default timezoneID when there is no timezoneID stored in EEPROM. (This would be a "first boot" scenario.)
when booting, tzLib will attempt to validate/update the timezone data is stored in EEPROM ... because the device could have been offline for "who knows how long?". If the device is offnet, the stored data will be used asis.
- any of the setZoneBy.... commands will perform the same validate/update procedure based on whatever parameters it gets. If the returned data is valid and differs from what is in EEPROM, the EEPROM data will be updated.

I just performed a setZoneByID("Europe/Amsterdam"), and tzLib set my device to these settings.

        tzLib.getHTTPStatus() returns:          HTTP Processing Completed Normally
        tzLib.getZone() returns:                Europe/Amsterdam
        tzLib.getZoneAbbr() returns:            CEST
        Particle Time.zone() returns:           1.00
        Particle isDst() returns:               true
        Particle Time.getDSTOffset() returns:   1.00
                        Local Time is:  Wed Apr 25 21:15:47 2018

The Particle Time.Zone() setting represents the "standard offset" (current offset when DST is false) Since DST is true, the current offset is (1.00 + 1.00), or 2.

rwpalmer commented 6 years ago

This looks other options exist: for getting from IP to timezone: http://ip-api.com/json looks like the best so far ... https://timezoneapi.io/developers/ip-address returns too much data ...

Still looking for the holy-grail

trunet commented 6 years ago

Nice. The implementation looks really good. Let me know if you need any help with testing or writing some code.

I didn't know these 2 APIs you sent, looks very promising. Probably after we setup one, we can make this modular to let the user choose which one he wants to use.

rwpalmer commented 6 years ago

I'm going to prototype a setZone by IP solution that uses "ip-api" to go from IP to timezone, and then let it fall through the setZonebyID logic from there. That should be relatively easy to implement. (I expect to have functional code by EOD tomorrow.)

I'm wresting with the idea of using a different name for this library because it is significantly different than the original TzLib. I think the original library is better suited for people who want an end-to-end solution that is 100% under their control. The new library offers more features, but relies on external sites over which we have no control. They could go away tomorrow ... Hmm, since the library makes different sites play together, maybe it should be called: "TzMaestro"?

It would be GREAT if you could help with the testing. I am also considering publishing the library on Particle as a "beta" ... to let others test and make suggestions. Perhaps someone there knows where better sources may exist. I'd also like to get the code reviewed by one of the Particle experts. I'm still relatively new to this platform.

trunet commented 6 years ago

I really think that you should keep all on one library. This library should work as a full standalone using the EEPROM but also using the third-party methods like GPS and APIs.

You can rely on #define #ifdef to hide pieces during compiling time to save flash and memory space if you don't use the other methods.

Duplicating the libraries will cause some code duplication, fixing bugs will need to be done on both libraries sometimes.

Anyway, the call is yours :grinning:

I can definitely help, just let me know when it's done.

rwpalmer commented 6 years ago

I've decided to leave tzLib ASIS on GitHub ... It's essentially a static solution that requires little maintenance, and it is mostly used by organizations who can pay me if they want changes made.

As I see it, the new library will take an evolutionary path to take advantage of Particle Community ideas, and new web sites. We also may need to quickly recover when sites are phased out. My current thought is to name the new library TwLib ... where the w stands for weather ... which we can get (along with sunrise and sunset) via OpenWeather.org: (https://openweathermap.org/current)

I particularly wanted to add the sunrise/sunset data ... for one of the projects I plan to do down the road.

Your thoughts?

trunet commented 6 years ago

that's fine for me.

rwpalmer commented 6 years ago

Alas, the openweathermap.org API was a disappointment on a couple of fronts ... I was able to get sunrise/sunset data from api.sunrise-sunset.org, but I decided to scrap the idea "for now" because their json had features that the TzLib parser was not designed to handle.

Current Status: There is one section of code that I don't like. My plan is to fix that, and then prep a package for you. The package will include one of the example programs that I have been working with. It has four particle functions. They set zone by ID, GPS, IP, and to the configured "default" zone.

I want to mention that the code will write 128 bytes to the device's EEPROM. The location of the 128 bytes is configurable. (default = bytes 0-127).

If all goes as planned, I'll have something to you by EOD Monday.

rwpalmer commented 6 years ago

The library is still a bit raw ... but you can download it from http://208.85.39.75/TzCfg/ The simplest implementation would be something like this:

#include <TzCfg.h>

TzCfg tzCfg;

void setup() {
    Serial.begin(115400);
    tzCfg.begin();
    tzCfg.setApiKey_timezonedb((char*)"<your timezonedb API key>");
    tzCfg.setTimezoneByIP();
    Serial.printf("\n\t\t\tLocal Time is:\t%s\r\n", (const char*)Time.format(Time.now(), TIME_FORMAT_DEFAULT));
}

void loop() {
    tzCfg.maintainLocalTime();
}

To upload the library to Particle via the Particle CLI:

unzip the download.
make the TzCfg folder your current folder
enter the command "particle library upload" This will create a private library that you can add to projects.

The library includes one example project. It allows you to run all of the setTimezoneBy commands and lets you erase the TzBlock from EEPROM.

Of course, you can also create a project and manually copy the library source files to your project ... your call.

trunet commented 6 years ago

thanks... I will take a look this week.

rwpalmer commented 6 years ago

Just FYI: Code just updated ... more solid now.

rwpalmer commented 6 years ago

I have removed the code from the link I gave you earlier in this thread.

I've created another GitHub repository: https://github.com/rwpalmer/TzCfg. My 'first draft' documentation is there ... no code yet.

I've just uploaded the latest code to Particle as a private library. I will publish it as soon as I am comfortable doing so. I'll put code on GitHub soon after that.

Thanks again for the timezonedb idea. I was not aware of the site. Would you mind a public thank you on the tzCfg readme page? If not, please let me know if you want me to add a link to work that you have done, or a bio page.

Cheers

trunet commented 6 years ago

hi. thanks for the really nice work. I just replaced my ugly implementation with your library and it worked flawlessly.

I'm ok with the credits, although I didn't do anything apart of bringing the idea.

BR, Wagner

rwpalmer commented 6 years ago

Well, you lit the match, so we should bask in the glow of the fire together ...

FYI: I am currently working on another Particle library for the dfPlayer ... which is an MP3 player that can be controlled by a serial interface. One of the nice things about the device is that it can be used without a library. However, the native command set is hard to work with and the the documentation is poor. I think a lot of people would benefit from a library ... at least to get started. I was starting to get ready to publish that library when I saw your comment on GitHub.

Perhaps we can do a project together sometime ... I'd like that.

rwpalmer commented 6 years ago

FYI: I published the library on Particle ... The code matured a lot from the versions I posted on my server. If you opt to use my code, please use the published version.

rwpalmer / tzLib

Use timezonedb.com API instead of hosting the PHP yourself #1