Client authentication triggering server rate limiting

[DBMandrake]

In recent months Honeywell have made substantial changes to their API servers, one of these is that new, aggressive rate limiting has been applied for client connection authentication attempts.

This means that even polling the servers once every 5 minutes (performing a full username/password authentication) for the purposes of graphing zone temperatures is now intermittently triggering rate limiting which locks out the client for as much as 10 minutes or more.

Investigation by a few people on the Automated home forum including 'gordonb3' have discovered that it's the OAuth authentication attempts that are being rate limited and rejected, not the actual use of the API calls once the client has a client access token, which lasts (at least for now on the V2 API) for 30 minutes from the time of last use.

So for their own graphing systems (which don't use this python client library) they have adapted to the new rate limit restrictions by locally caching the client access token and keeping track of it's expiry time and only performing a full username/password re-authentication when absolutely needed.

From what I can see while this library caches the client access token within a given script instance it does not have any means to cache and reuse the token among multiple consecutive script instances, (eg on disk) instead performing a full username/password authentication when each new script instance calls the EvohomeClient() method.

I use the evohome-client library with evohome-munin which is a Munin plugin. The architecture of munin plugins is that each graph for each zone is generated by calling a separate instance of the script sequentially one at a time.

Evohome-munin already implements disk caching of the returned zone data so that only the first instance of the script actually calls EvohomeClient() and the following instances read the zone data from disk cache and therefore only one authentication attempt is made per 5 minutes, but even this is triggering the rate limiting at the servers.

To fully solve this problem evohome-client would need to cache the client access token for a given username/password pair to disk along with a token expiry time, and then when another instance of a script calls EvohomeClient() check to see if a recently used still valid access token already exists and use that directly, bypassing the username/password OAuth steps and only attempting OAuth authentication if the token is expired or is no longer working.

Is there any possibility that cross instance caching of the client access token can be added to this library ? Without it, it's semi-unusable now for any non-persistent client script, thanks to the rate limiting Honeywell have applied, not just for 5 minute polling for graphing, but for any application that may want to make arbitrarily timed connections in response to user queries or actions. I started to look at this myself but quickly realised my python skills are not up to the task of attempting this sort of rewrite of the library.

Some discussion about the details of this problem can be found on the following pages of the Automated Heating Forum thread:

https://www.automatedhome.co.uk/vbulletin/showthread.php?5723-Evohome-app-broken/page2

Posts by gordonb3 are particularly helpful. Both V1 and V2 API authentication are affected by rate limiting and it appears that authentication attempts to both API's may be counted together towards total rate limiting and that once you are rate limited, authentication attempts on both API's fail, so sharing the cached client access token between V1 and V2 API's may be necessary to fully support clients that use both API's. (As the evohome-munin script does)

[zxdavb]

I just discovered that installing does not update

This worked for me:

pip install --upgrade evohomeclient

[paulvee]

I managed to get both V1 and V2 working. Good work guys. The documentation for V1 is exactly how I would have wanted it. Thank you! Here is my suggestion for my mostly borrowed V2, but feel free to clean it up : `#!/usr/bin/python

import time import json import io import datetime import sys

from evohomeclient2 import EvohomeClient # needs version 0.3.1 (sudo pip show evohomeclient)

username = "xxxxxxx" password = "xxxxxxx" tokens = "/home/pi/V2_access_token" # for safety, put this into the home directory

def main():

try:
    # do we have stored tokens?
    with io.open(tokens, "r") as f:
        token_data = json.load(f)
        access_token = token_data[0]
        refresh_token = token_data[1]
        access_token_expires = datetime.datetime.strptime(token_data[2], "%Y-%m-%d %H:%M:%S.%f")
except (IOError, ValueError):
    access_token = None
    refresh_token = None
    access_token_expires = None

try:
    client = EvohomeClient(username, password, refresh_token=refresh_token, \
        access_token=access_token, access_token_expires=access_token_expires)#, debug=True)

    for device in client.temperatures():
        print (device)

except Exception as error:
    sys.stderr.write('Connection failed: ' + str(error) + '\n')
    sys.exit(1)

# save session-id's so we don't need to re-authenticate every polling cycle.
with io.open(tokens, "w") as f:
    token_data = [ client.access_token, client.refresh_token, str(client.access_token_expires) ]
    json.dump(token_data, f)

if name == 'main': main() `

[paulvee]

Tks zxdavb! I looked for that option, but my version of pip does not show that with pip --help. I specifically had to use version 9.0.3 of pip otherwise evohomeclient would not be installed on my Raspberry Pi with Debian stretch.

After an upgrade of pip, it now works. That problem I had must have been fixed with the new update. Problem solved.

[paulvee]

Just for those that need some help in understanding the of use the latest features, here is a simplified script that shows how to implement the version 0.3.1 additions of evohomeclient. I'm not a very good Python programmer, but this works for me in my application. Feel free to Pythonize it. (I can't get the code inclusion to take it all as code... so I had to trick it. disregard the tripple quotes in the middle)

Enjoy!

#!/usr/bin/python3.5
import time
import json
import io
import os, sys

from evohomeclient import EvohomeClient  # needs version 0.3.1

DEBUG = True
username = "xxxxx"
password = "xxxxx"
session_id = "/home/pi/.V1_session_id"
evo_saved_data = "/home/pi/.evohome_data.dat"

def get_evo_data():
    try:
        with io.open(session_id, "r") as fin:
            user_data = json.load(fin)
    except (IOError, ValueError):
        user_data = None # no previously stored data available

try:
    client = EvohomeClient(username, password, user_data=user_data) #, debug=True)
    data = []
    if DEBUG : print("use online evohome data") # for testing
    for device in client.temperatures():
        # no check for valid temps here: 128 degrees C means sensor is not working (batt dead?)
        print(device['name'].replace(" ", "_"), device['temp'], device['setpoint']) # or use whatever you use to process the data
        # save the data when we have a communication error, so we can satisfy the regular db updates
        data.append(device)

    # store the evohome data
    with io.open(evo_saved_data, "w") as fout:
        json.dump(data, fout)

except Exception as error:
    if DEBUG : sys.stderr.write('Connection failed: ' + str(error) + '\n')
    # get the previously stored last data and use that instead
    payload = []
    with io.open(evo_saved_data, "r") as fin:
        payload = json.load(fin)

    if DEBUG : print("use previously stored data") # for testing
    for thermostat in payload:
        name = thermostat['name'].replace(" ", "_")
        temp = float("{0:.2f}".format(thermostat["temp"]))
        setpoint = float("{0:.2f}".format(thermostat["setpoint"]))
        print( name, temp, setpoint ) # or use whatever you use to process the data

# save the session-id so we don't need to re-authenticate every polling cycle.
with io.open(session_id, "w") as fout:
    json.dump(client.user_data, fout)
return

`