Client authentication triggering server rate limiting

[DBMandrake]

In recent months Honeywell have made substantial changes to their API servers, one of these is that new, aggressive rate limiting has been applied for client connection authentication attempts.

This means that even polling the servers once every 5 minutes (performing a full username/password authentication) for the purposes of graphing zone temperatures is now intermittently triggering rate limiting which locks out the client for as much as 10 minutes or more.

Investigation by a few people on the Automated home forum including 'gordonb3' have discovered that it's the OAuth authentication attempts that are being rate limited and rejected, not the actual use of the API calls once the client has a client access token, which lasts (at least for now on the V2 API) for 30 minutes from the time of last use.

So for their own graphing systems (which don't use this python client library) they have adapted to the new rate limit restrictions by locally caching the client access token and keeping track of it's expiry time and only performing a full username/password re-authentication when absolutely needed.

From what I can see while this library caches the client access token within a given script instance it does not have any means to cache and reuse the token among multiple consecutive script instances, (eg on disk) instead performing a full username/password authentication when each new script instance calls the EvohomeClient() method.

I use the evohome-client library with evohome-munin which is a Munin plugin. The architecture of munin plugins is that each graph for each zone is generated by calling a separate instance of the script sequentially one at a time.

Evohome-munin already implements disk caching of the returned zone data so that only the first instance of the script actually calls EvohomeClient() and the following instances read the zone data from disk cache and therefore only one authentication attempt is made per 5 minutes, but even this is triggering the rate limiting at the servers.

To fully solve this problem evohome-client would need to cache the client access token for a given username/password pair to disk along with a token expiry time, and then when another instance of a script calls EvohomeClient() check to see if a recently used still valid access token already exists and use that directly, bypassing the username/password OAuth steps and only attempting OAuth authentication if the token is expired or is no longer working.

Is there any possibility that cross instance caching of the client access token can be added to this library ? Without it, it's semi-unusable now for any non-persistent client script, thanks to the rate limiting Honeywell have applied, not just for 5 minute polling for graphing, but for any application that may want to make arbitrarily timed connections in response to user queries or actions. I started to look at this myself but quickly realised my python skills are not up to the task of attempting this sort of rewrite of the library.

Some discussion about the details of this problem can be found on the following pages of the Automated Heating Forum thread:

https://www.automatedhome.co.uk/vbulletin/showthread.php?5723-Evohome-app-broken/page2

Posts by gordonb3 are particularly helpful. Both V1 and V2 API authentication are affected by rate limiting and it appears that authentication attempts to both API's may be counted together towards total rate limiting and that once you are rate limited, authentication attempts on both API's fail, so sharing the cached client access token between V1 and V2 API's may be necessary to fully support clients that use both API's. (As the evohome-munin script does)

[zxdavb]

My thoughts...

My understanding is that the two versions of evohomeclient use different authentication schemes, only v2 uses OAuth - thus you can't authenticate once for both APIs.

I saw a post by gordonb3 that said the v2 access token lasts an hour - all mine last 30 mins (e.g. "expires_in":1799). The v2 client re-authenticates (using username, password) before this token expires (see below), but it could easily be modified to use a refresh_token instead.

if datetime.now() > self.access_token_expires:
    self._basic_login()

Having evohomeclient place an access_token (or refresh_token) in permanent storage doesn't seem to be good security to me. I seriously doubt whether @watchforstock would go for it. However, the client could be written to accept a refresh_token rather than a set of credentials? Maybe.

I note also that the current v2 client does not expose the refresh_token (this is easily remedied).

I wrote the evohome integration for Home Assistant, and:

1. I have seen similar problems (HTTP_429) & came to the same conclusions, but
2. I am not as affected because I persist the client object between calls to the API (every 120-300 seconds)
3. My impression is that the rate limits are separate for each version of the API

I guess I'm saying the proper 'solution' should be within the code using the evohomeclient client library, and not for evohomeclient to provide a work-around. I expect this is not easily done in all cases, though...

Instantiating the client once per polling cycle is similar to where a piece of code calls evohomeclient v2 once for each zone, instead of once for the location -they're both just a bad idea.

[DBMandrake]

Thanks for the reply,

My understanding is that the two versions of evohomeclient use different authentication schemes, only v2 uses OAuth - thus you can't authenticate once for both APIs.

Ok, I wasn't sure about this as I don't know a lot about how the library works behind the scenes. Even if they don't both use OAuth, in my testing there seems to be some server side connection between the two API's when it comes to rate limiting.

For example aside from evohome-munin I have a couple of very basic tests scripts that I use to poll the V1 or V2 API's and print the zone temperature data.

If I manually poll one API a few times in a row until it gets rejected, and then try to poll the other API it too is rejected, for at least 5 minutes, whereas if I hadn't polled the first API a few times the other API is not rejected. It seems that once you hit the rate limit your IP address is blocked and all further connection attempts of any sort are rejected until you back off. That's my finding anyway. I suspect they have had to implement this sort of behaviour to fight DDoS attacks.

I saw a post by gordonb3 that said the v2 access token lasts an hour - all mine last 30 mins (e.g. "expires_in":1799).

Yes I see 30 minutes reported as well when I test. I think the post of gordon's you're referring to is from a few months ago, which would suggest that Honeywell have reduced the token lifetime since that time. So whatever fix is used, the code really needs to calculate the token length from the current and expiry time and use that rather than any hard coded expiry time.

Having evohomeclient place an access_token (or refresh_token) in permanent storage doesn't seem to be good security to me. I seriously doubt whether @watchforstock would go for it. However, the client could be written to accept a refresh_token rather than a set of credentials? Maybe.

I note also that the current v2 client does not expose the refresh_token (this is easily remedied).

Yes on thinking about it in hindsight I think you're right about the client library itself not writing the token in permanent storage for security reasons. It should be the calling script that explicitly decides whether to save the token and where.

In that case what would be needed is

1) A way for the calling script to save the most recent token and token expiry time to a place of its choosing before exiting - I found I could read client.access_token and client.access_token_expires successfully from my script, but from what you say if the library has refreshed the token meanwhile it will be out of date. (However in my specific use case of a short lived script it would not have had time to need refreshing, so I would have got away with it)

2) Some way for the calling script to call the library with a token instead of a username and password and have everything initialised as normal. Whether that's modifying the EvohomeClient() call, adding a new one that takes tokens instead of usernames/passwords I don't know.

It would be up to the calling script to check the expiry time on the saved token and discard it if it is too old, and also deal with a possible connection failure if the token fails by then requesting a username/password based login.

Although that's more work for the client script it could easily be incorporated in the evohome-munin script and I'm sure in others.

I guess I'm saying the proper 'solution' should be within the code using the evohomeclient client library, and not for evohomeclient to provide a work-around. I expect this is not easily done in all cases, though...

With some more thinking about it I agree. Although it's a pity that the client script now has to worry about the fact that Honeywell rate limits authentication. It detracts a bit from the original nice clean abstraction the library had, and a method of operation that worked fine for years.

Instantiating the client once per polling cycle is similar to where a piece of code calls evohomeclient v2 once for each zone, instead of once for the location -they're both just a bad idea.

I agree in principle, but in the specific case of evohome-munin it's just the way munin works, it calls one plugin per graph (zone) and there isn't really anything I can do about that. The evohome-munin plugin already caches zone data returned so it effectively only calls evohome-client once per 5 minute graphing period regardless of the number of zones and gets all zone data from that one call.

The only other possibility would be to write an intermediate python daemon that is trying to keep a persistent connection open with evohome-client all the time, and pass requests via it from each plugin script instance, but that's an awful lot of complication for a problem that shouldn't really exist.

I prefer the idea of being able to save the token and token expiry before exit and then use it as an alternative means of creating a new connection with evohome-client next time a script is instantiated within the token expiry time.

[watchforstock]

All, thanks for your input. I've had a quick look and I think with some small changes to the evohomeclient2 library, we can make it possible to inject previous state. I also think that the library can do some work to help the caller (for example, by taking the previous state as well as the username and password and transparently calling for login if required).

I've started making the changes this evening, but (ironically enough) in testing have been hitting the authentication limit which has slowed me down! Hopefully I'll be able to push a change in the next couple of days for testing / comment.

[DBMandrake]

That’s great news. :)

Is there anything that can be done for the V1 API as well ?

I actually use the V1 API for everything except querying hot water on/off status due to the increased temperature accuracy it offers.

Because the method for the V1 API doesn’t support a debug output mode I can’t see exactly how the V1 API is failing when rate limited (not sure if it’s only authentication failing or any query) however it is definitely rate limited in some way just like the V2 API.

Could a corresponding debug option be added to the V1 Evohomeclient() method to assist in diagnosing how it fails ?

[watchforstock]

I've made the various changes necessary (including adding a debug option to the v1 client). I'm pretty sure these changes should be backwards compatible, but at the moment these changes are in a branch (https://github.com/watchforstock/evohome-client/tree/allow_session_reuse) until I've had some confirmation this is actually the case.

Please see examples below. In the v1 client, a call needs to have been made before the user_data variable is populated. In the v2 client, it will have authenticated as part of the constructor.

For the v1 client, I can't see any information coming back from the server as to the length of the session token validity. As such, at the moment the library doesn't know when it's become invalid and so doesn't automatically reauthenticate.

For the v2 client, the library will automatically refresh credentials when they timeout (either during calls for the same client or when credentials that have been stored have expired).

I'd welcome feedback on whether this seems to be helping and ideally confirmation that it hasn't broken any existing code. If anyone can find details of when the session in the v1 client is valid till then we can wrap the same auto-reauthenticate logic in there too

# Evohome 1 example
from evohomeclient import EvohomeClient

user_data = None
c = EvohomeClient(username, password, debug=True, user_data=user_data)
for t in c.temperatures():
    print(t)
print(c.user_data)

d = EvohomeClient(username, password, debug=True, user_data=user_data)
for t in c.temperatures():
    print(t)

# Evohome 2 example

from evohomeclient2 import EvohomeClient

access_token = None
access_token_expires = None

c = EvohomeClient(username, password, access_token=access_token, access_token_expires=access_token_expires)

# Save these and restore later
access_token = c.access_token
access_token_expires = c.access_token_expires

d = EvohomeClient(username,password, access_token=access_token, access_token_expires=access_token_expires)

[DBMandrake]

This looks great.

I'm a bit tied up for a few days for any in depth testing but I'll do some testing as soon as I can and report back.

I'll also test my existing client scripts work as-is with the updated library to ensure there are no backward compatibility problems.

[gordonb3]

Hi, just checking in...

Yes, the OAuth fixed timeout used to be 3599 seconds for as long as I can remember. Haven't checked recently, but I think that when I added implementing the refresh token I also started using the expiration field to determine when to use that token.

The V1 API does not use OAuth but resets the expiration time with every call you make to it. I determined this timeout to be 15 minutes.

Edit: hmmm, no I didn't. I hardcoded the 3599 value. Need to fix that.

[zxdavb]

@watchforstock

I'm just testing a PR with a slight consolidation of your code - it removes all the self_headers stuff. Will wait 30 mins & submit it for your consideration...

def headers(self):
    if self.access_token is None or self.access_token_expires is None:
    # token is invalid
        self._basic_login()
    elif datetime.now() > self.access_token_expires - timedelta(seconds = 30):
    # token has expired
        self._basic_login()

    return \
        {
            'Authorization': 'bearer ' + self.access_token,
            'Accept': 'application/json, application/xml, text/json, text/x-json, text/javascript, text/xml'
        }

In addition, there will need to be code to catch invalid tokens - I can have a go at this.

I'd also like to explore using the refresh_token rather than re-authenticating after 30 mins, you up for that?

[gordonb3]

If you require a pointer, just look here: https://github.com/gordonb3/evohomeclient

Most of that library is directly linked to your python code so you only have to do the reverse of what I did for the code that was added later.

[zxdavb]

If you require a pointer, just look here: https://github.com/gordonb3/evohomeclient

OMG - you think I can read c++! 🤣 (... updating LinkedIn profile ...)

[watchforstock]

@gordonb3 @zxdavb Thanks both. Always happy to take PRs that tidy things up or improve it! Adding use of the refresh_token seems like a natural step since we're improving that area of the code.

[gordonb3]

You don't have to be able to read it, just understand the structure of it. And as said, the structure of that C++ code is closely linked to the python code from this project. Of course the main point here is how to construct the web request and that should be easy enough to read in there?

[zxdavb]

@watchforstock How would you feel about putting a wrapper around the requests object?

[watchforstock]

@zxdavb re: wrapping requests - I'm not necessarily against it, but not sure I understand the purpose of wrapping it that you're thinking? Is it to manage headers / authentication or something else?

[zxdavb]

I was thinking forward to how we could incorporate the refresh token.

Actually, thinking about it now, wrapping the requests object is not that great an idea.

[gordonb3]

Hmmm... that's interesting.

I'm still seeing 3599 seconds for the token expiration. Could that be related to the application auth key? I'm still using the old one that Andrew sniffed from the original app, but of course Joe Zwack gave you a new one some two years back. What happens to that expiration time if you revert to commit 78961ce ?

[zxdavb]

Oh! I can't wait to get home and test that!

I don't think it changes anything, because we should be heading towards utilising the refresh_token rather than simply reauthenticating when the access_token runs out (as is currently the case).

This enhancement should be achievable with no visibility to the those who utilise evohomeclient2 (I can't speak with any confidence for the v1 API).

[zxdavb]

@gordonb3 What's you experience with the refresh_token?

Specifically, how long until it expires and Honeywell requires re-authentication with username/password?

I am just thinking whether evohomeclient2 should support access tokens only, or refresh tokens only, or both?

[gordonb3]

In testing I found that re-using a refresh token from the previous day did not work, but they do appear to stay valid for at least several hours after the access token expired. From what I can see refresh tokens never expire if you successfully keep using them to request a new access token.

evohomeclient2 should support both because refresh tokens do not appear to be affected by the new rate limiting and only the access token allows you to query the system and submit overrides.

Btw, I do agree with Andrew that the library should not handle store and retrieve of these tokens automatically, i.e. without the user being aware of this happening.

[zxdavb]

If the refresh_token (RT) will do the job, why not just use it to get a new access_token (AT), even if the old AT has not expired? It would make for much simpler code...

The calling code would only have to retrieve/store/pass 1 token, instead of 3 (two tokens and a date).

[zxdavb]

This is the version I'm testing now:

def _headers(self):
    if self.access_token is None or self.access_token_expires is None:
        self._basic_login()

    elif datetime.now() > self.access_token_expires - timedelta(seconds=30):
        self._basic_login()

    return {'Accept': HEADER_ACCEPT,
            'Authorization': 'bearer ' + self.access_token}

def _basic_login(self):
    """Obtain an access token from the vendor.

    First, try using the refresh_token, if one is provided, otherwise
    authenticate using the user credentials."""

    self.access_token = None
    self.access_token_expires = None

    if self.refresh_token is not None:
        credentials = {'grant_type': "refresh_token",
                       'scope': "EMEA-V1-Basic EMEA-V1-Anonymous",
                       'refresh_token': self.refresh_token}

        if not self._obtain_access_token(credentials):
            self.refresh_token = None

    if self.refresh_token is None:
        credentials = {'grant_type': "password",
                       'scope': "EMEA-V1-Basic EMEA-V1-Anonymous EMEA-V1-Get-Current-User-Account",
                       'Username': self.username,
                       'Password': self.password}

        self._obtain_access_token(credentials)

def _obtain_access_token(self, credentials):
    """Get an access token using the supplied credentials."""
    url = 'https://tccna.honeywell.com/Auth/OAuth/Token'
    headers = {
        'Accept': HEADER_ACCEPT,
        'Authorization': HEADER_AUTHORIZATION_BASIC
    }
    data = {
        'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
        'Host': 'rs.alarmnet.com/',
        'Cache-Control': 'no-store no-cache',
        'Pragma': 'no-cache',
        'Connection': 'Keep-Alive'
    }
    data.update(credentials)

    response = requests.post(url, data=data, headers=headers)
    if response.status_code != requests.codes.ok:                            # pylint: disable=no-member
        response.raise_for_status()

    try:  # validate the access token
        tokens = self._convert(response.text)

        self.access_token = tokens['access_token']
        self.access_token_expires = (datetime.now() + timedelta(seconds=tokens['expires_in']))
        if credentials['grant_type'] == "password":
            self.refresh_token = tokens['refresh_token']

    except KeyError:
        return False

    return True

[watchforstock]

Looks good. Not sure I can keep up with your pace :)

[zxdavb]

I had the day off work!

[zxdavb]

Sorry, but the latest PR is failing after 60 mins (and not 30 mins!).

2019-02-21 22:41:54 DEBUG (SyncWorker_17) [custom_components.climate.evohome_cc] _update_state_data(): client.locations[loc_idx].locationId = 2738909
2019-02-21 22:41:54 ERROR (MainThread) [homeassistant.helpers.entity] Update for climate.my_home fails
Traceback (most recent call last):
  File "/srv/hass/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 221, in async_update_ha_state
    await self.async_device_update()
  File "/srv/hass/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 349, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/dbonnes/.homeassistant/custom_components/climate/evohome_cc.py", line 892, in update
    self._update_state_data(evo_data)
  File "/home/dbonnes/.homeassistant/custom_components/climate/evohome_cc.py", line 750, in _update_state_data
    client.locations[loc_idx].status()[GWS][0][TCS][0])
  File "/srv/hass/lib/python3.6/site-packages/evohomeclient2/location.py", line 23, in status
    r = requests.get('https://tccna.honeywell.com/WebAPI/emea/api/v1/location/%s/status?includeTemperatureControlSystems=True' % self.locationId, headers=self.client._headers())
  File "/srv/hass/lib/python3.6/site-packages/evohomeclient2/__init__.py", line 49, in _headers
    self._basic_login()
  File "/srv/hass/lib/python3.6/site-packages/evohomeclient2/__init__.py", line 69, in _basic_login
    if not self._obtain_access_token(credentials):
  File "/srv/hass/lib/python3.6/site-packages/evohomeclient2/__init__.py", line 102, in _obtain_access_token
    response.raise_for_status()
  File "/srv/hass/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://tccna.honeywell.com/Auth/OAuth/Token

I guess I'm doing something wrong with a header, or some such... Will look at it Saturday.

[zxdavb]

@gordonb3 Am I reading your code right - it seems that every time you use a refresh token to obtain a new access token, you also get a new refresh token?

/* 
 * Renew the Authorization token
 * Throws std::invalid_argument from (web) send_receive_data
 */
bool EvohomeClient::renew_login()
{
    std::vector<std::string> lheader;
    lheader.push_back("Authorization: Basic YjAxM2FhMjYtOTcyNC00ZGJkLTg4OTctMDQ4YjlhYWRhMjQ5OnRlc3Q=");
    lheader.push_back("Accept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml");
    lheader.push_back("charsets: utf-8");

    std::stringstream pdata;
    pdata << "installationInfo-Type=application%2Fx-www-form-urlencoded;charset%3Dutf-8";
    pdata << "&Host=rs.alarmnet.com%2F";
    pdata << "&Cache-Control=no-store%20no-cache";
    pdata << "&Pragma=no-cache";
    pdata << "&grant_type=refresh_token";
    pdata << "&scope=EMEA-V1-Basic%20EMEA-V1-Anonymous";
    pdata << "&refresh_token=" << v2refresh_token;
    pdata << "&Connection=Keep-Alive";

    std::string s_res;
    try
    {
        s_res = send_receive_data("/Auth/OAuth/Token", pdata.str(), lheader);
    }
    catch (...)
    {
        throw;
    }
    if (s_res[0] == '[') // got unnamed array as reply
    {
        s_res[0] = ' ';
        size_t len = s_res.size();
        len--;
        s_res[len] = ' ';
    }

    Json::Value j_login;
    Json::Reader jReader;
    if (!jReader.parse(s_res.c_str(), j_login))
        return false;

    std::string szError = "";
    if (j_login.isMember("error"))
        szError = j_login["error"].asString();
    if (j_login.isMember("message"))
        szError = j_login["message"].asString();
    if (!szError.empty())
        return false;

    v2refresh_token = j_login["refresh_token"].asString();
    std::stringstream atoken;
    atoken << "Authorization: bearer " << j_login["access_token"].asString();

[gordonb3]

You are reading correct but interpreting wrong ;)

As far as I know the refresh token does not change. However since the refresh token is part of the response my first idea was that I should process/save it. That was before I fully understood the OAuth mechanism. I just left that bit in there in the final code because, well..., it didn't hurt. And because I'm lazy.

[zxdavb]

Ha! My c++ skills are flying.

My (very recent) reading of the OAuth protocol is that it has the option of issuing a new refresh token every time it is used & that's what I think is happening here...

"The server may issue a new refresh token in the response, but if the response does not include a new refresh token, the client assumes the existing refresh token will still be valid."

...and testing confirms this is the case.

This is why my code worked for 60 mins, then failed - the refresh_token worked at 30 mins, but was stale after 60 mins.

[zxdavb]

OK, I think I've done it - I have submitted a PR #64

[ctwins]

Hello all, (sorry for unperfect english) I just discover the new (experimental ?) branch and these messages. For information, I integrated your client in my Jeedom evohome plugin since last year (https://github.com/ctwins/evohome4jeedom), and updated up to the last 0.2.8. As you, the "Too many request errors" appear frequently from few weeks, although my reading period is set to 10mn min. (I do a double call to the V2 first, V1 next, to obtain the 0.01 measures). So, I did exactly what you did, in your new branch, by reinjecting session (V1) and/or accessToken/tokenTimeout (V2) which are returned by each call. As you, I didn't find the refresh call gives the soluce on a long time, and chosen to renew the token just before the 30min of living (which mark the V1 session to renew in the next call). BTW, Jeedom plugin is PHP, and not as a good PHP writer, I choosen to keep the Python layer as 'bridge' to the Honeywell server, which manages all what I need. Of course, I embed now - since 0.3.0 version - the client directly inside my plugin (see my github). I hope it can help, although a bit late at this time regarding your work(s).

I have more adjustments to do, as the PHP caller could now request to more than 1 location (from a loop), and as I said above, if I mark the V1 session to renew, and try to renew on the n+1 call (of the location loop), the bad error could appear again.

[DBMandrake]

Watching this test branch develop with interest, but I've been holding back testing until it takes shape as changes seemed to be moving very quickly... or is it now at a point where obvious issues are sorted and wider testing is warranted ?

[watchforstock]

It has been moving quickly, but feels like it's getting there and would be worthy of others testing - especially if you've previously had issues with being throttled. It feels like quite a significant change to the library and as such I'd be keen to have it tested by a number of people across a number of application.

[zxdavb]

Speaking as someone who has submitted some of those recent PRs, it would be great to have some testing.

I think there may be some breaking changes, but only in contrived situations.

I've taken the opportunity to do a bit of a refactor, and Andrew has kindly accepted those PRs...

I think I see an opportunity for a few more improvements, but I don't have any plans to make any PRs that might cause breaking changes down the line.

So my vote is start testing!

[DBMandrake]

Ok, I'll have a go. I don't know a lot about how python libraries (modules ? plugins ?) reside in the system - can there only be one version of the library installed at once ?

Am I best to just git clone and checkout the test branch and run setup.py to install it, then if I need to revert, just checkout the regular branch and run setup.py to overwrite it and that will happily go back to an older version ?

Is accessing the new token caching functionality from a client script perspective still as originally proposed back in post 6, especially in regards to saving the token in the first pass given the discussion about the refresh token handling that came later on ?

I'll try to test both the new functionality as well as compatibility with unmodified client scripts.

Edit: Is there also an easy method to verify which version I currently have installed at any given time ?

[gordonb3]

Interesting. During testing I found that during at least on one particular day the returned refresh token was the same as before. I was also able to re-use that same key in multiple refreshes on that day. Actually, I'm pretty sure I had that behaviour every time I was working on my solution.

Then again, I also still do get the 3599 second expiration time, so this might be just one other appkey related difference.

[watchforstock]

Just to beware (@zxdavb in particular) that I've just merged a load of pylint changes and added some build/test capability).

[DBMandrake]

I think I might be doing something wrong here...

I updated my local git repo with 'git pull --rebase', then checked out the allow_session_reuse branch.

I then ran sudo python setup.py build and sudo python setup.py install:

pi@pi1monitor:~/Github/evohome-client $ sudo python setup.py build
running build
running build_py
copying evohomeclient/__init__.py -> build/lib.linux-armv6l-2.7/evohomeclient
copying evohomeclient/tests.py -> build/lib.linux-armv6l-2.7/evohomeclient
copying evohomeclient2/location.py -> build/lib.linux-armv6l-2.7/evohomeclient2
copying evohomeclient2/gateway.py -> build/lib.linux-armv6l-2.7/evohomeclient2
copying evohomeclient2/zone.py -> build/lib.linux-armv6l-2.7/evohomeclient2
copying evohomeclient2/controlsystem.py -> build/lib.linux-armv6l-2.7/evohomeclient2
copying evohomeclient2/__init__.py -> build/lib.linux-armv6l-2.7/evohomeclient2
copying evohomeclient2/hotwater.py -> build/lib.linux-armv6l-2.7/evohomeclient2
copying evohomeclient2/base.py -> build/lib.linux-armv6l-2.7/evohomeclient2

pi@pi1monitor:~/Github/evohome-client $ sudo python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to evohomeclient.egg-info/requires.txt
writing evohomeclient.egg-info/PKG-INFO
writing top-level names to evohomeclient.egg-info/top_level.txt
writing dependency_links to evohomeclient.egg-info/dependency_links.txt
reading manifest file 'evohomeclient.egg-info/SOURCES.txt'
writing manifest file 'evohomeclient.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-armv6l/egg
running install_lib
running build_py
creating build/bdist.linux-armv6l/egg
creating build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/location.py -> build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/gateway.py -> build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/zone.py -> build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/controlsystem.py -> build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/__init__.py -> build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/hotwater.py -> build/bdist.linux-armv6l/egg/evohomeclient2
copying build/lib.linux-armv6l-2.7/evohomeclient2/base.py -> build/bdist.linux-armv6l/egg/evohomeclient2
creating build/bdist.linux-armv6l/egg/evohomeclient
copying build/lib.linux-armv6l-2.7/evohomeclient/__init__.py -> build/bdist.linux-armv6l/egg/evohomeclient
copying build/lib.linux-armv6l-2.7/evohomeclient/tests.py -> build/bdist.linux-armv6l/egg/evohomeclient
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/location.py to location.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/gateway.py to gateway.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/zone.py to zone.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/controlsystem.py to controlsystem.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/hotwater.py to hotwater.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient2/base.py to base.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-armv6l/egg/evohomeclient/tests.py to tests.pyc
creating build/bdist.linux-armv6l/egg/EGG-INFO
copying evohomeclient.egg-info/PKG-INFO -> build/bdist.linux-armv6l/egg/EGG-INFO
copying evohomeclient.egg-info/SOURCES.txt -> build/bdist.linux-armv6l/egg/EGG-INFO
copying evohomeclient.egg-info/dependency_links.txt -> build/bdist.linux-armv6l/egg/EGG-INFO
copying evohomeclient.egg-info/requires.txt -> build/bdist.linux-armv6l/egg/EGG-INFO
copying evohomeclient.egg-info/top_level.txt -> build/bdist.linux-armv6l/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/evohomeclient-0.2.8-py2.7.egg' and adding 'build/bdist.linux-armv6l/egg' to it
removing 'build/bdist.linux-armv6l/egg' (and everything under it)
Processing evohomeclient-0.2.8-py2.7.egg
Removing /usr/local/lib/python2.7/dist-packages/evohomeclient-0.2.8-py2.7.egg
Copying evohomeclient-0.2.8-py2.7.egg to /usr/local/lib/python2.7/dist-packages
evohomeclient 0.2.8 is already the active version in easy-install.pth

Installed /usr/local/lib/python2.7/dist-packages/evohomeclient-0.2.8-py2.7.egg
Processing dependencies for evohomeclient==0.2.8
Searching for requests==2.9.1
Best match: requests 2.9.1
Adding requests 2.9.1 to easy-install.pth file

Using /usr/local/lib/python2.7/dist-packages
Finished processing dependencies for evohomeclient==0.2.8

The version number reported by pip show is the same (I presume it hasn't been bumped) but when trying debug=True for the V1 API it complains the keyword is invalid:

Traceback (most recent call last):
  File "./evohome_V1", line 6, in <module>
    client = EvohomeClient('s*******@*****.com', '**********', debug=True)
TypeError: __init__() got an unexpected keyword argument 'debug'

Have I missed something ? I'm not entirely sure whether the modified version is installed correctly or not, or how to check when the version number hasn't been changed. I tried the new V1 API debug option as a quick way of trying to determine whether the correct version is installed, but the fact that didn't work suggests that it may not have overwritten the old version.

[watchforstock]

I'm not 100% sure. I've just checked and the debug option is definitely there. I tend to check out the repo and then put a test.py file in the checkout folder. It can then import the library and it'll use the library in the current folder first. That saves any worry about versioning. Alternatively there may be a flag to force an update, but I don't know as I rarely use in the install...

[DBMandrake]

That was it. Copying my quick test script into the checked out repo and running it from there seems to have worked, confirming that build and install steps did NOT actually replace the installed version, presumably due to the version number being the same.

To test properly with my actual graphing script I'll need to find out how to force an overwrite of the "same" version of the installed module.

Here is what I'm getting from the V1 API debug option when I'm being rated limited BTW:

pi@pi1monitor:~/Github/evohome-client $ ./evohome_V1 INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com send: 'POST /WebAPI/api/Session HTTP/1.1\r\nHost: tccna.honeywell.com\r\nContent-Length: 119\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.9.1\r\nConnection: keep-alive\r\ncontent-type: application/json\r\n\r\n{"Username": "s*******@****.com", "Password": "*******", "ApplicationId": "91db1612-73fd-4500-91b2-e63b069b185c"}' reply: 'HTTP/1.1 429 Too Many Requests\r\n' header: Cache-Control: no-cache header: Pragma: no-cache header: Content-Type: application/json; charset=utf-8 header: Expires: -1 header: Server: Microsoft-IIS/8.5 header: Server: Web1 header: Date: Mon, 25 Feb 2019 23:13:49 GMT header: Content-Length: 123 header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc0245525d5f4f58455e445a4a42378b;expires=Mon, 25-Feb-2019 23:15:49 GMT;path=/;secure;httponly DEBUG:requests.packages.urllib3.connectionpool:"POST /WebAPI/api/Session HTTP/1.1" 429 123 Traceback (most recent call last): File "./evohome_V1", line 12, in <module> for device in client.temperatures(): File "/home/pi/Github/evohome-client/evohomeclient/__init__.py", line 135, in temperatures self._populate_full_data(force_refresh) File "/home/pi/Github/evohome-client/evohomeclient/__init__.py", line 71, in _populate_full_data self._populate_user_info() File "/home/pi/Github/evohome-client/evohomeclient/__init__.py", line 121, in _populate_user_info response.raise_for_status() File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 840, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://tccna.honeywell.com/WebAPI/api/Session

[DBMandrake]

Found reference to a --force option for python setup.py install but it didn't seem to work for me.

I then tried sudo pip uninstall evohomeclient - this said it had uninstalled it but my test script (not in the git repo) was still working! Ran it again without sudo and it attempted a second time to remove it although claims it failed. After that my test script failed to find the library as expected. I then installed the new version and it seems to be correctly in place now as the new debug option is working outside of the git repo directory.

So it seems that to be certain you need to uninstall the old version, run a test script to verify 100% that it is really gone and then reinstall it. It's possible that I may have installed the previous versions both as a normal user and root via sudo. (assuming python has both system wide and per user library paths....)

[DBMandrake]

I updated the installed version of the library (so that my evohome-munin installation starts using it) and it has run overnight with unmodified client scripts with no obvious regressions - I still see the same intermittent failures, no better or worse than before. So regression-wise things look OK so far.

Starting to look at coding specifically to take advantage of the new mechanisms so I'm looking for a little clarification on the example code earlier in this thread.

Please see examples below. In the v1 client, a call needs to have been made before the user_data variable is populated. In the v2 client, it will have authenticated as part of the constructor.

Can you clarify what you mean by "a call" ? Do you mean calling c = EvohomeClient(username, password, debug=True, user_data=user_data) alone won't cause user_data to be populated, and that it will only be populated when calling any of the subsequent methods ? Does this mean the initial EvohomeClient() call doesn't actually initiate a server connection and it is deferred until an actual request is made ?

For the v1 client, I can't see any information coming back from the server as to the length of the session token validity. As such, at the moment the library doesn't know when it's become invalid and so doesn't automatically reauthenticate.

So from a client script perspective what does this mean ? That I should take note of the time user_data is populated and only try to use it if it is within our (assumed) 15 minute timeout period, otherwise call the function only with username and password ?

What happens if I supply a username and password and user_data but the session is no longer valid, are you saying the call will just fail and need to be re-tried manually without submitting user_data ?

For the v2 client, the library will automatically refresh credentials when they timeout (either during calls for the same client or when credentials that have been stored have expired).

So for the V2 client you're suggesting to always submit the previous access token (if one was saved) along with the username and password and the library will take care of testing the token and falling back to username/password if it doesn't work ? Or should I still be checking the token expiry myself first and only submitting the token if I'm sure it has not expired yet ?

[zxdavb]

it has run overnight with unmodified client scripts with no obvious regressions

Woot! Same here (on HomeAssistant with the custom component version of the evhome plug-in)

Can you clarify what you mean by "a call" ?

Evoking the client via (say) c = EvohomeClient(username, password, debug=True, user_data=user_data) will not cause user_data to be populated:

class EvohomeClient:
    def __init__(self, username, password):
        self.username = username
        self.password = password
        self.user_data = None
        self.full_data = None
        self.gateway_data = None
        self.reader = codecs.getdecoder("utf-8")

By a call he means:

temps = c.temperatures()

This is not the case for EvohomeClient2().

[DBMandrake]

Got it.

[zxdavb]

For the v2 client, there are two things worth understanding.

a) Whenever a call is made to Honeywell's servers (buy using an EvohomeClient2 method), the auth_token, if provided, will be used rather than generating a new one.

A new authorization token will automatically be requested if the old one is invalid, and whenever it has expired. For this to happen, there needs to be a valid set of credentials identifying the user's account..

The code can handle an expired/null token, but not all (contrived) 'invalid' tokens (e.g. the the token = "hello", and it expires a month into the future) - such tokens may result in unpredictable behaviour.

b) Whenever a new access token needed (i.e. re-authentication), if a refresh_token is provided, it will be used in preference to any user credentials (I believe this doesn't count against an API limit). If that fails for any reason, the user credentials will be used instead (this counts).

Theory says (I haven't tested it) that you can supply a refresh token, and no username/password, and that would be OK.

Note the refresh token is refreshed every time a new access token is created.

[DBMandrake]

Not sure if I've made a mistake here or not, but during my testing it seems the V2 API is calling the OAuth URL (and hence still suffering throttling) regardless of whether I pass it a valid token or not.

I'm embarrassed to post my quick hacky test script as it will show what a terrible python programmer I am, but here it is anyway:

#!/usr/bin/python

import time
import json
import io
import datetime

from evohomeclient2 import EvohomeClient

try:
        f = io.open('V2_access_token', "rb")

        list = json.load(f)
        access_token = list[0]
        access_token_expires = datetime.datetime.strptime(list[1], "%Y-%m-%d %H:%M:%S.%f")

        f.close()

        print 'DEBUG: SAVED access_token: ' + str(access_token)
        print 'DEBUG: SAVED access_token_expires: ' + str(access_token_expires)
except:
        access_token = None
        access_token_expires = None

client = EvohomeClient('******@*****.com', '**********', access_token=access_token, access_token_expires=access_token_expires, debug=True)

print 'DEBUG: RECEIVED access_token: ' + str(client.access_token)
print 'DEBUG: RECEIVED access_token_expires: ' + str(client.access_token_expires)

for device in client.temperatures():
    print device

f = io.open('V2_access_token', "wb")

list = [ client.access_token, str(client.access_token_expires) ]

json.dump(list, f)
f.close()

The idea is at the end after printing zone temperatures I try to save the token and expiry time in a json file then reload it and use it next time. What I'm finding though is that on the second invocation where the token file exists my debug lines near the beginning of the script are printing what looks like a valid token and expiry time retrieved from the json file, however the library still seems to be calling OAuth and (sometimes) getting throttled regardless as shown in this log:

pi@pi1monitor:~ $ ./evohome_V2_token
DEBUG: SAVED access_token: Qun1QjW1DoNok2F_fPO0Zq4vRMEbSkpZOo-7tp9-R1rRJ65LmBtLlGZTmiUPf3VNQAq6YxdsX7IT3CHN5C2mDYiWTOEiRa16rWDaKPVshH0SZMsD3sefu5Ic3Whl7vazgarG47l9dKkJ7s8A7YIkEd-ZrBsaruXkNdv7sPSY7OT_ldyYo2_U55Zzp22B597p2QMJLa5mwy3Kje4bkMY5gVEcYRaOzxHw4R6lOemeljgYrti8m81RP474XliyVbvz2cU-lB2S0jQrscFRY_VKTiRdnq9uNfQAqSBMnWnz0ctoezFyaD--aMGq_qOVETGBupyERi_Yc-FMFhIC58vVUdkP6xZXlx4il0GbLzglSN59JQxCxt0Kc91j3POM8Z3ItRP4FUF7gE0WDbYW1pBjn_5koUjCG_e6DZ9M_NLDN1iaIUqZLaZi0qKuFq-OvsjW7j7AObdQYxyWgY3Dro9gI5NRIO4vIaKyy_o_U4S704lfdNE2w3hYhdR91z_I5d-ylXO_VyE8dyrFChtF56IAwgeBB853AiQmFq3BnmSC4BHBsDzn
DEBUG: SAVED access_token_expires: 2019-02-26 15:50:00.122586
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'POST /Auth/OAuth/Token HTTP/1.1\r\nHost: tccna.honeywell.com\r\nContent-Length: 304\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml\r\nUser-Agent: python-requests/2.9.1\r\nConnection: keep-alive\r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: Basic NGEyMzEwODktZDJiNi00MWJkLWE1ZWItMTZhMGE0MjJiOTk5OjFhMTVjZGI4LTQyZGUtNDA3Yi1hZGQwLTA1OWY5MmM1MzBjYg==\r\n\r\nUsername=********%40******.com&Host=rs.alarmnet.com%2F&Password=**********&Pragma=no-cache&Cache-Control=no-store+no-cache&scope=EMEA-V1-Basic+EMEA-V1-Anonymous+EMEA-V1-Get-Current-User-Account&grant_type=password&Content-Type=application%2Fx-www-form-urlencoded%3B+charset%3Dutf-8&Connection=Keep-Alive'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json;charset=UTF-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: Set-Cookie: thlang=en-US; expires=Tue, 26-Feb-2069 15:20:17 GMT; path=/
header: Server: Web1
header: Date: Tue, 26 Feb 2019 15:20:16 GMT
header: Content-Length: 34
header: Set-Cookie: NSC_UDDOB-TTM-WT=ffffffff090ecc0545525d5f4f58455e445a4a42378b;expires=Tue, 26-Feb-2019 15:22:17 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"POST /Auth/OAuth/Token HTTP/1.1" 429 34
Traceback (most recent call last):
  File "./evohome_V2_token", line 26, in <module>
    client = EvohomeClient('*******@*****.com', '*********', access_token=access_token, access_token_expires=access_token_expires, debug=True)
  File "build/bdist.linux-armv6l/egg/evohomeclient2/__init__.py", line 50, in __init__
  File "build/bdist.linux-armv6l/egg/evohomeclient2/__init__.py", line 197, in _login
  File "build/bdist.linux-armv6l/egg/evohomeclient2/__init__.py", line 189, in _basic_login
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://tccna.honeywell.com/Auth/OAuth/Token

Any thoughts ? I’m not entirely sure whether I am correctly saving and restoring the token and datetime object but I think I am.

[zxdavb]

Review https://github.com/watchforstock/evohome-client/wiki/Access_tokens-(v2-client)

Once your access token expires, your code must re-authenticate via username/password.

Save and restore the refresh_token.

[DBMandrake]

Ah, looks like I am following outdated advice from post #6. I must have missed it in the discussion but that wiki page is the first example code I've seen where I actually have to save and restore 3 items. I'll give that a try.

[zxdavb]

Yes, things can move quickly here...

[DBMandrake]

Looks like my copy of the library doesn't have a refresh_token in the Evohomeclient() call. So I need to update my installation again ?

Edit: Nope. I'm up to date on the allow_session_reuse branch but refresh_token is not defined. What gives ? :)

[zxdavb]

Yes, things can move quickly here...

dbonnes@vm-builder:~/evohome-client/evohomeclient2$ cat __init__.py | grep __init__ | head -n1

    def __init__(self, username, password, debug=False, refresh_token=None,

mebbe git fetch --all?

[DBMandrake]

    def __init__(self, username, password, debug=False, refresh_token=None,
        super(EvohomeClient, self).__init__(debug)

I'm testing directly in the git repo to make sure my script is using the checked out version, and I checked the above file and it shows the definition for refresh_token=None, and yet:

pi@pi1monitor:~/Github/evohome-client $ git pull --rebase
Current branch allow_session_reuse is up to date.
pi@pi1monitor:~/Github/evohome-client $ git branch
* allow_session_reuse
  master
pi@pi1monitor:~/Github/evohome-client $ ./evohome_V2_token
Traceback (most recent call last):
  File "./evohome_V2_token", line 28, in <module>
    client = EvohomeClient('*********@*****.com', '**********', refresh_token=refresh_token, access_token=access_token, access_token_expires=access_token_expires, debug=True)
NameError: name 'refresh_token' is not defined

?