Client authentication triggering server rate limiting

[DBMandrake]

In recent months Honeywell have made substantial changes to their API servers, one of these is that new, aggressive rate limiting has been applied for client connection authentication attempts.

This means that even polling the servers once every 5 minutes (performing a full username/password authentication) for the purposes of graphing zone temperatures is now intermittently triggering rate limiting which locks out the client for as much as 10 minutes or more.

Investigation by a few people on the Automated home forum including 'gordonb3' have discovered that it's the OAuth authentication attempts that are being rate limited and rejected, not the actual use of the API calls once the client has a client access token, which lasts (at least for now on the V2 API) for 30 minutes from the time of last use.

So for their own graphing systems (which don't use this python client library) they have adapted to the new rate limit restrictions by locally caching the client access token and keeping track of it's expiry time and only performing a full username/password re-authentication when absolutely needed.

From what I can see while this library caches the client access token within a given script instance it does not have any means to cache and reuse the token among multiple consecutive script instances, (eg on disk) instead performing a full username/password authentication when each new script instance calls the EvohomeClient() method.

I use the evohome-client library with evohome-munin which is a Munin plugin. The architecture of munin plugins is that each graph for each zone is generated by calling a separate instance of the script sequentially one at a time.

Evohome-munin already implements disk caching of the returned zone data so that only the first instance of the script actually calls EvohomeClient() and the following instances read the zone data from disk cache and therefore only one authentication attempt is made per 5 minutes, but even this is triggering the rate limiting at the servers.

To fully solve this problem evohome-client would need to cache the client access token for a given username/password pair to disk along with a token expiry time, and then when another instance of a script calls EvohomeClient() check to see if a recently used still valid access token already exists and use that directly, bypassing the username/password OAuth steps and only attempting OAuth authentication if the token is expired or is no longer working.

Is there any possibility that cross instance caching of the client access token can be added to this library ? Without it, it's semi-unusable now for any non-persistent client script, thanks to the rate limiting Honeywell have applied, not just for 5 minute polling for graphing, but for any application that may want to make arbitrarily timed connections in response to user queries or actions. I started to look at this myself but quickly realised my python skills are not up to the task of attempting this sort of rewrite of the library.

Some discussion about the details of this problem can be found on the following pages of the Automated Heating Forum thread:

https://www.automatedhome.co.uk/vbulletin/showthread.php?5723-Evohome-app-broken/page2

Posts by gordonb3 are particularly helpful. Both V1 and V2 API authentication are affected by rate limiting and it appears that authentication attempts to both API's may be counted together towards total rate limiting and that once you are rate limited, authentication attempts on both API's fail, so sharing the cached client access token between V1 and V2 API's may be necessary to fully support clients that use both API's. (As the evohome-munin script does)

[zxdavb]

Is this amount of repetition of the new debug mode enabled/disabled debug message to be expected ?

I'm on it!

[zxdavb]

@DBMandrake This is a consequence of PR #74

What platform are you using? How have you enabled debug/logging mode?

[DBMandrake]

I actually do doubt that they might ever change the timeout on the v1 API sessions though, because this would also affect their original phone app (and possibly RFG100 and wifi controller connections as well).

I'm pretty sure neither the RFG100 or Wifi controller use either of these two public API's to communicate with the Honeywell servers, but another API dedicated to upstream communication from the thermostat devices.

Almost invariably when either or both of the API's has been down, the controller itself remains connected and able to communicate without issues.

Also neither of the API's has the full functionality required such as pushing firmware updates to the Wifi model, or reporting heat demand.

I have it on good authority that even though the public API's that we use don't report heat demand from zones, the heat demand information is in fact being sent up to Honeywell from the controller and stored as their engineers have access to that heat demand (and other) information if you grant them rights. (Honeywell currently have full monitoring rights on my system dating back to troubleshooting a previous issue and it shows up in the TCC portal with the option for me to remove their rights)

[DBMandrake]

@DBMandrake This is a consequence of PR #74

What platform are you using? How have you enabled debug/logging mode?

Raspberry Pi 1. I have debug mode on, contrary to what those messages say. There is one other instance of this message right at the beginning which says debugging is on. Here is a full dump:

pi@pi1monitor:~ $ ./evohome_V2_token
DEBUG:evohomeclient2.base:__init__(): Debug mode is explicitly enabled.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'POST /Auth/OAuth/Token HTTP/1.1\r\nHost: tccna.honeywell.com\r\nContent-Length: 304\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml\r\nUser-Agent: python-requests/2.9.1\r\nConnection: keep-alive\r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: Basic NGEyMzEwODktZDJiNi00MWJkLWE1ZWItMTZhMGE0MjJiOTk5OjFhMTVjZGI4LTQyZGUtNDA3Yi1hZGQwLTA1OWY5MmM1MzBjYg==\r\n\r\nUsername=*******%40*****.com&Password=*********&Connection=Keep-Alive&Host=rs.alarmnet.com%2F&Pragma=no-cache&Cache-Control=no-store+no-cache&scope=EMEA-V1-Basic+EMEA-V1-Anonymous+EMEA-V1-Get-Current-User-Account&grant_type=password&Content-Type=application%2Fx-www-form-urlencoded%3B+charset%3Dutf-8'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json;charset=UTF-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: Set-Cookie: thlang=en-US; expires=Thu, 28-Feb-2069 15:54:42 GMT; path=/
header: Server: Web1
header: Date: Thu, 28 Feb 2019 15:54:42 GMT
header: Content-Length: 1240
header: Set-Cookie: NSC_UDDOB-TTM-WT=ffffffff090ecc1d45525d5f4f58455e445a4a42378b;expires=Thu, 28-Feb-2019 15:56:43 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"POST /Auth/OAuth/Token HTTP/1.1" 200 1240
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'GET /WebAPI/emea/api/v1/userAccount HTTP/1.1\r\nHost: tccna.honeywell.com\r\nConnection: keep-alive\r\nAccept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml\r\nAccept-Encoding: gzip, deflate\r\nAuthorization: bearer wB6g6wnMWWo72wOjo8ryhH21Lo9yedOiDOJtmoOR-rTuQXY86bK9nOdYgc-n7m4MOY6VZk1nt5iDycziW6UdsqDZ5YgSHC7UyPIrk7t5wpVo7f_87wcJOsv_yJJd510s8VSZ4BHWkIHbq5WbYgySKQJeZp6NFF2I0Mnd_x6xfPM_ILMrDRtBYnlyh3rGBxJUC2MJG97mZ9OeBFUtYFCU5W4O6FT9Ur_EIpawaX096841--LtxGjVu6IQ0hxBpSn3LJpeDSciMo_U9DOWnQVr5i-a5WOqrF4j6u3xePYvvhjd1z_zRTKqRDiVer92QK5Fh4ozk8anZ9AEPJ-DM-yhPS7ykVlsPapit7ZshqZfN_1V27JrXpIG9INO0sqIJuR6SP8EaoM6_rEUrtrAMBf6hVUQf2hQGgZ3XuOYcOaP5YA98s4zbjz8tknfWgLI8xFIM4E-8U0VB7VCBjvv_tEBs6sqZoVLly21nkl3TAa1lrRBIAyh65Z7E5gnb4zcVKXESa3gWaDi2FD4KM8PNqPypFXCmHxQmcUe3qPqLAnRd1WkwrAi\r\nUser-Agent: python-requests/2.9.1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json; charset=utf-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: Server: Web1
header: Date: Thu, 28 Feb 2019 15:54:43 GMT
header: Content-Length: 261
header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc0345525d5f4f58455e445a4a42378b;expires=Thu, 28-Feb-2019 15:56:44 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"GET /WebAPI/emea/api/v1/userAccount HTTP/1.1" 200 261
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'GET /WebAPI/emea/api/v1/location/installationInfo?userId=1108128&includeTemperatureControlSystems=True HTTP/1.1\r\nHost: tccna.honeywell.com\r\nConnection: keep-alive\r\nAccept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml\r\nAccept-Encoding: gzip, deflate\r\nAuthorization: bearer wB6g6wnMWWo72wOjo8ryhH21Lo9yedOiDOJtmoOR-rTuQXY86bK9nOdYgc-n7m4MOY6VZk1nt5iDycziW6UdsqDZ5YgSHC7UyPIrk7t5wpVo7f_87wcJOsv_yJJd510s8VSZ4BHWkIHbq5WbYgySKQJeZp6NFF2I0Mnd_x6xfPM_ILMrDRtBYnlyh3rGBxJUC2MJG97mZ9OeBFUtYFCU5W4O6FT9Ur_EIpawaX096841--LtxGjVu6IQ0hxBpSn3LJpeDSciMo_U9DOWnQVr5i-a5WOqrF4j6u3xePYvvhjd1z_zRTKqRDiVer92QK5Fh4ozk8anZ9AEPJ-DM-yhPS7ykVlsPapit7ZshqZfN_1V27JrXpIG9INO0sqIJuR6SP8EaoM6_rEUrtrAMBf6hVUQf2hQGgZ3XuOYcOaP5YA98s4zbjz8tknfWgLI8xFIM4E-8U0VB7VCBjvv_tEBs6sqZoVLly21nkl3TAa1lrRBIAyh65Z7E5gnb4zcVKXESa3gWaDi2FD4KM8PNqPypFXCmHxQmcUe3qPqLAnRd1WkwrAi\r\nUser-Agent: python-requests/2.9.1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json; charset=utf-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: Server: Web1
header: Date: Thu, 28 Feb 2019 15:54:44 GMT
header: Content-Length: 11808
header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc1a45525d5f4f58455e445a4a42378b;expires=Thu, 28-Feb-2019 15:56:45 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"GET /WebAPI/emea/api/v1/location/installationInfo?userId=*******&includeTemperatureControlSystems=True HTTP/1.1" 200 11808
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
DEBUG:evohomeclient2.base:__init__(): Debug mode was not explicitly enabled.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'GET /WebAPI/emea/api/v1/location/******/status?includeTemperatureControlSystems=True HTTP/1.1\r\nHost: tccna.honeywell.com\r\nConnection: keep-alive\r\nAccept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml\r\nAccept-Encoding: gzip, deflate\r\nAuthorization: bearer wB6g6wnMWWo72wOjo8ryhH21Lo9yedOiDOJtmoOR-rTuQXY86bK9nOdYgc-n7m4MOY6VZk1nt5iDycziW6UdsqDZ5YgSHC7UyPIrk7t5wpVo7f_87wcJOsv_yJJd510s8VSZ4BHWkIHbq5WbYgySKQJeZp6NFF2I0Mnd_x6xfPM_ILMrDRtBYnlyh3rGBxJUC2MJG97mZ9OeBFUtYFCU5W4O6FT9Ur_EIpawaX096841--LtxGjVu6IQ0hxBpSn3LJpeDSciMo_U9DOWnQVr5i-a5WOqrF4j6u3xePYvvhjd1z_zRTKqRDiVer92QK5Fh4ozk8anZ9AEPJ-DM-yhPS7ykVlsPapit7ZshqZfN_1V27JrXpIG9INO0sqIJuR6SP8EaoM6_rEUrtrAMBf6hVUQf2hQGgZ3XuOYcOaP5YA98s4zbjz8tknfWgLI8xFIM4E-8U0VB7VCBjvv_tEBs6sqZoVLly21nkl3TAa1lrRBIAyh65Z7E5gnb4zcVKXESa3gWaDi2FD4KM8PNqPypFXCmHxQmcUe3qPqLAnRd1WkwrAi\r\nUser-Agent: python-requests/2.9.1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json; charset=utf-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: Server: Web1
header: Date: Thu, 28 Feb 2019 15:54:45 GMT
header: Content-Length: 4104
header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc0045525d5f4f58455e445a4a42378b;expires=Thu, 28-Feb-2019 15:56:46 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"GET /WebAPI/emea/api/v1/location/*******/status?includeTemperatureControlSystems=True HTTP/1.1" 200 4104

[zxdavb]

I have debug mode on, contrary to what those messages say.

Yes, a confusing message. What is means is that you didn't:

c = EvohomeClient2(blah, blah, debug=True)

Or did you?

[zxdavb]

OK, I need to ask @watchforstock something in PR #75

[DBMandrake]

I have debug mode on, contrary to what those messages say.

Yes, a confusing message. What is means is that you didn't:

c = EvohomeClient2(blah, blah, debug=True)

Or did you?

I did. I called:

client = EvohomeClient('**********', '**********', refresh_token=refresh_token, access_token=access_token, access_token_expires=access_token_expires, debug=True)

Debug mode is enabled otherwise it wouldn't have shown all the debug text. :)

By the way, I only see the repeated debug lines in the V2 API calls.

[zxdavb]

@DBMandrake I've got my head around it: there's a thing in the code, where it isn't as you'd expect - this is the cause of the issue.

The 'bug' was exposed via (but, I'd argue, not caused by) PR #74

I am happy to push a change, but just want to check with @watchforstock first.

[gordonb3]

I'm pretty sure neither the RFG100 or Wifi controller use either of these two public API's to communicate with the Honeywell servers, but another API dedicated to upstream communication from the thermostat devices.

Obviously true. You are aware though that both user APIs are hosted on the same public IP? I also happen to know that the user API for the Lyric range is hosted on that same IP as well and I'm assuming (or rather: hoping) there is some kind of redundancy on this hosting. As such I see no reason why this same IP could not also serve controller communications.

Which brings me to the point that I don't think that the session ID in the v1 API is controlled by the API itself but is in fact part of the active page server (probably tomcat).

Almost invariably when either or both of the API's has been down, the controller itself remains connected and able to communicate without issues.

Actually, in most cases I'm able to connect to the portal IP just fine. The problem as I see it is usually with the back-end being unresponsive or throwing errors. And in fact I have had occasions where the web portal appeared to work correctly but the commands never reached the home system. Mostly notable by the portal reporting override end times that are in the past, which seems to indicate that the home system is supposed to reset those states.

[watchforstock]

There have been a lot of changes on this branch, covering more than just the session re-use. Unless any one objects, I'm inclined to merge this branch into master. Before cutting a new release, we can continue to test and fix issues on master, but it allows other changes to be tracked separately.

Thanks again for everyone's input and sorry that I'm not able always able to keep up with the conversation!

[gordonb3]

I did. I called:

client = EvohomeClient('**********', '**********', refresh_token=refresh_token, access_token=access_token, access_token_expires=access_token_expires, debug=True)

That actually does not look right. Seems you copied the method's definition and simply used what are in fact default values as internal variable names., which amazingly appears to work for the token values but causes the warning on the debug value.

You should not state debug=True in your call but either not include that parameter at all to use the specified default or write True or False

[DBMandrake]

I did. I called:

client = EvohomeClient('**********', '**********', refresh_token=refresh_token, access_token=access_token, access_token_expires=access_token_expires, debug=True)

That actually does not look right. Seems you copied the method's definition and simply used what are in fact default values as internal variable names., which amazingly appears to work for the token values but causes the warning on the debug value. You should not state debug=True in your call but either not include that parameter at all to use the specified default or write True or False

Err... I didn't just copy the method definition - I'm using the example client call that @watchforstock provided very early in this discussion around post 6.

Python allows for both positional arguments and named arguments in function calls (not sure if that's the right terminology) as well as a mixture of both.

So specifying debug=True as an argument is perfectly valid syntax and is identical to just specifying True in the correct positional argument, the difference being that you can put debug=True anywhere in the list of arguments, and also use it even if you're skipping over several parameters, (For example leaving out the new access_client variable etc) which is handy when expanding a function beyond the original arguments.

So, nothing wrong with that. In fact if you check the docs over at https://evohome-client.readthedocs.io/en/latest/api2/index.html you'll see that debug=True is specifically stated as the recommended way to enable debugging, and it's the way I've always done it.

Likewise there is nothing wrong with refresh_token=refresh_token etc in the EvohomeClient() call, all that is saying is pass the refresh_token variable as an argument to the named argument refresh_token.

It works because earlier in the script (which I've already posted further up) I load the stored tokens from a json file back into variables called refresh_token etc and then pass them. So it works by design not by accident. :)

[DBMandrake]

Obviously true. You are aware though that both user APIs are hosted on the same public IP?

Yes I actually mentioned earlier that both V1 and V2 API's were hosted by the same server at the same hostname when I was commenting that the authentication rate limiting seems to be shared between them.

I also happen to know that the user API for the Lyric range is hosted on that same IP as well and I'm assuming (or rather: hoping) there is some kind of redundancy on this hosting. As such I see no reason why this same IP could not also serve controller communications.

It's quite possible that the same physical server provides controller communications. All I said was that it's not using the same API's as the client apps. Because when those API's have been down the controller stays connected and in fact my conradconnect graphs continued to update through periods where my own evohome-munin graphs stopped working completely due to the V1 API being down and iPhone app due to the V2 API being down. (I suspect official Honeywell partners like Alexa, IFTTT, conradconnect etc have yet another API or at least their own individual API keys)

Which brings me to the point that I don't think that the session ID in the v1 API is controlled by the API itself but is in fact part of the active page server (probably tomcat).

Could well be.

Actually, in most cases I'm able to connect to the portal IP just fine. The problem as I see it is usually with the back-end being unresponsive or throwing errors. And in fact I have had occasions where the web portal appeared to work correctly but the commands never reached the home system. Mostly notable by the portal reporting override end times that are in the past, which seems to indicate that the home system is supposed to reset those states.

When I say the "server is down" you're taking that a bit to literally. I don't mean it's crashed or lost its network connection etc, just that it's not responding correctly. As you say the front end web service usually seems to be up and it seems like its back end database problems causing the issues, especially when you get situations where client app actions are acknowledged to the app but not correctly forwarded on to the controller.

[DBMandrake]

There have been a lot of changes on this branch, covering more than just the session re-use. Unless any one objects, I'm inclined to merge this branch into master. Before cutting a new release, we can continue to test and fix issues on master, but it allows other changes to be tracked separately.

Thanks again for everyone's input and sorry that I'm not able always able to keep up with the conversation!

Yes I wondered about the unrelated changes sneaking into this branch. :)

The code seems good so far but it would be nice to get the V1 automatic re-authentication in and tested before a major version number bump.

[watchforstock]

I've just pushed a change to the branch which should auto re-authenticate once if it gets a 401 error. I tried it, and immediately got a 429, but I don't know whether that's because other systems were accessing the api at the same time. Would welcome others trying it and seeing if you get the same result.

[DBMandrake]

I've just pushed a change to the branch which should auto re-authenticate once if it gets a 401 error. I tried it, and immediately got a 429, but I don't know whether that's because other systems were accessing the api at the same time. Would welcome others trying it and seeing if you get the same result.

Gave it a quick try this morning but although it looks like it might be trying to re-authenticate (I'm not sure) the final request still fails with HTTP 401 Unauthorised. Perhaps it is re-authenticating but forgetting to update the session details to the new session ?

pi@pi1monitor:~ $ ./evohome_V1_token
<type 'dict'>
DEBUG: SAVED user_data: {u'sessionId': u'05018FC0-D766-40E8-B64D-9171E121D38D', u'userInfo': {u'username': u'********', u'city': u'********', u'isActivated': True, u'firstname': u'********', u'lastname': u'********', u'userID': ********, u'zipcode': u'********', u'telephone': u'', u'deviceCount': 0, u'streetAddress': u'********', u'securityQuestion3': u'NotUsed', u'securityQuestion2': u'NotUsed', u'securityQuestion1': u'NotUsed', u'country': u'GB', u'userLanguage': u'en-GB', u'tenantID': 5, u'latestEulaAccepted': False}}
DEBUG: SAVED last_access_date: 2019-02-28 16:24:36.601056
DEBUG:evohomeclient:__init__(): Debug mode is explicitly enabled.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'GET /WebAPI/api/locations?userId=********&allData=True HTTP/1.1\r\nHost: tccna.honeywell.com\r\nContent-Length: 2\r\nAccept-Encoding: gzip, deflate\r\nsessionId: 05018FC0-D766-40E8-B64D-9171E121D38D\r\nAccept: */*\r\nUser-Agent: python-requests/2.9.1\r\nConnection: keep-alive\r\n\r\n{}'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json; charset=utf-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: WWW-Authenticate: Bearer
header: Server: Web1
header: Date: Fri, 01 Mar 2019 06:09:20 GMT
header: Content-Length: 74
header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc1c45525d5f4f58455e445a4a42378b;expires=Fri, 01-Mar-2019 06:11:21 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"GET /WebAPI/api/locations?userId=********&allData=True HTTP/1.1" 401 74
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'POST /WebAPI/api/Session HTTP/1.1\r\nHost: tccna.honeywell.com\r\nContent-Length: 119\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.9.1\r\nConnection: keep-alive\r\ncontent-type: application/json\r\n\r\n{"Username": "********", "Password": "********", "ApplicationId": "91db1612-73fd-4500-91b2-e63b069b185c"}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json; charset=utf-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: Server: Web1
header: Date: Fri, 01 Mar 2019 06:09:22 GMT
header: Content-Length: 584
header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc0245525d5f4f58455e445a4a42378b;expires=Fri, 01-Mar-2019 06:11:22 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"POST /WebAPI/api/Session HTTP/1.1" 200 584
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): tccna.honeywell.com
send: 'GET /WebAPI/api/locations?userId=********&allData=True HTTP/1.1\r\nHost: tccna.honeywell.com\r\nContent-Length: 2\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.9.1\r\nConnection: keep-alive\r\ncontent-type: application/json\r\n\r\n{}'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Cache-Control: no-cache
header: Pragma: no-cache
header: Content-Type: application/json; charset=utf-8
header: Expires: -1
header: Server: Microsoft-IIS/8.5
header: WWW-Authenticate: Bearer
header: Server: Web1
header: Date: Fri, 01 Mar 2019 06:09:23 GMT
header: Content-Length: 74
header: Set-Cookie: NSC_UDDOB-XfcBqj-TTM-WT=ffffffff090ecc1c45525d5f4f58455e445a4a42378b;expires=Fri, 01-Mar-2019 06:11:23 GMT;path=/;secure;httponly
DEBUG:requests.packages.urllib3.connectionpool:"GET /WebAPI/api/locations?userId=********&allData=True HTTP/1.1" 401 74
Traceback (most recent call last):
  File "./evohome_V1_token", line 29, in <module>
    for device in client.temperatures():
  File "build/bdist.linux-armv6l/egg/evohomeclient/__init__.py", line 133, in temperatures
  File "build/bdist.linux-armv6l/egg/evohomeclient/__init__.py", line 83, in _populate_full_data
  File "build/bdist.linux-armv6l/egg/evohomeclient/__init__.py", line 189, in _do_request
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://tccna.honeywell.com/WebAPI/api/locations?userId=********&allData=True

[gordonb3]

Well, well. Never too old to learn. Not that it will make me a fan of python, you know why, but that kind of flexibility is pretty cool.

[DBMandrake]

Well, well. Never too old to learn. Not that it will make me a fan of python, you know why, but that kind of flexibility is pretty cool.

Don't worry about it. Even though I've tinkered with python for a few years now I use it so infrequently that I still consider myself a beginner, so by default I assume everyone around me knows more about it than I do! :) But the named arguments for function calls is one thing I'm quite familiar with and like about Python.

Not only does it make it easier to extend function calls with optional arguments later on without worrying about counting and ordering potentially empty in between arguments, I think it makes the code more readable as you can see what each argument is based on its name without referring to the referenced function or documentation. I think some Python programmers like to use argument naming as a matter of course for this reason even where it's not necessary...

[DBMandrake]

I've just pushed a change to the branch which should auto re-authenticate once if it gets a 401 error. I tried it, and immediately got a 429, but I don't know whether that's because other systems were accessing the api at the same time. Would welcome others trying it and seeing if you get the same result.

One other thought I have about this - are the changes that you're working on here also going to help with persistent scripts ?

Did the library previously ever try to automatically re-authenticate long lived V1 sessions where the session id timed out during the execution of the client ?

For example if I wrote a test script that called EvohomeClient(), queried zone temperatures, slept for 20 minutes without exiting or performing further queries, then queried zone temperatures again, would it have failed with the old library due to the session id expiring, and would you expect that to trigger a re-authentication and succeed with these new changes ? (Once the bugs are ironed out of course)

[watchforstock]

One other thought I have about this - are the changes that you're working on here also going to help with persistent scripts ?

Once it's working (!), yes it should. Previously there was no automatic retry capability so the script would have errored after the session token had expired

[DBMandrake]

One other thought I have about this - are the changes that you're working on here also going to help with persistent scripts ?

Once it's working (!), yes it should. Previously there was no automatic retry capability so the script would have errored after the session token had expired

Cool. That's what I thought.

Looking closer at the debug log I posted this morning I see the nature of the problem. The first attempt to use the old session id to get zone data GET /WebAPI/api/locations?userId=........ is passing the session id as you'd expect, then the authentication request is succeeding, but the second call to GET /WebAPI/api/locations?userId=........ to get the zone data is not posting any session id at all - not even an empty one...

Hopefully that's a nice obvious mistake to find. :)

[watchforstock]

One other thought I have about this - are the changes that you're working on here also going to help with persistent scripts ?

Once it's working (!), yes it should. Previously there was no automatic retry capability so the script would have errored after the session token had expired

Cool. That's what I thought.

Looking closer at the debug log I posted this morning I see the nature of the problem. The first attempt to use the old session id to get zone data GET /WebAPI/api/locations?userId=........ is passing the session id as you'd expect, then the authentication request is succeeding, but the second call to get the zone data is not posting any session id at all - not even an empty one...

Hopefully that's a nice obvious mistake to find. :)

That'll teach me to make "easy" changes when I'm tired! I'll take a look...

[DBMandrake]

A bit off topic here, but the only reason evohome-munin polls both V1 and V2 API's from the same script (thus further angering the rate limiting gods) is because the V2 API rounds temperature readings unacceptably (to the nearest half degree, and then biases it towards the set point half a degree as well, like the controller itself) while the V1 API whilst giving high resolution un-rounded temperatures doesn't have any way to report hot water on/off status.

I can get current hot water temperature from the V1 API, however am I correct that there is no way to retrieve current hot water on/off status from the V1 API ? If I could somehow dig that information out of the V1 API I could eliminate my use of the V2 API completely and simplify the script a lot in the process especially when I would only have user_data to save instead of also having to save access_token, refresh_token and access_token_expiry, as well as using two sets of API calls.

[watchforstock]

A bit off topic here, but the only reason evohome-munin polls both V1 and V2 API's from the same script (thus further angering the rate limiting gods) is because the V2 API rounds temperature readings unacceptably (to the nearest half degree, and then biases it towards the set point half a degree as well, like the controller itself) while the V1 API whilst giving high resolution un-rounded temperatures doesn't have any way to report hot water on/off status.

I can get current hot water temperature from the V1 API, however am I correct that there is no way to retrieve current hot water on/off status from the V1 API ? If I could somehow dig that information out of the V1 API I could eliminate my use of the V2 API completely and simplify the script a lot in the process especially when I would only have user_data to save instead of also having to save access_token, refresh_token and access_token_expiry, as well as using two sets of API calls.

I've just done a quick test, and it looks like the setpoint is always 0 whether the hot water is on or not. I had to override our DHW as it's not scheduled so it's possible that the web service wasn't up to date, but on the face of it it doesn't look like there is a way I'm afraid...

[watchforstock]

A bit off topic here, but the only reason evohome-munin polls both V1 and V2 API's from the same script (thus further angering the rate limiting gods) is because the V2 API rounds temperature readings unacceptably (to the nearest half degree, and then biases it towards the set point half a degree as well, like the controller itself) while the V1 API whilst giving high resolution un-rounded temperatures doesn't have any way to report hot water on/off status. I can get current hot water temperature from the V1 API, however am I correct that there is no way to retrieve current hot water on/off status from the V1 API ? If I could somehow dig that information out of the V1 API I could eliminate my use of the V2 API completely and simplify the script a lot in the process especially when I would only have user_data to save instead of also having to save access_token, refresh_token and access_token_expiry, as well as using two sets of API calls.

I've just done a quick test, and it looks like the setpoint is always 0 whether the hot water is on or not. I had to override our DHW as it's not scheduled so it's possible that the web service wasn't up to date, but on the face of it it doesn't look like there is a way I'm afraid...

I tell a lie. I've just looked more closely at the data returned and it gives us a status. I'll push a change that reveals this in the temperatures() call

[zxdavb]

No. You don't need to - I'm onnit.

Just give us a sec to submit a commit.

[watchforstock]

@DBMandrake Take a look at commit b1a4fba0a4cf93aedf6526aa0ef3ef89494e8607. I've added some extra information which I think tells you what you want in the status field. My hot water zone now returns: {'thermostat': 'DOMESTIC_HOT_WATER', 'id': nnnnnn, 'name': '', 'temp': 23.79, 'setpoint': 0, 'status': 'Scheduled', 'mode': 'DHWOff'}

[watchforstock]

@zxdavb Sorry didn't see your comment before I put a commit in!

[zxdavb]

Don't worry about it. Most of my commit was de-linting.

I'll hold it in reserve until we sort out the unauthorisamed/session expiry bit.

[DBMandrake]

@DBMandrake Take a look at commit b1a4fba. I've added some extra information which I think tells you what you want in the status field. My hot water zone now returns: {'thermostat': 'DOMESTIC_HOT_WATER', 'id': nnnnnn, 'name': '', 'temp': 23.79, 'setpoint': 0, 'status': 'Scheduled', 'mode': 'DHWOff'}

Brilliant! I'll do some testing today. :)

If the hot water status call works as anticipated I can eliminate use of the V2 API entirely in evohome-munin, greatly simplifying the script and reducing API calls and authentication attempts to Honeywell, as they don't like two authentication requests in close time proximity even when they are to different API's. (I already have to insert a delay of at least 20 seconds between V1 and V2 authentication or the second one always fails)

Since we've spent all this time working on and testing session reuse for the V2 API, to ensure that's going to work reliably for others I'll continue to use both V1 and V2 API initially with the session reuse changes, ensure session reuse works reliably with both API's over the next few days and only then strip it back to use just the V1 API that I'll need.

[zxdavb]

If the hot water status call works as anticipated I can eliminate use of the V2 API entirely in

Thinking about this - I'd use v1 for attributes, and maybe v2 for methods - I can think of at least 1 method not supported in v1, AutoWithReset, or will v1 support that as well?

Are there any other differences?

[DBMandrake]

Quick question @watchforstock, the temperatures call now seems to provide extra information aside from the hot water status such as status and mode for heating zones that weren't there before. For example, before:

{'temp': 36.29, 'setpoint': 0, 'thermostat': u'DOMESTIC_HOT_WATER', 'name': u'', 'id': 2079484}
{'temp': 20.8, 'setpoint': 18.0, 'thermostat': u'EMEA_ZONE', 'name': u'Bathroom', 'id': 1377389}
{'temp': 14.84, 'setpoint': 5.0, 'thermostat': u'EMEA_ZONE', 'name': u'Dining Room', 'id': 2379191}

after

{'status': u'Scheduled', 'thermostat': u'DOMESTIC_HOT_WATER', 'temp': 36.29, 'setpoint': 0, 'mode': u'DHWOn', 'id': 2079484, 'name': u''}
{'status': u'Scheduled', 'thermostat': u'EMEA_ZONE', 'temp': 20.8, 'setpoint': 18.0, 'mode': u'Off', 'id': 1377389, 'name': u'Bathroom'}
{'status': u'Hold', 'thermostat': u'EMEA_ZONE', 'temp': 14.84, 'setpoint': 5.0, 'mode': u'Off', 'id': 2379191, 'name': u'Dining Room'}

The schedule status looks handy - the zone that says hold has a permanent override applied, and one with a temporary override says Temporary.

I'm curious to know what "Mode" is for a heating zone, when all my zones say off, when they are in fact active at the time. Is that just an artefact of adding the Mode support for the hot water zone and should just be ignored ?

I guess because the new fields are members of a dictionary they shouldn't cause any problems with old scripts that aren't expecting them, as long as they access the fields in a proper pythonic way and don't do dumb stuff like doing text processing on a textual output of the dictionary...

A quick test suggests that DHW on/off status is correctly reported, which is great. Likewise I've just verified that the auto re-authenticate for the V1 API is also working correctly. For example if I fudge the stored session ID it fails, re-authenticates and succeeds as it should.

So as far as I can see from some quick testing it is all working as it should so I'm going to start work on adapting the evohome-munin scripts to make use of the all the new functionality. As I mentioned earlier I'll keep the V2 API use in place initially to help test the V2 session restore functionality of the new library but eventually remove it in favour of the new V1 hot water status and using only one API.

[DBMandrake]

If the hot water status call works as anticipated I can eliminate use of the V2 API entirely in

Thinking about this - I'd use v1 for attributes, and maybe v2 for methods - I can think of at least 1 method not supported in v1, AutoWithReset, or will v1 support that as well?

Are there any other differences?

I think the V1 API is best if you are just polling temperature data for graphing purposes as evohome-munin does, as you get high resolution (two decimal place) temperature readings that aren't biased and rounded, the API is lighter and quicker to give results, and the session id won't time out if you keep using it, so in theory once authenticated you can keep re-using the same session id, so your chances of being rate limited are almost zero.

The V2 API seems better suited for control applications where you are trying to read or modify schedules, (which V1 can't do at all) check on the current state the system is in, override the state, apply zone overrides and so on.

[DBMandrake]

Took me only about half an hour to rewrite evohome-munin to make use of the new session resume features, (including a bunch of temporary debug code to monitor everything including session-id's saved and restored) so I'll let it run overnight to see how reliable it is. So far in a couple of hours running polling every 5 minutes not a single failed poll attempt - quite a revelation compared to how it was before! :)

These changes to the library are looking really good.

[watchforstock]

Quick question @watchforstock, the temperatures call now seems to provide extra information aside from the hot water status such as status and mode for heating zones that weren't there before. For example, before:

Yes I pulled through some information that I thought looked useful, but I'll admit that I didn't spend a long time trying to work out the possible values, so I'm not sure how relevant mode is for the normal heating zones.

Glad it looks like the changes have made a tangible difference to reliability!

[DBMandrake]

Glad it looks like the changes have made a tangible difference to reliability!

Yes, working well. I ended up going straight to modifying evohome-munin to use only the V1 API and pull hot water status from that - works perfectly and the code is so much cleaner and simpler than before because of it.

To test the V2 API I have one of my standalone test scripts running from a cron job every 5 minutes logging the results to a log file. I've offset this cron job by one minute from evohome-munin to avoid any simultaneous authentication attempts.

Both have been flawless without a single failed response, so very happy about that result after months of intermittent problems!

I should mention that I've been testing b1a4fba0a4cf93aedf6526aa0ef3ef89494e8607 - I see significant changes to the V1 re-authentication mechanism have been made since then so I guess I should update to the latest tomorrow and re-test with that just to make sure everything is still OK.

[watchforstock]

I've now merged this into master. Please raise any issues as separate issues now and we'll track/fix as appropriate. Really appreciate everyone's input into making and testing this significant change

[DBMandrake]

@watchforstock Just wondering whether the session reuse changes that are now in master have been pushed to an official release in pip yet and if so what the version number is ?

[paulvee]

After reading through a lot of the relevant posts, I'm trying to make this work, but I'm not able to on my RPi model 3 with latest Debian updates. First of all, using the latest version of pip in combination with evohomeclient gives me errors. According to Google, I needed to go back to version 9.0.3. That indeed works without errors if you use sudo to circumvent access right issues: sudo pip install evohomeclient To make sure I installed the latest version that I believe is 2.8.10, I first uninstalled evohomeclient before installing it again.

With this install, I get access to access_token and access_token_expires, but not refresh_token. AttributeError: EvohomeClient instance has no attribute 'refresh_token'

When I look into my installation at: /usr/local/lib/python2.7/dist-packages/evohomeclient2/init.py The file is very different from what I see on the website, and there is no mentioning of refresh_token in the code. It seems to me that the pip install is not pulling in the correct files.

I guess that DBMandrake is correct in that the latest files are not in the master.

Furthermore, it would be very beneficial for mere mortals like myself to have a simple example of a script that features the new changes. It is sorely lacking in whatever documentation/information I have been able to find. This is by no means any critique to the folks that contributed, but you guys are way beyond most with your skills and can figure it out yourself.

[watchforstock]

@DBMandrake @paulvee Although the latest changes are on master, I've not released that yet as a new version - I'll aim to fix that tonight so should make the latest version pip installable then.

As for documentation, I agree it's not great. @zxdavb did write up some usage information for the v2 client in the wiki here: https://github.com/watchforstock/evohome-client/wiki/Access_tokens-(v2-client) There's been a lot of changes to the library since I last looked at the readthedocs information so it's probably due an update there too...

[watchforstock]

To make sure I installed the latest version that I believe is 2.8.10, I first uninstalled evohomeclient before installing it again.

@paulvee Interesting that this was installed for you. The latest "official" releases are listed here: https://pypi.org/project/evohomeclient/#history and 0.2.8 is the latest. The ones with the 4th bit of versioning (you installed 0.2.8.10) are only the test PyPI repository as I was testing some release configuration so the libraries there were not intended for use so I'm not sure why pip is picking them up. Hopefully an updated proper release later will sort that out.

[paulvee]

Hi Andrew,

Thanks for the very fast reply.

I look forward to test the latest version.

I'm by no means an accomplished Python programmer, but I get by with structured code OK, OO is still beyond my skills.

I did see the example but I don't think it's complete enough, for me at least (;-)).

I can try to get two examples together, because I need them anyway. One will be for a version that gets executed repeatedly, like with cron, and one version that stays active and uses its own timing.

I'm willing to post them for scrutiny and hopefully inclusion in the documentation.

Keep up the excellent work you guys do!

Regards,

Paul

Van: Andrew Stock notifications@github.com Verzonden: donderdag 14 maart 2019 11:07 Aan: watchforstock/evohome-client evohome-client@noreply.github.com CC: paulvee pw.versteeg@gmail.com; Mention mention@noreply.github.com Onderwerp: Re: [watchforstock/evohome-client] Client authentication triggering server rate limiting (#57)

@DBMandrake https://github.com/DBMandrake @paulvee https://github.com/paulvee Although the latest changes are on master, I've not released that yet as a new version - I'll aim to fix that tonight so should make the latest version pip installable then.

As for documentation, I agree it's not great. @zxdavb https://github.com/zxdavb did write up some usage information for the v2 client in the wiki here: https://github.com/watchforstock/evohome-client/wiki/Access_tokens-(v2-client) There's been a lot of changes to the library since I last looked at the readthedocs information so it's probably due an update there too...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/watchforstock/evohome-client/issues/57#issuecomment-472784680 , or mute the thread https://github.com/notifications/unsubscribe-auth/AC9OwtUsGBGAHyqM3ejeJfrrcZxxhnwRks5vWh9bgaJpZM4a_mZd . https://github.com/notifications/beacon/AC9Owp2N-MduEBaEVNzzlpcoZvlu83lWks5vWh9bgaJpZM4a_mZd.gif

[zxdavb]

I have updated the wiki article, Access_tokens-(v2-client) - if people want to edit it, then go for it!

Perhaps we need a v1 equivalent? I don't feel best powered to write it.

And maybe link to then in read the docs?

[DBMandrake]

I have updated the wiki article, Access_tokens-(v2-client) - if people want to edit it, then go for it!

Perhaps we need a v1 equivalent? I don't feel best powered to write it.

And maybe link to then in read the docs?

I'm happy to copy and paste simplified chunks of the evohome-munin rewrite that I've done which demonstrates how to save/restore user_data for the V1 API to disk.

I have had everything finished, working and debugged for a while now. The only reason I haven't pushed a PR to the evohome-munin project yet is I was waiting for an official release on PIP as evohome-munin will now depend on the latest version of evohome-client, so I thought waiting for it to be officially released was best. :+1:

[DBMandrake]

To make sure I installed the latest version that I believe is 2.8.10, I first uninstalled evohomeclient before installing it again.

@paulvee Interesting that this was installed for you. The latest "official" releases are listed here: https://pypi.org/project/evohomeclient/#history and 0.2.8 is the latest. The ones with the 4th bit of versioning (you installed 0.2.8.10) are only the test PyPI repository as I was testing some release configuration so the libraries there were not intended for use so I'm not sure why pip is picking them up. Hopefully an updated proper release later will sort that out.

I checked last night and it was still V0.2.8 in PIP, at least on Raspbian on a Raspberry Pi.

"AttributeError: EvohomeClient instance has no attribute 'refresh_token'" definitely means the installed version of the library is 0.2.8 or earlier, as that's what I got when I deliberately uninstalled my version from git master and installed the pip version.

[DBMandrake]

I have updated the wiki article, Access_tokens-(v2-client) - if people want to edit it, then go for it!

Perhaps we need a v1 equivalent? I don't feel best powered to write it.

And maybe link to then in read the docs?

I've created a page for the V1 API here:

https://github.com/watchforstock/evohome-client/wiki/user_data-(V1-client)

I shamelessly copied your V2 wiki page as a template, hope you don't mind. :)

BTW the end of your V2 wiki page talks about refresh_token's being invalidated by new concurrent sessions but I think we established that this doesn't actually happen and I had just been testing wrong ?

[watchforstock]

@DBMandrake @zxdavb @paulvee I've just pushed version 0.3.1 to pypi here (https://pypi.org/project/evohomeclient/#history). It would have been 0.3.0 if the first push hadn't failed!

[zxdavb]

I shamelessly copied your V2 wiki page as a template, hope you don't mind. :)

Yay, Open Source!

BTW the end of your V2 wiki page talks about refresh_token's being invalidated by new concurrent sessions but I think we established that this doesn't actually happen and I had just been testing wrong ?

What are people's interpretation of the following code:

from evohomeclient2 import EvohomeClient

username = "billg@microsoft.com"
password = "P@ssw0rd!"

c = EvohomeClient(username, password)
d = EvohomeClient(username, "bad_password", refresh_token=c.refresh_token)
e = EvohomeClient(username, "bad_password", refresh_token=d.refresh_token)
f = EvohomeClient(username, "bad_password", refresh_token=e.refresh_token)
g = EvohomeClient(username, "bad_password", refresh_token=f.refresh_token)
h = EvohomeClient(username, "bad_password", refresh_token=g.refresh_token)

print('c.refresh_token = ' + c.refresh_token)
print('d.refresh_token = ' + d.refresh_token)
print('e.refresh_token = ' + e.refresh_token)
print('f.refresh_token = ' + f.refresh_token)
print('g.refresh_token = ' + g.refresh_token)
print('h.refresh_token = ' + h.refresh_token)

c.access_token = None
d.access_token = None
e.access_token = None
f.access_token = None
g.access_token = None
h.access_token = None

print('userId' in c.user_account())
print('userId' in d.user_account())
print('userId' in e.user_account())
print('userId' in f.user_account())
print('userId' in g.user_account())
print('userId' in h.user_account())

print('c.refresh_token = ' + c.refresh_token)
print('d.refresh_token = ' + d.refresh_token)
print('e.refresh_token = ' + e.refresh_token)
print('f.refresh_token = ' + f.refresh_token)
print('g.refresh_token = ' + g.refresh_token)
print('h.refresh_token = ' + h.refresh_token)

[zxdavb]

FWIW, it looks something like only the latest 2 refresh tokens are valid.

[paulvee]

Something for "young players", I just discovered that installing does not update. You have to do an uninstall and then do a new install. If in doubt, run sudo pip search evohomeclient or sudo pip show evohomeclient