API Changed again!

[dwhile]

Hive have changed the API again so the plugin no longer works.

[russdan]

The entire auth process has changed again - thread here has solved how to create a login token

https://www.domoticz.com/forum/viewtopic.php?p=262748#p262748

bottom line is first you need to install a couple of node packages, generate a HiveTokens.json using the instructions above. This seems to be a one-off process as the cognito/refresh-token process updates the token subsequently.

I've managed to re-write plugin.urllib.py to handle the new token 'refresh' process (userid and password no longer needed as managed through the SRP process above) - all it needs now is a new parameter on the plugin page to point to the token but I can't get the "Parameter" stuff to work in the plugin so its hardcoded at present which clearly isn't ideal...

Changes needed are:

in def onStart(self)

add the following line above self.multiplier = int(Parameters['Mode1']):

self.hivetoken = r'/home/pi/domoticz/scripts/node_modules/amazon-user-pool-srp-client/HiveTokens.json'

(this is the hardcoded bit which needs fixing via a parameter) - this is wherever your HiveTokens.jsonfile is located:

then replace the def GetSessionID(self) code block with the following:

    def GetSessionID(self):
            Domoticz.Log("Attempting token refresh using : " + self.hivetoken)
            with open(self.hivetoken, 'r') as file:
                postbody = file.readline()
            file.close

            url = 'https://beekeeper-uk.hivehome.com/1.0/cognito/refresh-token'

            data = postbody.encode('ascii')
            req = Request(url, data = data, unverifiable = True)
            req.add_header('Content-Type', 'application/json')

            req = urlopen(req)
            status_code = req.getcode()

            if status_code == 200:
                Domoticz.Log("Hive Login Successful: Status code " + str(status_code))

                r = req.read().decode('utf-8')
                self.sessionId = json.loads(r)["token"]
                file_handle = open(self.hivetoken, 'w')
                file_handle.write(r)
                file_handle.close()
            else:
                self.sessionId = ''
                Domoticz.Log("Hive Login Failed: Status code " + str(status_code))

Sorry, I should have done this as a proper git diff but I've done too many other experimental changes on the plugin code to do that immediately. I will reclone the repository and do a clean 'diff' later today, but hopefully this helps....

@imcfarla2003 hopefully this helps? I haven't looked at getting the non-urllib version of the plugin working as that seems an order of magnitude harder with the equivalent processes!

[russdan]

Still more to tweak, not quite working as desired....

2020-12-05 08:43:07.203 (Hive) Deleting Session
2020-12-05 08:43:07.322 (Hive) HTTP Error 401: Unauthorized
2020-12-05 08:43:07.323 (Hive) Attempting token refresh using : /home/pi/domoticz/scripts/node_modules/amazon-user-pool-srp-client/HiveTokens.json
2020-12-05 08:43:07.520 (Hive) Hive Login Successful: Status code 200
2020-12-05 08:43:07.647 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 907 in '/home/pi/domoticz/plugins/Hive/plugin.py', function onHeartbeat
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 192 in '/home/pi/domoticz/plugins/Hive/plugin.py', function onHeartbeat
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 874 in '/home/pi/domoticz/plugins/Hive/plugin.py', function GetDevices
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-05 08:43:07.647 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default

will have to look at that later.....

[russdan]

It seems to be onStop which is deleting the session which shouldn't be needed, so at present I have basically commented out onStop so it does nothing... seems to be holding up currently (has been running for about an hour successfully)... I will produce a diff file against the latest code base....

[imcfarla2003]

Thanks for doing the heavy lifting on this.Once you have a diff I can re-engineer it into the non urllib plugin.On 6 Dec 2020 17:23, russdan notifications@github.com wrote: It seems to be onStop which is deleting the session which shouldn't be needed, so at present I have basically commented out onStop so it does nothing... seems to be holding up currently (has been running for about an hour successfully)... I will produce a diff file against the latest code base....

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[russdan]

OK, here's the diff file:

plugin.urllib..py.dff.txt

It seems to be working with all the onStop stuff removed, has been running successfully for ~16 hours now.

I attempted to add the following to store the hardcoded token location: <param field="hivetoken" label="Hive SRP Token Location" width="800px" required="true" default="\/home\/pi\domoticz\/plugins\/Hive\/HiveToken.json"/>

and then parse hivetoken as a raw string, but the UI didn't persist the value entered (ie each time I returned to the Domoticz hardware page the new field was blank again) so clearly there's something else with the plugin stuff to make parameters work!

I'm sure its feasible to integrate a call to the Amazon SRP token generation code into here also (its only code!) so that from the outside no-one knows/cares about the HiveToken.json creation, but that's another enhancement I think :)

There are more changes which BG made to the "nodes" code (now called "products" but the json format is very different to the "nodes" code, not just the new beekeeper API endpoint, and I don't have plugs/bulbs etc so only have part of the picture) - I started to try and make those changes but its non-trivial.... I'll raise a new issue with my work so far as I'm sure at some stage they will drop all the non-beekeeper endpoints!

[russdan]

Anyone wanting to try this can git-apply the patch:

1. Copy the patch plugin.urllib..py.dff.txt file above to your ~/domoticz/plugin/Hive directory (ie wherever your hive-plug code is).
2. Do git apply plugin.urllib..py.dff.txt (this requires that you're using the plugin.urllib.py version of the code which must be renamed as plugin.py).
3. Follow the steps in https://www.domoticz.com/forum/viewtopic.php?p=262748#p262748 to create your HiveToken.json.
4. Restart Domoticz (or disable/enable the plugin to restart it).

After that the patch should make the necessary changes to get the plugin working again with the new authentication, until the base plugin code has been updated fully.

[dwhile]

Hi

Just followed your instructions above and although it now authenticates - it doesn't seem to update all devices. I have plugs and lamps - it works for temperature but not for the other devices.

[dwhile]

Ok

All sorted now. The changes to the plugin.py needed to be made to the url version for the candle lamps so that it worked correctly.

[Mark-Sellwood]

I have it working too, my only issue was that the HiveTokens.json ended up in /home/pi/node_modules/amazon-user-pool-srp-client & not /home/pi/domoticz/plugins/domoticz-hive so i edited plugin.py to point to /home/pi/node_modules/amazon-user-pool-srp-client

[russdan]

I think it depends where you npm installed from in the first steps, I think I was in ~/domoticz at the time. It needs a menu option to avoid the hard coded path but I failed to get that working earlier and at least wanted to get a patch out so the plugin worked again... Glad it's working, at least until Hive change something again! I'll have to have another go at getting the menu option working.....

[Mark-Sellwood]

Maybe take a look at https://github.com/999LV/BatteryLevel or https://github.com/FilipDem/Domoticz-NEST-plugin as they both have boxes for some of the settings. Till Hive play again!

[russdan]

Thanks, I think I understand how the code works, it's just it doesn't seem to work and persist the new 'token location' value... I must be missing something in the onStart code which handles persistence and haven't managed to find what as this doesn't seem covered in any examples so far.....

[dwhile]

Ok Here's what I did and it seems to work:

Added the following below the password field (around line 20) <param field="Mode4" label="Hive json location" width="200px" required="true" default=""/>

In onStart change the line that sets the hive token (around line 80) to: self.hivetoken = Parameters["Mode4"] And it should all work - well at least it does for me :)

[imcfarla2003]

Thanks to everyone for helping on this.

I have updated the urllib plugin to use pycognito to get access codes from AWS using the username and password configured. You will need to run pip3 install pycognito (as root preferably) and then update to the latest version of the plugin.

Can I get some feedback please?

[Mark-Sellwood]

I have just upgraded & will report back.

[russdan]

Can I get some feedback please?

@imcfarla2003 - looks good - great spot on pycognito, keeps all the AWS auth stuff under the covers where it should be :)

So far looks like back to normal, pulling thermostat details, temps etc.

Awesome update, thank you!

[dwhile]

Ok - I have updated and everything works OK except for the candle bulbs. I have created a pull request that fixes this

[dwhile]

Just noticed an error in the log file. `

`2020-12-27 12:19:35.035 (House) Deleting Session 2020-12-27 12:19:36.645 (House) HTTP Error 401: Unauthorized 2020-12-27 13:21:15.441 (House) Deleting Session 2020-12-27 13:21:15.577 (House) HTTP Error 401: Unauthorized 2020-12-27 14:23:07.015 (House) Deleting Session 2020-12-27 14:23:07.134 (House) HTTP Error 401: Unauthorized``

It appears after the session has been deleted (onStop called) I'm guessing its trying to do a refresh before getting a new session

[russdan]

I saw this earlier (https://github.com/imcfarla2003/domoticz-hive/issues/21#issuecomment-739148872) and in my code I had completely commented-out onStop (all I left was a Domoticz.Log message) and that seemed to fix the issue - essentially I don't believe with the new auth method there is a need to delete sessions via the onStop method. I see in the latest version of the plugin.urllib.py this code is intact, so my thoughts are to remove it....

[imcfarla2003]

I have now removed the onStop code as per your suggestion. Hopefully this is now all good (for a while)

[russdan]

Hmm, something not quite right yet.... this happened last night, and the only reason it corrected itself is my hotwater boost code kicked in at 09:35 which then created a new session, so the OnHeatbeat code isn't picking up the HTTPError to then force a new token....

2020-12-29 00:09:22.928 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 00:09:22.928 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 00:09:22.928 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 00:09:22.928 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 00:09:22.928 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 00:09:22.928 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 00:09:22.928 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 01:00:00.412 Status: Starting automatic database backup procedure...
2020-12-29 01:00:03.202 Status: Ending automatic database backup procedure...
2020-12-29 01:10:48.417 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 01:10:48.417 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 01:10:48.417 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 01:10:48.417 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 01:10:48.417 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 01:10:48.418 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 01:10:48.418 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 01:10:48.418 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 01:10:48.418 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 01:10:48.418 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 02:00:00.811 Status: Starting automatic database backup procedure...
2020-12-29 02:00:04.051 Status: Ending automatic database backup procedure...
2020-12-29 02:12:05.335 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 02:12:05.335 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 02:12:05.336 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 02:12:05.336 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 03:00:00.492 Status: Starting automatic database backup procedure...
2020-12-29 03:00:03.269 Status: Ending automatic database backup procedure...
2020-12-29 03:13:19.226 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 03:13:19.227 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 04:00:00.691 Status: OpenZWave: Heal Network command initiated...
2020-12-29 04:00:00.691 Status: Starting automatic database backup procedure...
2020-12-29 04:00:03.110 Status: Ending automatic database backup procedure...
2020-12-29 04:14:48.751 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 04:14:48.751 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 04:14:48.752 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 05:00:15.316 Status: Starting automatic database backup procedure...
2020-12-29 05:00:18.085 Status: Ending automatic database backup procedure...
2020-12-29 05:16:00.662 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 05:16:00.667 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 06:00:01.140 Status: Starting automatic database backup procedure...
2020-12-29 06:00:03.362 Status: Ending automatic database backup procedure...
2020-12-29 06:17:18.159 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 06:17:18.159 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 06:17:18.159 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 06:17:18.159 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 06:17:18.159 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 06:17:18.159 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 06:17:18.160 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 06:17:18.160 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 06:17:18.160 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 06:17:18.160 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 07:00:00.601 Status: Starting automatic database backup procedure...
2020-12-29 07:00:03.916 Status: Ending automatic database backup procedure...
2020-12-29 07:18:35.114 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 07:18:35.114 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 07:18:35.115 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 07:40:31.817 Status: User: Admin initiated a switch command (743/Loo Light Off Delay/On)
2020-12-29 07:41:20.787 Status: User: Admin initiated a switch command (743/Loo Light Off Delay/On)
2020-12-29 08:00:00.405 Status: Starting automatic database backup procedure...
2020-12-29 08:00:04.729 Status: Ending automatic database backup procedure...
2020-12-29 08:20:03.853 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 08:20:03.853 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 08:20:03.854 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 09:00:01.074 Status: Starting automatic database backup procedure...
2020-12-29 09:00:08.562 Status: Ending automatic database backup procedure...
2020-12-29 09:21:29.694 Error: (Hive) 'onHeartbeat' failed 'HTTPError'.
2020-12-29 09:21:29.694 Error: (Hive) ----> Line 921 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 09:21:29.694 Error: (Hive) ----> Line 205 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function onHeartbeat
2020-12-29 09:21:29.694 Error: (Hive) ----> Line 888 in '/home/pi/domoticz/plugins/domoticz-hive/plugin.py', function GetDevices
2020-12-29 09:21:29.694 Error: (Hive) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen
2020-12-29 09:21:29.694 Error: (Hive) ----> Line 531 in '/usr/lib/python3.7/urllib/request.py', function open
2020-12-29 09:21:29.694 Error: (Hive) ----> Line 641 in '/usr/lib/python3.7/urllib/request.py', function http_response
2020-12-29 09:21:29.695 Error: (Hive) ----> Line 569 in '/usr/lib/python3.7/urllib/request.py', function error
2020-12-29 09:21:29.695 Error: (Hive) ----> Line 503 in '/usr/lib/python3.7/urllib/request.py', function _call_chain
2020-12-29 09:21:29.695 Error: (Hive) ----> Line 649 in '/usr/lib/python3.7/urllib/request.py', function http_error_default
2020-12-29 09:35:01.039 Status: Hive Hotwaterboost: New Hive Session ID needed - calling login function
2020-12-29 09:35:01.342 Status: Hive Hotwaterboost: Login Status code 200
2020-12-29 09:35:01.349 Status: Hive Hotwaterboost: New Hive Session ID created: eyJraWQiOiI4Wjgza2E........

I'll take a look later to see if I can understand what's not quite right.... shouldn't be too hard to respond to the error :) although must admit I thought the right code was already there to do so!

[dwhile]

This appears to be tied in with the onStop message. I think we can safely remove the calls to onStop in GetDevices and GetWeatherURL.

It looks like the token is only valid for 1 hour. After that a new token is needed.

This is from my log:

2020-12-29 14:01:46.647 (House) Getting Data 2020-12-29 14:01:46.802 (House) GetDevices onStop 2020-12-29 14:01:46.802 (House) onStop Called 2020-12-29 14:01:46.802 (House) Attempting token refresh using : eyJraWQiOiI4Wj gza2EwZE9lSk9pa1psUnVoakkySGRxbEZVQ0hWcHdyUVJTNk9lVGY0PSIsImFsZyI6IlJTMjU2In0.ey JzdWIiOiJhOTNhMTQ5OC0xODE4LTQyNzAtOTQ5Yy1jZGY5YzI1NWVmYTIiLCJ6b25laW5mbyI6IkV1cm 9wZVwvTG9uZG9uIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLW lkcC5ldS13ZXN0LTEuYW1hem9uYXdzLmNvbVwvZXUtd2VzdC0xX1NhbU5mb1d0ZiIsInBob25lX251bW Jlcl92ZXJpZmllZCI6dHJ1ZSwiY29nbml0bzp1c2VybmFtZSI6ImQ5MTQyMWQxLTg3MTEtNDU2Mi1hMT MxLTY5MTkyMDExMDBmMyIsImxvY2FsZSI6ImVuX0dCIiwiY3VzdG9tOmxvY2F0aW9uIjoiZXUtd2VzdC 0xIiwiYXVkIjoiM3JsNGkwYWpybXRkbThzYnJlNTRwOWR2ZDkiLCJldmVudF9pZCI6IjljMjE2NzQzLT UzZjEtNGYwYy1hMTliLTE1YTEzZjQ1ZjExOSIsInRva2VuX3VzZSI6ImlkIiwiYXV0aF90aW1lIjoxNj A5MjQ2ODQ0LCJleHAiOjE2MDkyNTA0NDQsImlhdCI6MTYwOTI0Njg0NCwiY3VzdG9tOmVudmlyb25tZW 50IjoicHJvZCIsImVtYWlsIjoiZHdoaWxlQHdoaWxlLm9yZy51ayJ9.uxjzsiAG6b7mQvZnFFNsd5d3e 5p5sA34h9f5ZsyEA8Iipbbx3n0J5zPoIuOaiN1TwAWJwIzhlDGzwi26rU_UmD-VaSGzbsmHaZowCdK9c aybxCkbd89A_xPzpquKDU8DQyrro13wUKIBgRK30yZdgz-x6tLCW84AWB82rOYErrOjaG2-LUatePnyv 1Rwqy4EaXdahvhJSQu7KcI9RawHxKqFT-z4S7zR9f9sTPZXN8DOwwpgXUjPFNh0h5UjRKP7snlFOkB1b jXbZqlTiesMUmHEL7n1sZcII5dtXyB6MEyWNSM4Ea82Gv63vvz4Fu8nS57K4NAuGVlR7ims_LO11w 2020-12-29 14:01:47.076 (House) Hive Login Successful: Status code 200 2020-12-29 14:01:47.078 (House) New token: eyJraWQiOiI4Wjgza2EwZE9lSk9pa1psUnVo akkySGRxbEZVQ0hWcHdyUVJTNk9lVGY0PSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJhOTNhMTQ5OC0x ODE4LTQyNzAtOTQ5Yy1jZGY5YzI1NWVmYTIiLCJ6b25laW5mbyI6IkV1cm9wZVwvTG9uZG9uIiwiZW1h aWxfdmVyaWZpZWQiOnRydWUsImlzcyI6Imh0dHBzOlwvXC9jb2duaXRvLWlkcC5ldS13ZXN0LTEuYW1h em9uYXdzLmNvbVwvZXUtd2VzdC0xX1NhbU5mb1d0ZiIsInBob25lX251bWJlcl92ZXJpZmllZCI6dHJ1 ZSwiY29nbml0bzp1c2VybmFtZSI6ImQ5MTQyMWQxLTg3MTEtNDU2Mi1hMTMxLTY5MTkyMDExMDBmMyIs ImxvY2FsZSI6ImVuX0dCIiwiY3VzdG9tOmxvY2F0aW9uIjoiZXUtd2VzdC0xIiwiYXVkIjoiM3JsNGkw YWpybXRkbThzYnJlNTRwOWR2ZDkiLCJldmVudF9pZCI6IjljMjE2NzQzLTUzZjEtNGYwYy1hMTliLTE1 YTEzZjQ1ZjExOSIsInRva2VuX3VzZSI6ImlkIiwiYXV0aF90aW1lIjoxNjA5MjQ2ODQ0LCJleHAiOjE2 MDkyNTQxMDYsImlhdCI6MTYwOTI1MDUwNywiY3VzdG9tOmVudmlyb25tZW50IjoicHJvZCIsImVtYWls IjoiZHdoaWxlQHdoaWxlLm9yZy51ayJ9.BnioXVLi6bQR4O8btex-9L2-KeqPtlorzLb8BjDDvwbjiRa B4NYokk1dBWzx5ESwYs6ItMrYXMltlwBDRXL8ykXcsroKiSktYIj56o_ZujrF6WGDOwXxmVjEsWyWE3h kKZVDFGNisfmelcO-vmeLuR79ptLwEC_1Z3JXN__Gzw67iZ6N-B3Iuka6nDWuGIlOwmHNb73W49eOfP3 FZSbc7I-1F5gRhXg2ZY4LwIGyX0cb6kubUihY2GOZgty0N-mnYrijth-Z8JQL0ZscRwdScEwPvU5HDIF 7CgDClhvAjluXEJub7AuRotGEGBkpFP6jJv0Gc9JrXX7EA-LhQuOFYg 2020-12-29 14:01:47.210 Error: (House) 'onHeartbeat' failed 'HTTPError'. 2020-12-29 14:01:47.211 Error: (House) ----> Line 913 in '/opt/domoticz/plugins/Hive/plugin.py', function onHeartbeat 2020-12-29 14:01:47.211 Error: (House) ----> Line 194 in '/opt/domoticz/plugins/Hive/plugin.py', function onHeartbeat 2020-12-29 14:01:47.211 Error: (House) ----> Line 880 in '/opt/domoticz/plugins/Hive/plugin.py', function GetDevices 2020-12-29 14:01:47.211 Error: (House) ----> Line 222 in '/usr/lib/python3.7/urllib/request.py', function urlopen`

As you can see it successfully does the token refresh but then generates the error.

[russdan]

Yes, agreed, I see the same pattern....

What's strange is in my code I have already disabled onStop()at the function itself, so all it does is log "(Hive) Deleting Session" but does nothing more....

Is it related to the way its regenerating the token? Its doing a urlopen, then gets a 401 so does a token refresh, then issues a urlopen again, but "headers" in the second call hasn't been updated with the new token so the call after the token refresh fails, then works next time round?

    def GetDevices(self):
            nodes = False
            headers = {'Content-Type': 'application/vnd.alertme.zoo-6.2+json',
                       'Accept': 'application/vnd.alertme.zoo-6.2+json',
                       'X-AlertMe-Client': 'swagger',
                       'X-Omnia-Access-Token': self.sessionId}
            url = self.Honeycomb + '/omnia/nodes'
            req = Request(url, headers = headers, unverifiable = True)

            try:
                r = urlopen(req).read().decode('utf-8')
            except HTTPError as e:
                if e.code == 401: # Unauthorised - need new sessionId
                    #self.onStop()
                    self.GetSessionID()
                    r = urlopen(req).read().decode('utf-8')

so doesn't it need to look more like:

            try:
                r = urlopen(req).read().decode('utf-8')
            except HTTPError as e:
                if e.code == 401: # Unauthorised - need new sessionId
                    self.GetSessionID()
                    headers = {'Content-Type': 'application/vnd.alertme.zoo-6.2+json',
                               'Accept': 'application/vnd.alertme.zoo-6.2+json',
                               'X-AlertMe-Client': 'swagger',
                               'X-Omnia-Access-Token': self.sessionId}
                    req = Request(url, headers = headers, unverifiable = True)
                    r = urlopen(req).read().decode('utf-8')

?

I've made these changes so will see if that helps! I think the error isn't functional (ie things still work) but would be nice to remove it as it may cause confusion!

[dwhile]

Ah

I think I see the problem. The headers in the URL request are not regenerated so I think it is using the old session id when it does the second call after getting the 401. I think the code in the HTTPError exception needs to have the headers regenerated with the new session id and then the url request regenerated as you have suggested. I think that will get rid of the error.

[imcfarla2003]

I don't think it gets a 401so I don't think that the sessionId gets renewed - trying a debug run now but will have to wait a bit for it to error again.

[russdan]

Hi, I restarted with my changes above at 15:43 and at 16:44 and 17:46 it did generate a 401 and successfully refreshed the token without the HTTPError seen previously... obviously that's only two passes around the loop so far but seems to be handling at least that specific error condition now. I added a dump of the HTTPError in the logging so if it does generate non-401 errors I should be able to see those too....

[imcfarla2003]

I have committed your changes and it is looking good on my system.

[imcfarla2003]

@russdan can you open a new issue with your hotwater boost code so I can see if we can implement it?

[russdan]

@russdan can you open a new issue with your hotwater boost code so I can see if we can implement it?

Sort of... its linked in with a custom hotwater sensor I made based on an article in Hackspace magazine (https://hackspace.raspberrypi.org/issues/4/pdf) so the entire code itself won't be completely useful, but I can share the relevant Hive water boost code, no problem...

[dwhile]

I made the changes as well as suggested by russdan and all is looking good.

[russdan]

@imcfarla2003 I think we can close this one now :) Phew that was fun..... good team effort though I think...

[crgrimwade]

Hi Been following with interest - and have made all the changes (I think). But am getting the following:

2020-12-30 21:19:04.280 (Hive) Starting 2020-12-30 21:19:04.280 (Hive) DomoticzVersion Available 2020.2 2020-12-30 21:19:04.280 (Hive) TimedOut available 2020-12-30 21:19:04.280 (Hive) Creating Session 2020-12-30 21:19:04.277 Status: (Hive) Entering work loop. 2020-12-30 21:19:04.278 Status: (Hive) Initialized version 2.1-urllib, author 'imcfarla, MikeF and roadsnail' 2020-12-30 21:19:08.469 (Hive) Attempting token refresh using : /home/pi/node_modules/amazon-user-pool-srp-client/HiveTokens.json 2020-12-30 21:19:08.753 (Hive) Hive Login Successful: Status code 200 2020-12-30 21:19:08.755 Error: (Hive) 'onStart' failed 'ValueError'. 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 904 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function onStart 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 103 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function onStart 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 197 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function onHeartbeat 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 878 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function GetDevices 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 328 in '/usr/lib/python3.7/urllib/request.py', function init 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 354 in '/usr/lib/python3.7/urllib/request.py', function full_url 2020-12-30 21:19:08.755 Error: (Hive) ----> Line 383 in '/usr/lib/python3.7/urllib/request.py', function _parse

By the way I had to turn off 2FA before managing to get a token - with 2FA turned on it kept sending me codes via SMS. Have not yet turned this back on. PS: I assumed that the line self.hivetoken = r'/home/pi/domoticz/scripts/node_modules...... should have been self.hivetoken = '/home/pi/domoticz/scripts/node_modules...

[crgrimwade]

followed by ... 2020-12-30 21:41:18.394 Error: (Hive) 'onHeartbeat' failed 'ValueError'. 2020-12-30 21:41:18.394 Error: (Hive) ----> Line 925 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function onHeartbeat 2020-12-30 21:41:18.394 Error: (Hive) ----> Line 197 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function onHeartbeat 2020-12-30 21:41:18.394 Error: (Hive) ----> Line 878 in '/home/pi/domoticz/plugins/HivePlug/plugin.py', function GetDevices 2020-12-30 21:41:18.394 Error: (Hive) ----> Line 328 in '/usr/lib/python3.7/urllib/request.py', function init 2020-12-30 21:41:18.395 Error: (Hive) ----> Line 354 in '/usr/lib/python3.7/urllib/request.py', function full_url 2020-12-30 21:41:18.395 Error: (Hive) ----> Line 383 in '/usr/lib/python3.7/urllib/request.py', function _p

[russdan]

You should just be able to do a git clone of the latest code and it will work. Looks like you've applied old fixes which have been superseded now... Small possibility that not everything has not yet been pushed to the latest code, but basically it's only the last tidy up of the httperror which was largely cosmetic.

[crgrimwade]

Sorry maybe I am being a bit slow - I have replaced my plugin.py with the lastest plugin.urllib.py from github. Seems to be working again although not sure how as it does not seem to be using the HiveTokens.json file at all.

[dwhile]

The HiceTokens.json file isn't used in the latest version. The token is obtained and stored internally using pycognito to get the token from AWS.

[crgrimwade]

Got it - thanks