pytradfri 4.0.0 not working

[badblocks]

Where are you using pytradfri (eg stand-alone, Home Assistant etc)

standalone

Version of pytradfri

4.0.0

Expected behaviour

working again with gateway firmware 1.2.42

Actual behaviour

running this:

#!/usr/bin/env python3

import sys
from pytradfri import Gateway
from pytradfri.api.libcoap_api import APIFactory

api_factory = APIFactory(sys.argv[1])
with open('/mnt/rw/ikea/gateway_psk.txt', 'a+') as file:
    file.seek(0)
    psk = file.read()
    if psk:
        api_factory.psk = psk.strip()
    else:
        psk = api_factory.generate_psk(sys.argv[2])
        print('Generated PSK: ', psk)
        file.write(psk)
api = api_factory.request

root@raspberrypi:~# /opt/testing 192.168.21.26 <devicekey>

gives:

Traceback (most recent call last):
  File "/opt/testing", line 13, in <module>
    psk = api_factory.generate_psk(sys.argv[2])
  File "/usr/local/lib/python3.5/dist-packages/pytradfri/api/libcoap_api.py", line 156, in generate_psk
    self._psk = self.request(command)
  File "/usr/local/lib/python3.5/dist-packages/pytradfri/api/libcoap_api.py", line 93, in request
    return self._execute(api_commands)
  File "/usr/local/lib/python3.5/dist-packages/pytradfri/api/libcoap_api.py", line 87, in _execute
    api_command.result = _process_output(return_value, parse_json)
  File "/usr/local/lib/python3.5/dist-packages/pytradfri/command.py", line 71, in result
    self._result = self._process_result(value)
  File "/usr/local/lib/python3.5/dist-packages/pytradfri/gateway.py", line 28, in process_result
    return result[ATTR_PSK]
TypeError: 'NoneType' object is not subscriptable

???

[ggravlingen]

Was a gateway_psk.txt file written in your pytradfri-folder?

[badblocks]

Yes, but empty.

[ggravlingen]

To get you going, can you please testing rewriting the example-file so that the psk is printed to prompt:

@asyncio.coroutine
def run():
    # Assign configuration variables.
    # The configuration check takes care they are present.
    api_factory = APIFactory(sys.argv[1])
    psk = yield from api_factory.generate_psk(sys.argv[2])
    print('Generated PSK: ', psk)

And then you store the psk in the gateway_psk.txt-file.

[ggravlingen]

It should work if you have the psk, there are two of us that has got it working.

[hanpal]

Maybe an update of of README.me would help? With reference to the vital examples.

Also, will this really work now?

python3 -i -m pytradfri IP KEY

[ggravlingen]

These are the results I got when testing the new version:

:/usr/src/app# python3 example_async.py 192.168.0.129
decrypt_verify(): found 24 bytes cleartext
decrypt_verify(): found 52 bytes cleartext
decrypt_verify(): found 291 bytes cleartext
decrypt_verify(): found 283 bytes cleartext
decrypt_verify(): found 237 bytes cleartext
decrypt_verify(): found 239 bytes cleartext
decrypt_verify(): found 310 bytes cleartext
decrypt_verify(): found 235 bytes cleartext
[<65538 - Koksbord (FLOALT panel WS 30x30)>, <65540 - Hall 2 (TRADFRI bulb E27 opal 1000lm)>, <65539 - Hall 1 (TRADFRI bulb E27 W opal 1000lm)>, <65537 - Kökslampa (TRADFRI bulb E27 WS opal 980lm)>, <65542 - Läslampa (TRADFRI bulb E27 CWS opal 600lm)>]
Is on: True
Dimmer: 110
Name: Läslampa
decrypt_verify(): found 284 bytes cleartext
Received message for: <Light #0 - name: Koksbord, state: on, dimmer: 254, hex_color: efd275, xy_color: (32886, 27217), >
decrypt_verify(): found 238 bytes cleartext
Received message for: <Light #0 - name: Hall 2, state: on, dimmer: 193, hex_color: None, xy_color: (None, None), >
decrypt_verify(): found 240 bytes cleartext
Received message for: <Light #0 - name: Hall 1, state: on, dimmer: 193, hex_color: None, xy_color: (None, None), >
decrypt_verify(): found 292 bytes cleartext
Received message for: <Light #0 - name: Kökslampa, state: on, dimmer: 254, hex_color: 0, xy_color: (32538, 27213), >
decrypt_verify(): found 311 bytes cleartext
Received message for: <Light #0 - name: Läslampa, state: on, dimmer: 110, hex_color: efd275, xy_color: (32886, 27217), >
decrypt_verify(): found 8 bytes cleartext
decrypt_verify(): found 8 bytes cleartext
decrypt_verify(): found 312 bytes cleartext
Received message for: <Light #0 - name: Läslampa, state: on, dimmer: 110, hex_color: efd275, xy_color: (32886, 27217), >

:/usr/src/app# python3 example_sync.py 192.168.0.129
[<65538 - Koksbord (FLOALT panel WS 30x30)>, <65540 - Hall 2 (TRADFRI bulb E27 opal 1000lm)>, <65539 - Hall 1 (TRADFRI bulb E27 W opal 1000lm)>, <65537 - Kökslampa (TRADFRI bulb E27 WS opal 980lm)>, <65542 - Läslampa (TRADFRI bulb E27 CWS opal 600lm)>]
Sleeping to start observation task
Received message for: <Light #0 - name: Koksbord, state: on, dimmer: 254, hex_color: efd275, xy_color: (32886, 27217), >
True
254
Koksbord
Received message for: <Light #0 - name: Koksbord, state: on, dimmer: 254, hex_color: efd275, xy_color: (32886, 27217), >

[badblocks]

@ggravlingen: I first have to install aiocoap support. I will let you know if this works...

[dperezbr]

I think this error happened to me before when I tried to get a second pre_shared_key for the same identity. Maybe the identity must be a random string...anyway I don't get why we don't need the security key now. Is this more secure?

[ggravlingen]

@dperezbr this is how the gateway works, not much we can do I’m afraid 😊

[jtulak]

I had the same issue as OP with 4.0.1. I did not yet tried the aiocoap version (it complains about installed python 3.4.2, while required is 3.4.4 - sadly, Raspbian Jessie has 3.4.2 as the newest one), but if I manually write the xxx-xx-xxx code from Tradfri application into the file, it gets to line 57, where I get another issue:

Traceback (most recent call last):
  File "example_sync.py", line 107, in <module>
    run()
  File "example_sync.py", line 57, in run
    devices_commands = api(devices_command)
  File "/home/jtulak/tradfri/pytradfri/pytradfri/api/libcoap_api.py", line 93, in request
    return self._execute(api_commands)
  File "/home/jtulak/tradfri/pytradfri/pytradfri/api/libcoap_api.py", line 82, in _execute
    raise RequestTimeout() from None
pytradfri.error.RequestTimeout

[jtulak]

Update: I managed to get newer python and install aicoap, but the example_async.py is failing as well. I verified I can ping the Tradfri gate, and before the gateway fw update, I used this library for weeks.

Traceback (most recent call last):
  File "example_async.py", line 115, in <module>
    asyncio.get_event_loop().run_until_complete(run())
  File "/home/jtulak/berryconda3/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "example_async.py", line 45, in run
    psk = yield from api_factory.generate_psk(sys.argv[2])
  File "/home/jtulak/tradfri/pytradfri/pytradfri/api/aiocoap_api.py", line 193, in generate_psk
    self._psk = yield from self.request(command)
  File "/home/jtulak/tradfri/pytradfri/pytradfri/api/aiocoap_api.py", line 149, in request
    result = yield from self._execute(api_commands)
  File "/home/jtulak/tradfri/pytradfri/pytradfri/api/aiocoap_api.py", line 139, in _execute
    _, res = yield from self._get_response(msg)
  File "/home/jtulak/tradfri/pytradfri/pytradfri/api/aiocoap_api.py", line 92, in _get_response
    r = yield from pr.response
  File "/home/jtulak/berryconda3/lib/python3.6/site-packages/aiocoap/protocol.py", line 814, in _run_outer
    yield from cls._run(app_request, response, weak_observation, protocol, log, exchange_monitor_factory)
  File "/home/jtulak/berryconda3/lib/python3.6/site-packages/aiocoap/protocol.py", line 863, in _run
    blockresponse = yield from blockrequest.response
  File "/home/jtulak/berryconda3/lib/python3.6/site-packages/aiocoap/protocol.py", line 693, in _init_phase2
    yield from self.protocol.fill_remote(self.app_request)
  File "/home/jtulak/berryconda3/lib/python3.6/site-packages/aiocoap/protocol.py", line 426, in fill_remote
    raise RuntimeError("No transport could route message")
RuntimeError: No transport could route message

[JXGA]

I too am having the same problem as @jtulak, RuntimeError: No transport could route message

I am running on an RPI3 (without any HomeAssistant installed), with Python 3.5. I have installed aicoap. Trying to run the example program; python3.5 testp.py 192.168.0.12 KEY

Thanks to all involved for the work on this !

[ggravlingen]

@xt16johnny can you please check the post about gateway_psk.txt above and see if it helps?

[JXGA]

@ggravlingen Unfortunately no... I created example_async.py as follows;

`#!/usr/bin/env python3

import asyncio import logging import sys

from pytradfri import Gateway from pytradfri.api.aiocoap_api import APIFactory

root = logging.getLogger() root.setLevel(logging.INFO)

try:

pylint: disable=ungrouped-imports

from asyncio import ensure_future

except ImportError:

Python 3.4.3 and earlier has this as async

# pylint: disable=unused-import
from asyncio import async
ensure_future = async

@asyncio.coroutine def run():

Assign configuration variables.

# The configuration check takes care they are present.
api_factory = APIFactory(sys.argv[1])
psk = yield from api_factory.generate_psk(sys.argv[2])
print('Generated PSK: ', psk)

asyncio.get_event_loop().run_until_complete(run())`

Full response is;

pi@MagicPi:~/MagicHome $ python3.5 example_async.py 192.168.0.12 KEY Traceback (most recent call last): File "example_async.py", line 31, in <module> asyncio.get_event_loop().run_until_complete(run()) File "/usr/local/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete return future.result() File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result raise self._exception File "/usr/local/lib/python3.5/asyncio/tasks.py", line 241, in _step result = coro.throw(exc) File "example_async.py", line 27, in run psk = yield from api_factory.generate_psk(sys.argv[2]) File "/usr/local/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 193, in generate_psk self._psk = yield from self.request(command) File "/usr/local/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 149, in request result = yield from self._execute(api_commands) File "/usr/local/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 139, in _execute _, res = yield from self._get_response(msg) File "/usr/local/lib/python3.5/site-packages/pytradfri/api/aiocoap_api.py", line 92, in _get_response r = yield from pr.response File "/usr/local/lib/python3.5/asyncio/futures.py", line 361, in __iter__ yield self # This tells Task to wait for completion. File "/usr/local/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup future.result() File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result raise self._exception File "/usr/local/lib/python3.5/site-packages/aiocoap/protocol.py", line 814, in _run_outer yield from cls._run(app_request, response, weak_observation, protocol, log, exchange_monitor_factory) File "/usr/local/lib/python3.5/site-packages/aiocoap/protocol.py", line 863, in _run blockresponse = yield from blockrequest.response File "/usr/local/lib/python3.5/asyncio/futures.py", line 361, in __iter__ yield self # This tells Task to wait for completion. File "/usr/local/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup future.result() File "/usr/local/lib/python3.5/asyncio/futures.py", line 274, in result raise self._exception File "/usr/local/lib/python3.5/site-packages/aiocoap/protocol.py", line 693, in _init_phase2 yield from self.protocol.fill_remote(self.app_request) File "/usr/local/lib/python3.5/site-packages/aiocoap/protocol.py", line 426, in fill_remote raise RuntimeError("No transport could route message") RuntimeError: No transport could route message

If I amend the example_sync to have a similar code, it simply runs and does not print a key. I have had no joy with the gateway_psk.txt file with any test.

Thanks again

[jtulak]

OK, the "RuntimeError: No transport could route message" with async version could be because I forgot to install dtlssocket. It should be installing now (pip install DTLSSocket), but it is taking reaaally long time (I see gcc in htop, so I guess it is ok so far, just slow). @xt16johnny can you verify if you have the same issue?

[ggravlingen]

@jtulak took me about an hour (even more) to install on my rpi yesterday. So hang in there! :smile:

[ppietikainen]

For whatever reason generating a new psk with example_sync.py doesn't work, but after using example_async.py (with pip3 install DTLSSocket) a gateway_psk.txt is generated that works with both... First I thought my gateway already had a "pytradfri" identity, but changing that from the code still didn't work.

[jtulak]

I can confirm what @ppietikainen wrote just above. After installing DTLSSocket, the asynchronous example created the PSK file and with it, the synchronous example works as well. I see that #98 already fixes the documentation, so we can return to why the synchronous example does not work. 😃

[badblocks]

@ggravlingen: I can confirm now that my issues with >=4.0.0 have been solved. Sync & async working!

[JXGA]

@ggravlingen : All going, thank you for your time on this guys!

[ggravlingen]

@badblocks can the issue be closed?

[ppietikainen]

I think I found the example_sync.py problem: ['coap-client', '-u', 'Client_identity', '-k', 'CENSORED', '-v', '0', '-m', 'post', '-f', '-', 'coaps://192.168.1.X:5684/15011/9063']

which is missing -e '{"9090":"pytradfrii"}' ? (gotta run so no patch :) )

[hanpal]

I'm not convinced regarding the PSK handling, saving this to the file system. Why not just have a separate script that generates the PSK. It might save it to a file for backup but further use of the PSK is through a setting in the scripts just like IP and KEY was previously handled. IP and PSK instead.

Also, when running CLI, having KEY as a parameter is confusing since this is only used the first time (when PSK has not been created yet) according to ongoing pull request. Why not just have the PSK as parameter instead?

Seems that my question above was motivated:

Also, will this really work now? python3 -i -m pytradfri IP KEY

I haven't upgraded yet but i think that I'm going to put IP and PSK in a separate file that I import from various scripts. And a separate script just for generating the PSK but I will copy paste this to my file with IP and PSK.

Today in various scripts:

from credentials import ip, key

After upgrade to 4.x:

from credentials import ip, psk

Just some thoughts...

[ggravlingen]

@hanpal I'm fixing the problem with python3 -i -m pytradfri IP KEY at the moment.

[hanpal]

I know, just not convinced by the file handling used. Maybe better to separate the PSK generation completely from the example/CLI-scripts.

[ggravlingen]

@hanpal python3 -i -m pytradfri IP KEY/PSK should work in 4.0.2 that I just pushed to pypi.

[Lakitna]

I'm still having problems in 4.0.2. I'm using the synchronous functionality.

$ python3 example_sync.py <IP_OF_GATEWAY> <KEY_ON_GATEWAY>
Traceback (most recent call last):
  File "example_sync.py", line 107, in <module>
    run()
  File "example_sync.py", line 49, in run
    psk = api_factory.generate_psk(sys.argv[2])
  File "/Users/lakitna/Documents/Github/pytradfri/pytradfri/api/libcoap_api.py", line 156, in generate_psk
    self._psk = self.request(command)
  File "/Users/lakitna/Documents/Github/pytradfri/pytradfri/api/libcoap_api.py", line 93, in request
    return self._execute(api_commands)
  File "/Users/lakitna/Documents/Github/pytradfri/pytradfri/api/libcoap_api.py", line 87, in _execute
    api_command.result = _process_output(return_value, parse_json)
  File "/Users/lakitna/Documents/Github/pytradfri/pytradfri/command.py", line 71, in result
    self._result = self._process_result(value)
  File "/Users/lakitna/Documents/Github/pytradfri/pytradfri/gateway.py", line 28, in process_result
    return result[ATTR_PSK]
TypeError: 'NoneType' object is not subscriptable

The gateway_psk.txt file is being created, but it's empty.

I get the same error when I run python3 -i -m pytradfri IP KEY/PSK like @ggravlingen said.

[badblocks]

@Lakitna: Did you install this first? pip install -r requirements.txt I had the same error but I think after this it was working.

[ggravlingen]

@Lakitna I suppose that your gateway_psk.txt is empty? I was able to reproduce this error and couldn't get nor CLI or example scripts to work in my test environment. What I did to temporarily solve it was this:

% coap-client -u "Client_identity" -k KEY_ON_GATEWAY -v 0 -e '{"9090":"Lakitna"}' -m POST "coaps://192.168.0.129:5684/15011/9063"

This is returned
% {"9091":"PSK","9029":"1.2.0042"}

I then edited libcoap_api.py as a temporary workaround (replace PSK with output from above.)

    def _base_command(self, method):
        """Return base command."""
        return [
            'coap-client',
            '-u',
            'Lakitna',
            '-k',
            'PSK',
            '-v',
            '0',
            '-m',
            method
        ]

Ping @lwis: should we randomize the key generation, as has also been proposed downstream in Home Assistant?

[lwis]

@ggravlingen can't see why not.

[schliflo]

May I suggest to use a combination of the name and a timestamp like pytradfri_YYYYMMDDHHMMSS instead of randomizing the identity used for key generation? This would guarantee unique identities each time you request a PSK and possibly makes future debugging a bit easier.

[Lakitna]

@ggravlingen Your temporary fix does the trick, thanks for the help!

[Siopaos]

I was getting this timeout error at first:

python3 -i -m pytradfri 192.168.1.34 mykeywashere DEBUG:pytradfri.api.libcoap_api:Executing 192.168.1.34 post ['15011', '9063']: {'9090': 'Client_identity'} Traceback (most recent call last): File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.5/dist-packages/pytradfri-4.0.3-py3.5.egg/pytradfri/main.py", line 26, in File "/usr/local/lib/python3.5/dist-packages/pytradfri-4.0.3-py3.5.egg/pytradfri/api/libcoap_api.py", line 156, in generate_psk File "/usr/local/lib/python3.5/dist-packages/pytradfri-4.0.3-py3.5.egg/pytradfri/api/libcoap_api.py", line 93, in request File "/usr/local/lib/python3.5/dist-packages/pytradfri-4.0.3-py3.5.egg/pytradfri/api/libcoap_api.py", line 82, in _execute pytradfri.error.RequestTimeout

which turned to the following after I tried to apply the temporary fix for Lakitna:

DEBUG:pytradfri.api.libcoap_api:Received: Traceback (most recent call last): File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/pi/pytradfri/pytradfri/main.py", line 26, in psk = api_factory.generate_psk(sys.argv[2]) File "/home/pi/pytradfri/pytradfri/api/libcoap_api.py", line 156, in generate_psk self._psk = self.request(command) File "/home/pi/pytradfri/pytradfri/api/libcoap_api.py", line 93, in request return self._execute(api_commands) File "/home/pi/pytradfri/pytradfri/api/libcoap_api.py", line 87, in _execute api_command.result = _process_output(return_value, parse_json) File "/home/pi/pytradfri/pytradfri/command.py", line 71, in result self._result = self._process_result(value) File "/home/pi/pytradfri/pytradfri/gateway.py", line 28, in process_result return result[ATTR_PSK] TypeError: 'NoneType' object is not subscriptable

Do you have any idea by this log what the problem is?

[Lakitna]

Is there a planning for these breaking bugs? I'd offer to do it myself but I fear this kind of stuff is beyond my capabilities.

@Siopaos from your error: psk = api_factory.generate_psk(sys.argv[2])

It looks like it still tries to generate a new PSK. Did you save the one from @ggravlingen his temporary bugfix in a file called gateway_psk.txt in the pytradfri root? If you did, can you double check if it actually attempts to use it?

[badblocks]

@Siopaos & @Lakitna: Have you installed this first?

pip install -r requirements.txt
(installing aiocoap==0.4a1 and DTLSSocket==0.1.4)

I had the same error but, if I remember correctly, after this it was working.

[ggravlingen]

Update: I've done an implementation in pytradfri of the PR below from the Home Assistant project by @NovapaX. It's working locally and I will push (and merge to main branch) as soon as I've finished the PR on color management.

https://github.com/home-assistant/home-assistant/pull/10414

[Siopaos]

@Lakitna , dank je voor de tip. :)

I think I did, but it still doesn't work. I will try to double check it when I have some more time.

@badblocks yes I installed those, didn't help me though. also thanks for your hint though. :)

[ggravlingen]

This is working in my environment to generate psk and is leaning on the implementation in Home Assistant: https://gist.github.com/ggravlingen/459787e88bab4785fbd1a982970dba1f

Needs some cleanup but will hopefully do the trick.

[Siopaos]

I edited a gateway_psk.txt in a wrong location, but after I fixed that mistake I still had another error. This one I fixed by editing this part in libcoap_api.py: def generate_psk(self, security_key): """ Generate and set a psk from the security key. """ if not self._psk: existing_psk_id = self._psk_id self._psk_id = 'Siopaos' self._psk = 'myPSKwashere'

Now it's finally working. Thanks for the help guys! :-)

[badblocks]

Is it a good idea to close this issue?

[ggravlingen]

Please hold off a bit closing it. I'm working on a fix that will avoid (hopefully) these psk-errors.