Leshan decoding error

[cdh4u]

I am running the client on a Windows machine. The client registers to a Leshan server. After that, I try to read a value (6/0/0 - Latitue ) from Leshan, I get the follwoing error alert in Leshan:

Unable to read resource /6/0/0 for python-client : 500 Invalid Response:Unable to decode response payload of request [ReadRequest [path=/6/0/0 format=ContentFormat [name=TLV, code=11542]]] from client [python-client]

In the client I see the following log:

2020-06-24 15:00:55,246 [DEBUG] read on 6/0/0 Received read 2020-06-24 15:00:55,247 [DEBUG] encode_resource(): 00000000: 32 37 2E 39 38 36 30 36 35 27.986065 2020-06-24 15:00:55,247 [DEBUG] Sending message <aiocoap.Message at 0x40963b8: Type.ACK 2.05 Content (MID 12371, token f4ca15c20d4affe1) remote <_Connection at 0x4090c28 on transport <_ProactorDatagramTransport fd=788 read=<_OverlappedFuture pending overlapped=<pending, 0x4075d38> cb=[_ProactorDatagramTransport._loop_reading()]>>, active>, 9 byte(s) payload>

[aellwein]

@cdh4u thanks for report, seems that lot of things changed since i used the client the last time. I need first to find the cause by investigating of what has changed in Leshan / content format of TLV since then.

[aellwein]

@cdh4u ok, i found and fixed the cause of the issue.

There were actually two causes, the first one is the obsolete IANA types referenced in the content-format of the lwm2mclient (and also the usage of the content_format message option has been changed meanwhile in aiocoap, so i had to adapt it).

The second issue is the rather ancient object definition file (lwm2m-object-definitions.json) which was created about 5 years ago and reflects the state of the LwM2M spec at that point of time, there were huge changes since then. The error you got was because longitude was defined to have "string" type, however, Leshan (1.0.1) expects a "float" type here. long, lat, altitude were adjusted to have "float" data type.

But Location object is the only place i adapted the definitions; please expect some resources to be missing or of wrong type, any help on updating this file will be appreciated.

P.S. i'm currently working (slow) on a rewrite of this software, so there will be radical changes to the lwm2mclient. This, however, will take some time.

[aellwein]

Can you please retest on Windows and tell me if the issue is now solved? Can confirm to have it running on Mac using lwm2mclient --address ::1 with Leshan 1.0.1

[cdh4u]

First, did you use aiocoap==0.4a1? You have that in the setup.py file.

I previously modified it to 4b3, and manually did the changes done by @chrysn in the simple6.py file, and then it worked (until I got the Leshan error).

However, now I get the following new error when I try to ryn client.py:

2020-06-26 23:15:04,648 [DEBUG] Using proactor: IocpProactor 2020-06-26 23:15:04,663 [DEBUG] Sending request - Token: 3b59, Remote: <_Connection at 0x3f7ce98 on transport <_ProactorDatagramTransport fd=788 read=<_OverlappedFuture pending overlapped=<pending, 0x3f92bd8> cb=[_ProactorDatagramTransport._loop_reading()]>>, active> 2020-06-26 23:15:04,663 [DEBUG] Sending message <aiocoap.Message at 0x3f7cf40: Type.CON POST (MID 641, token 3b59) remote <_Connection at 0x3f7ce98 on transport <_ProactorDatagramTransport fd=788 read=<_OverlappedFuture pending overlapped=<pending, 0x3f92bd8> cb=[_ProactorDatagramTransport._loop_reading()]>>, active>, 4 option(s), 27 byte(s) payload> 2020-06-26 23:15:04,663 [DEBUG] Exchange added, message ID: 641. 2020-06-26 23:15:04,663 [DEBUG] Incoming error 22 from <_Connection at 0x3f7ce98 on transport <_ProactorDatagramTransport fd=788>, active> 2020-06-26 23:15:04,663 [ERROR] Task exception was never retrieved future: <Task finished name='Task-1' coro=<Client.run() done, defined at client.py:154> exception=OSError(22, 'Invalid argument')> Traceback (most recent call last): File "client.py", line 167, in run response = await self.context.request(request).response File "C:\Users\lmfchh\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiocoap-0.4b3-py3.8.egg\aiocoap\protocol.py", line 731, in _run_outer await cls._run(app_request, response, weak_observation, protocol, log) File "C:\Users\lmfchh\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiocoap-0.4b3-py3.8.egg\aiocoap\protocol.py", line 783, in _run blockresponse = await blockrequest.response OSError: [Errno 22] Invalid argument

[cdh4u]

Is there a way to fetch the latest aircoap commits, that aren't part of a release yet?

[chrysn]

@cdh4u, it appears that you are using the aiocoap package in the form of a locally installed 0.4b3 version. You can install the latest one using pip3 install --upgrade "git+https://github.com/chrysn/aiocoap#egg=aiocoap" (as per installation docs), but the changes in there were primarily multicast related.

I have a rough idea where that exception could be raised from, maybe I can try on a colleague's Windows box. What server address did you put in, ::1? a fe80? a routable one?

(By the way, @aellwein are you sure the content_format changes were necessary? Message(..., content_format=x) should just be a shorthand for passing in the content_format into the .opt object).

[aellwein]

First, did you use aiocoap==0.4a1? You have that in the setup.py file.

I previously modified it to 4b3, and manually did the changes done by @chrysn in the simple6.py file, and then it worked (until I got the Leshan error).

@cdh4u sorry, i forgot to mention some things here. I stumbled upon another issue (chrysn/aiocoap#194) while testing the #11. That's why i had to use aiocoap from master branch, which is not released in PyPI yet.

I usually install dependencies in some virtualenv folder while development rather than install them system- or user-wide. It helps keeping the system clean and is very handy if working on multiple Python projects.

To install the dependencies in the local virtualenv, i usually do following:

• change dir to the project dir (lwm2mclient);
• run python3 -m venv .venv - this will create an empty virtualenv in .venv folder inside project, the name of the folder doesn't matter, i just use .venv for convenience ;
• run source .venv/bin/activate or .venv\Scripts\activate.bat (if on Windows) - to activate your virtualenv, this will patch your PATH to point to the .venv, so all Python-related installation will happen in there (remember to always activate your virtualenv if you change to a new shell, the PATH changes it does are not persistent to your system);
• Install aiocoap here using pip as mentioned by @chrysn.

With the latest aiocoap, lwm2mclient, Leshan (here i used latest released version 1.0.1), i managed to connect and read Location object (at least the attributes i defined in the definition 5 years ago).

(By the way, @aellwein are you sure the content_format changes were necessary? Message(..., content_format=x) should just be a shorthand for passing in the content_format into the .opt object).

@chrysn somehow it didn't work, Leshan claimed that the content format is null, i.e. not set. Then i saw in aiocoap code, that passing opts via kwargs actually does the same, so i changed this. Should i do more in-depth investigation on that? Maybe contribute some test to aiocoap?

[cdh4u]

@cdh4u, it appears that you are using the aiocoap package in the form of a locally installed 0.4b3 version. You can install the latest one using pip3 install --upgrade "git+https://github.com/chrysn/aiocoap#egg=aiocoap" (as per installation docs), but the changes in there were primarily multicast related.

What should I write in the setup.py file in order to use it?

Currently I have:

install_requires=[
    'aiocoap==0.4b3',
    'hexdump==3.3'
]

Sorry if this sounds like a stupid question, but I am realtively new to Python, so :)

I have a rough idea where that exception could be raised from, maybe I can try on a colleague's Windows box. What server address did you put in, ::1? a fe80? a routable one?

Actually, I realized there was a spelling error in the server address. I fixed it, and now it seems to work (0.4b3 + manual change in simple6.py). The client registers with the Leshan server, and I manage to succesfully read the Latitude value.

2020-06-28 11:21:35,736 [DEBUG] read on 6/0/0 2020-06-28 11:21:35,736 [DEBUG] encode_resource(): 00000000: 32 37 2E 39 38 36 30 36 35 27.986065 2020-06-28 11:21:35,736 [DEBUG] Sending message <aiocoap.Message at 0x38243b8: Type.ACK 2.05 Content (MID 38083, token 829018fefd04c19a) remote <_Connection at 0x3822e80 on transport <_ProactorDatagramTransport fd=712 read=<_OverlappedFuture pending overlapped=<pending, 0x380ebd8> cb=[_ProactorDatagramTransport._loop_reading()]>>, active>, 1 option(s), 9 byte(s) payload>

Sorry for the confusion :)

[cdh4u]

I usually install dependencies in some virtualenv folder while development rather than install them system- or user-wide. It helps keeping the system clean and is very handy if working on multiple Python projects. To install the dependencies in the local virtualenv, i usually do following:

change dir to the project dir (lwm2mclient); run python3 -m venv .venv - this will create an empty virtualenv in .venv folder inside project, the name of the folder doesn't matter, i just use .venv for convenience ; run source .venv/bin/activate or .venv\Scripts\activate.bat (if on Windows) - to activate your virtualenv, this will patch your PATH to point to the .venv, so all Python-related installation will happen in there (remember to always activate your virtualenv if you change to a new shell, the PATH changes it does are not persistent to your system); Install aiocoap here using pip as mentioned by @chrysn.

Thanks for the information! I will look into it! :)

[chrysn]

Thanks for clarifying -- I failed to reprduce the OSError (but made a bunch of tests work on Windows in the course of it, so nothing lost :-) ). The aiocoap install docs now do contain a pointer to virtual environments.

@aellwein on the content_format issue: I've extended the tests to have a baseline. If you could come up with anything that indicates a difference between test_opt_roundtripping and test_opt_construction around the encoding tests, that would be greatly appreciated.

[cdh4u]

First, did you use aiocoap==0.4a1? You have that in the setup.py file. I previously modified it to 4b3, and manually did the changes done by @chrysn in the simple6.py file, and then it worked (until I got the Leshan error).

@cdh4u sorry, i forgot to mention some things here. I stumbled upon another issue (chrysn/aiocoap#194) while testing the #11. That's why i had to use aiocoap from master branch, which is not released in PyPI yet. I usually install dependencies in some virtualenv folder while development rather than install them system- or user-wide. It helps keeping the system clean and is very handy if working on multiple Python projects. To install the dependencies in the local virtualenv, i usually do following:

change dir to the project dir (lwm2mclient); run python3 -m venv .venv - this will create an empty virtualenv in .venv folder inside project, the name of the folder doesn't matter, i just use .venv for convenience ; run source .venv/bin/activate or .venv\Scripts\activate.bat (if on Windows) - to activate your virtualenv, this will patch your PATH to point to the .venv, so all Python-related installation will happen in there (remember to always activate your virtualenv if you change to a new shell, the PATH changes it does are not persistent to your system); Install aiocoap here using pip as mentioned by @chrysn.

With the latest aiocoap, lwm2mclient, Leshan (here i used latest released version 1.0.1), i managed to connect and read Location object (at least the attributes i defined in the definition 5 years ago).

(By the way, @aellwein are you sure the content_format changes were necessary? Message(..., content_format=x) should just be a shorthand for passing in the content_format into the .opt object).

@chrysn somehow it didn't work, Leshan claimed that the content format is null, i.e. not set. Then i saw in aiocoap code, that passing opts via kwargs actually does the same, so i changed this. Should i do more in-depth investigation on that? Maybe contribute some test to aiocoap?

I tried with venv, and it worked fine on Windows. venv is really cool :)

I removed 'aiocoap==0.4b3' from setup.py, ran setup.py (to get hexdump), and ran pip install --upgrade "git+https://github.com/chrysn/aiocoap#egg=aiocoap")

[aellwein]

Yes, as I learnt in meanwhile, the way to describe the dependencies via setup.py is rather ancient. Using pip and describing the dependencies in requirements.txt is the modern way. I will do the necessary adaptions in lwm2mclient.

[cdh4u]

Yes, as I learnt in meanwhile, the way to describe the dependencies via setup.py is rather ancient. Using pip and describing the dependencies in requirements.txt is the modern way. I will do the necessary adaptions in lwm2mclient.

With the following modification you can include the git command in the setup.py file:

install_requires=[
    'hexdump==3.3',
    'aiocoap'
],
 dependency_links=[
    'git+https://github.com/chrysn/aiocoap#egg=aiocoap'
]

[aellwein]

Usually this is done other way around, instead of defining the dependencies in setup.py (_installrequires directive can be removed here completely) , a text file is created named by convention requirements.txt. This contains following:

hexdump==3.3
git+https://github.com/chrysn/aiocoap#egg=aiocoap
-e .

The last line installs the current package (lwm2mclient) in the "editable" mode, means you can still do the changes to the source code. Using pip install -r requirements.txt you can install all of the dependencies at once and have a fine grained control of what is installed and how it's done.

[cdh4u]

Usually this is done other way around, instead of defining the dependencies in setup.py (install_requires directive can be removed here completely) , a text file is created named by convention requirements.txt. This contains following: hexdump==3.3 git+https://github.com/chrysn/aiocoap#egg=aiocoap -e .

The last line installs the current package (lwm2mclient) in the "editable" mode, means you can still do the changes to the source code. Using pip install -r requirements.txt you can install all of the dependencies at once and have a fine grained control of what is installed and how it's done.

I first tried by adding the git stuff to the install_requires, but it did not work. But, I might have done something wrong - will try again later :)

[aellwein]

@cdh4u i adapted the installation instructions in the README to use the "modern" way and also made the lwm2mclient to use aiocoap from specific commit to not introduce breaking changes for now. Could you please test the setup as described on your system? Thanks!

[cdh4u]

@cdh4u i adapted the installation instructions in the README to use the "modern" way and also made the lwm2mclient to use aiocoap from specific commit to not introduce breaking changes for now. Could you please test the setup as described on your system? Thanks!

I will try it later.

One comment: in the usage section, would it be good to indicate that MacOS and Windows require the --address option?

[aellwein]

One comment: in the usage section, would it be good to indicate that MacOS and Windows require the --address option?

Done, thanks for suggestion.

[aellwein]

@cdh4u can this one be closed? Did the installation process work as described in the README?