search

niolabs / python-xbee

Python tools for working with XBee radios
MIT License
101 stars 45 forks source link

bytes rearranged in rf_data for rx_explicit #55

Closed andyka closed 7 years ago

andyka commented 7 years ago

First of all, thank you for a great library, it is very easy to use!

I am trying to communicate with ZigBee devices via XBee XStick2 ZB. Communication works perfectly, but the data I receive in rf_data field makes no sense (according to ZDO). After additional investigation I figured out that the bytes might be there, but they seem to be rearranged - and not in little endian / big endian kind of way (as far as I can tell).

The easiest example to see what is going on is when I send a request with cluster 0x0005. I get response with cluster 0x8005 as expected:

RESPONSE: {'profile': '\x00\x00', 'source_addr': '\xD2\xDD', 'dest_endpoint': '\x00', 'rf_data': '\xDD\x80\xD2\x05\x00', 'source_endpoint': '\x00', 'options': '\x41', 'source_addr_long': '\x00\x50\x43\xC9\xA3\x36\x8B\x61', 'cluster': '\x80\x05', 'id': 'rx_explicit'}

All other fields look OK, but rf_data has this value:

\xDD\x80\xD2\x05\x00

ZDO documentation says that the fields should be:

Active Endpoints Response Cluster ID: 0x8005 Description: Indicates the list of active endpoints supported on the device.

Field Name Size (bytes) Description
Status 1
Network Address 2 Indicates the 16-bit address of the responding device
Active Endpoint Count 1 Number of endpoints in the following endpoint list
Active Endpoint List Variable List of endpoints supported on the destination device. One byte per

endpoint.

Network Address in this case should be the same as in field source_addr, that is: \xD2\xDD. Note that I have tried this command with different devices and I always get the bytes from source_addr in the same places of rf_data (0 and 2). For instance:

source_addr rf_data
'\x00\x00' '\x00\x81\x00\x7E'
'\xD2\xDD' '\xDD\x80\xD2\x05\x00'
'\x6E\xF7' '\xF7\x80\x6E\x05\x00'
'\x9C\x81' '\x81\x80\x9C\x05\x00'
... ...

So, source_addr is there but on wrong address, and I am missing value \x01 (Active Endpoint Count). My guess is that there is either a problem with my USB stick or with parsing rf_data field (more likely?), but I was unable to find the reason (would need to know rx_explicit format to see what is going on).

I have also checked the code from http://www.desert-home.com/2014/10/ok-back-to-zigbee-protocol-and-xbees.html but unfortunately I don't have a ZigBee switch at hand. It seems that author (and commenters) had some problems with inexplicable bytes in rf_data too, though not as much as myself.

I would appreciate some pointers...

>>> xbee.__version__
'2.2.5'

(installed via pip install xbee)

EDIT: Upon further examination I discovered that the problem is not in parsing, but in the raw response as it is passed to XBeeBase._split_response(). For instance:

RESPONSE RAW DATA: '\x91\x00\x50\x43\xC9\xA3\x36\x8B\x61\x65\xAC\x00\x00\x80\x05\x00\x00\x41\xAC\x80\x65\x05\x00'
RESPONSE: {'profile': '\x00\x00', 'source_addr': '\x65\xAC', 'dest_endpoint': '\x00', 'rf_data': '\xAC\x80\x65\x05\x00', 'source_endpoint': '\x00', 'options': '\x41', 'source_addr_long': '\x00\x50\x43\xC9\xA3\x36\x8B\x61', 'cluster': '\x80\x05', 'id': 'rx_explicit'}

Last 5 bytes of raw response data already exhibit the issue with mangled bytes. Is this a problem with firmware? If so, which FW should I upgrade to? Latest? Thanks!

EDIT2: I have checked firmware version via XCTU and it is 21A7, which is latest "ZigBee Coordinator API" firmware. I have upgraded anyway, but there is no change. Also, XCTU doesn't find any ZigBee devices (though xbee library does, albeit with problems as outlined above).

mattdodge commented 7 years ago

Hey @andyka thanks for the detailed issue! Based on a first glance I'm not too sure what is causing it, maybe @bp1222 or @jamesleesaunders have a better idea?

jamesleesaunders commented 7 years ago

Hi @andyka Thanks for the great examples - one of the most detailed issues I have seen :-) I have not had a chance to fully look into this issue but it does looks very strange - I will see if I get the same affects with one of my Xbee's.

Coincidently you may be interested to know that I have progressed the great work which Desert Home @draythomp made, you may be interested in my project 'PyAlertMe' at: https://github.com/jamesleesaunders/PyAlertMe/ PyAlertMe is a set of classes which, when used in conjunction with a Digi XBee (ZigBee) module, can be used to simulate an AlertMe (Lowes Iris, Hive, British Gas Safe and Secure) Hub, SmartPlug or Sensor.

In particular you may be interested in this file: https://github.com/jamesleesaunders/PyAlertMe/blob/master/pyalertme/zbnode.py And associated test: https://github.com/jamesleesaunders/PyAlertMe/blob/master/tests/test_zbnode.py Although still not very polished this file attempts to document some of the AlertMe HA and ZDO Frame definitions. I would be SUPER interested if you have any comments or would like to help me out!

Personal plug over! I have not noticed the anomalies you describe in this issue but like you I have done a little digging into the ZDO spec and only just started adding them to PyAlertMe, I will get back to you if/when I attempt to recreate your issue.

andyka commented 7 years ago

Hi all! Thanks, I had no idea which part of the information might be crucial for solving this, so I went a little overboard with description... But I know that detailed bug report helps with solving the issue. :)

@jamesleesaunders Your code looks really nice! I don't have any AlertMe devices myself, just HA and LL. But it makes me think... Maybe it might be smart to put profile specific code in classes (ZBProfileAlertMe with AlertMe, ZBProfileHA deals with HA,...)? I see you are mixing "at" and "tx_explicit" messages - I had no idea one can do that, thanks! This will make my life a bit easier... :)

Yes, I think @draythomp made a real breakthrough here. Without his desert home write-up I probably wouldn't be able to get this far in such a short time.

I see you are using escaped=True here:

self._xbee = ZigBee(ser=self._serial, callback=self.receive_message, error_callback=self.xbee_error, escaped=True)

I tried this value before, but I couldn't communicate with my XBee then, so I am using escaped=False. I see now that I could send ATAP=2 to use escaped API (API 2), so I'll try that too. (EDIT: tried it, it doesn't fix this issue)

On a side note, I find it is much easier for me if I log all messages in a format where all characters are converted to \x form (even if they could be written as ascii). I was bitten by \x802(which is really \x80\x32). I made a short and dirty function for this, feel free to use anyway you want to:

def _repr_hex(data, except_keys=None):
    """ except_keys: list of strings; dictionary keys which should be exempt from escaping 
        to hex. Useful when you wish to display some values as strings. """
    if isinstance(data, basestring):
        return "'" + ''.join( [ "\\x%02X" % ord( x ) for x in data ] ) + "'"
    elif isinstance(data, dict):
        ret = ''
        for k, v in data.iteritems():
            if ret:
                ret += ", "
            if except_keys and k in except_keys:
                ret += repr(k) + ": " + repr(v)
            else:
                ret += repr(k) + ": " + _repr_hex(v)
        return '{' + ret + '}'
    else:
        return repr(data)

# Test:
#   print _repr_hex({"a": "\x01\x02", "b": "abc", "c": "let_me_be"}, except_keys=["c"])
# Output:
#   {'a': '\x01\x02', 'b': '\x61\x62\x63', 'c': 'let_me_be'}

# Use it like this:
#   self._logger.debug('Received Message: %s', _repr_hex(message))

Note that it is Python2 only (because of the way it checks if data is string).

If you could try with one of your own devices, that would be great... I have put together an example which queries only controller, but it should be enough to see if it's just my device which is weird: https://gist.github.com/andyka/6932bb750df460a7db4952313dfc6c7c

andyka commented 7 years ago

I tried downgrading firmware to 21A0, no change. Resetting config to factory defaults, no change. Downgrading to 218C, no change. I am running out of ideas... :-/

Could someone please try running the script on XStick ZB and then post the resulting output (or at least rf_data field for response with cluster \x80\x05)?

@jamesleesaunders I tried using your code, but I am not sure how to use it on sensors I have... I am guessing that SmartPlug is AlertMe smart plug exclusively? Do you have plans for this library to support other devices too?

jamesleesaunders commented 7 years ago

Hi @andyka I am heading off on holiday this morning for a week but before I headed off I just gave your script a quick whirl, here was my output...

MacBook-Pro:test jim$ python ./bug.py 
    RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x5A\x53', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
    RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x41\x50', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
    RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x41\x4F', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
    RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x45\x45', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
    RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x45\x4F', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
Sending request: get nodes list, but only ask for controller
    RECEIVED: {'profile': '\x00\x00', 'source_addr': '\x6B\xB3', 'dest_endpoint': '\x00', 'rf_data': '\x00\x00\x28\x3C\x00', 'source_endpoint': '\x00', 'options': '\x01', 'source_addr_long': '\x00\x13\xA2\x00\x40\xA2\x3B\x09', 'cluster': '\x80\x32', 'id': '\x72\x78\x5F\x65\x78\x70\x6C\x69\x63\x69\x74'}
    RECEIVED: {'profile': '\x00\x00', 'source_addr': '\xF9\xAB', 'dest_endpoint': '\x00', 'rf_data': '\x00\x00\x14\xC0\x00', 'source_endpoint': '\x00', 'options': '\x01', 'source_addr_long': '\x00\x0D\x6F\x00\x03\xBB\xB9\xF8', 'cluster': '\x80\x32', 'id': '\x72\x78\x5F\x65\x78\x70\x6C\x69\x63\x69\x74'}
Sending request: get node endpoints (just for controller)
Exception: TypeError("object of type 'NoneType' has no len()",)
Traceback (most recent call last):
  File "./bug.py", line 100, in <module>
    data = controller_addr_short[1] + controller_addr_short[0]
  File "build/bdist.macosx-10.6-intel/egg/xbee/backend/base.py", line 340, in send
  File "build/bdist.macosx-10.6-intel/egg/xbee/backend/base.py", line 131, in _build_command
TypeError: object of type 'NoneType' has no len()
MacBook-Pro:test jim$

Not had chance to debug why fails and haven't had time to digest the output but hopefully this may help you/us? This was running on XBee XB24-Z7PIT-004 ZigBee Stack Profile (ZS): 2 Encryption Enable (EE): 1 Encryption Options (EO): 0 Encryption Key (KY): None API Enable (AP): 2 API Output Mode (AO): 3

Sorry this is a bit rushed - But I do really want to work with you (as I am hoping maybe you might be able to assist me with PyAlertMe). I may have a look while on holiday, else it may be a week before you hear from me.

p.s Thanks for the _repr_hex() function! Very useful like you I have also been confused at times by the \x802 => \x80\x32 thing!

p.p.s PyAlertMe... I do need to document the library more but yes the SmartPlug example currently just emulates a AlertMe/Lowes Iris SmartPlug, but yes, I would love to make this library a little more generic and emulate standard HW devices also. I am not a Python expert so need some help getting getting the structure of these files/classes in a more appropriate shape.

Right, must pack the car now...

Jim

andyka commented 7 years ago

Hi @jamesleesaunders, thanks for running the script! It looks like sometimes controller doesn't answer in time, so running it again would exhibit the behavior.

But I think I know what was wrong. I had two bugs, which meant that changing just one thing didn't improve anything:

With this fixed, I now get:

REQUEST (tx_explicit): {'profile': '\x00\x00', 'dest_addr_long': '\x00\x13\xA2\x00\x40\xA3\xB7\x2F', 'dest_endpoint': '\x00', 'src_endpoint': '\x00', 'cluster': '\x00\x05', 'dest_addr': '\x00\x00', 'data': '\xAB\x00\x00'}
RECEIVED: {'profile': '\x00\x00', 'source_addr': '\x00\x00', 'dest_endpoint': '\x00', 'rf_data': **'\xAB\x00\x00\x00\x02\xE6\xE8'**, 'source_endpoint': '\x00', 'options': '\x01', 'source_addr_long': '\x00\x13\xA2\x00\x40\xA3\xB7\x2F', 'cluster': '\x80\x05', 'id': 'rx_explicit'}

This now makes sense:

Actually, if I had followed XBee User Guide more closely I would have noticed that the ZDO frames have the sequence number at the start (note that ZCL header is different - 3 to 5 bytes, as outlined in ZCL specs). But in my defense, code from desert-home has the same bug... ;)

The "invalid" rf_data in responses now makes sense. First byte is whatever I supplied and next one is status (80 is "invalid request", 81 is "device not found"). What follows is the rest of my response and some additional data, depending on status.

So, I think this is solved! I want to thank you all for your responses and for your kind help!

@jamesleesaunders I would love to keep in touch, maybe I can help out with the library. I'll send you an e-mail, in the mean time enjoy your holiday! :)

andyka commented 6 years ago

@jamesleesaunders Just letting you know that I have sent you an e-mail (but received no response yet), in case it went to spam. GitHub lacks PMs... :) Have fun!

jamesleesaunders commented 6 years ago

Hi @andyka Got your email thanks and will reply soon. Good news you got things working - glad you figured out the problem :-)

Regarding the ZDO 'Status' - I too was on a little paper chase trying to find these, If any use I did in the end find them in this document: http://www.cel.com/pdf/misc/zic09_zdp_api.pdf (Page 7, Section 3.2.4).

I am sure you have already read it but this is also a good doc: http://ftp1.digi.com/support/images/APP_NOTE_XBee_ZigBee_Device_Profile.pdf

Catch you on email!

andyka commented 6 years ago

I later found the statuses in ZCL specs: http://www.zigbee.org/wp-content/uploads/2014/10/07-5123-06-zigbee-cluster-library-specification.pdf (see chapter "2.6.3 Status Enumerations")

Thanks for the XBee docs! I also found these very informative because they show the packets exactly as they are, which helps debug the ZCL packets: (especially with regards to little/big endian) https://eewiki.net/download/attachments/24313921/XBee_ZB_User_Guide.pdf

Have fun!