Closed andyka closed 7 years ago
Hey @andyka thanks for the detailed issue! Based on a first glance I'm not too sure what is causing it, maybe @bp1222 or @jamesleesaunders have a better idea?
Hi @andyka Thanks for the great examples - one of the most detailed issues I have seen :-) I have not had a chance to fully look into this issue but it does looks very strange - I will see if I get the same affects with one of my Xbee's.
Coincidently you may be interested to know that I have progressed the great work which Desert Home @draythomp made, you may be interested in my project 'PyAlertMe' at: https://github.com/jamesleesaunders/PyAlertMe/ PyAlertMe is a set of classes which, when used in conjunction with a Digi XBee (ZigBee) module, can be used to simulate an AlertMe (Lowes Iris, Hive, British Gas Safe and Secure) Hub, SmartPlug or Sensor.
In particular you may be interested in this file: https://github.com/jamesleesaunders/PyAlertMe/blob/master/pyalertme/zbnode.py And associated test: https://github.com/jamesleesaunders/PyAlertMe/blob/master/tests/test_zbnode.py Although still not very polished this file attempts to document some of the AlertMe HA and ZDO Frame definitions. I would be SUPER interested if you have any comments or would like to help me out!
Personal plug over! I have not noticed the anomalies you describe in this issue but like you I have done a little digging into the ZDO spec and only just started adding them to PyAlertMe, I will get back to you if/when I attempt to recreate your issue.
Hi all! Thanks, I had no idea which part of the information might be crucial for solving this, so I went a little overboard with description... But I know that detailed bug report helps with solving the issue. :)
@jamesleesaunders Your code looks really nice! I don't have any AlertMe devices myself, just HA and LL. But it makes me think... Maybe it might be smart to put profile specific code in classes (ZBProfileAlertMe with AlertMe, ZBProfileHA deals with HA,...)? I see you are mixing "at" and "tx_explicit" messages - I had no idea one can do that, thanks! This will make my life a bit easier... :)
Yes, I think @draythomp made a real breakthrough here. Without his desert home write-up I probably wouldn't be able to get this far in such a short time.
I see you are using escaped=True
here:
self._xbee = ZigBee(ser=self._serial, callback=self.receive_message, error_callback=self.xbee_error, escaped=True)
I tried this value before, but I couldn't communicate with my XBee then, so I am using escaped=False
. I see now that I could send ATAP=2
to use escaped API (API 2), so I'll try that too. (EDIT: tried it, it doesn't fix this issue)
On a side note, I find it is much easier for me if I log all messages in a format where all characters are converted to \x
form (even if they could be written as ascii). I was bitten by \x802
(which is really \x80\x32
). I made a short and dirty function for this, feel free to use anyway you want to:
def _repr_hex(data, except_keys=None):
""" except_keys: list of strings; dictionary keys which should be exempt from escaping
to hex. Useful when you wish to display some values as strings. """
if isinstance(data, basestring):
return "'" + ''.join( [ "\\x%02X" % ord( x ) for x in data ] ) + "'"
elif isinstance(data, dict):
ret = ''
for k, v in data.iteritems():
if ret:
ret += ", "
if except_keys and k in except_keys:
ret += repr(k) + ": " + repr(v)
else:
ret += repr(k) + ": " + _repr_hex(v)
return '{' + ret + '}'
else:
return repr(data)
# Test:
# print _repr_hex({"a": "\x01\x02", "b": "abc", "c": "let_me_be"}, except_keys=["c"])
# Output:
# {'a': '\x01\x02', 'b': '\x61\x62\x63', 'c': 'let_me_be'}
# Use it like this:
# self._logger.debug('Received Message: %s', _repr_hex(message))
Note that it is Python2 only (because of the way it checks if data
is string).
If you could try with one of your own devices, that would be great... I have put together an example which queries only controller, but it should be enough to see if it's just my device which is weird: https://gist.github.com/andyka/6932bb750df460a7db4952313dfc6c7c
I tried downgrading firmware to 21A0
, no change. Resetting config to factory defaults, no change. Downgrading to 218C
, no change. I am running out of ideas... :-/
Could someone please try running the script on XStick ZB and then post the resulting output (or at least rf_data
field for response with cluster \x80\x05
)?
@jamesleesaunders I tried using your code, but I am not sure how to use it on sensors I have... I am guessing that SmartPlug
is AlertMe smart plug exclusively? Do you have plans for this library to support other devices too?
Hi @andyka I am heading off on holiday this morning for a week but before I headed off I just gave your script a quick whirl, here was my output...
MacBook-Pro:test jim$ python ./bug.py
RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x5A\x53', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x41\x50', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x41\x4F', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x45\x45', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
RECEIVED: {'status': '\x00', 'frame_id': '\x01', 'command': '\x45\x4F', 'id': '\x61\x74\x5F\x72\x65\x73\x70\x6F\x6E\x73\x65'}
Sending request: get nodes list, but only ask for controller
RECEIVED: {'profile': '\x00\x00', 'source_addr': '\x6B\xB3', 'dest_endpoint': '\x00', 'rf_data': '\x00\x00\x28\x3C\x00', 'source_endpoint': '\x00', 'options': '\x01', 'source_addr_long': '\x00\x13\xA2\x00\x40\xA2\x3B\x09', 'cluster': '\x80\x32', 'id': '\x72\x78\x5F\x65\x78\x70\x6C\x69\x63\x69\x74'}
RECEIVED: {'profile': '\x00\x00', 'source_addr': '\xF9\xAB', 'dest_endpoint': '\x00', 'rf_data': '\x00\x00\x14\xC0\x00', 'source_endpoint': '\x00', 'options': '\x01', 'source_addr_long': '\x00\x0D\x6F\x00\x03\xBB\xB9\xF8', 'cluster': '\x80\x32', 'id': '\x72\x78\x5F\x65\x78\x70\x6C\x69\x63\x69\x74'}
Sending request: get node endpoints (just for controller)
Exception: TypeError("object of type 'NoneType' has no len()",)
Traceback (most recent call last):
File "./bug.py", line 100, in <module>
data = controller_addr_short[1] + controller_addr_short[0]
File "build/bdist.macosx-10.6-intel/egg/xbee/backend/base.py", line 340, in send
File "build/bdist.macosx-10.6-intel/egg/xbee/backend/base.py", line 131, in _build_command
TypeError: object of type 'NoneType' has no len()
MacBook-Pro:test jim$
Not had chance to debug why fails and haven't had time to digest the output but hopefully this may help you/us? This was running on XBee XB24-Z7PIT-004 ZigBee Stack Profile (ZS): 2 Encryption Enable (EE): 1 Encryption Options (EO): 0 Encryption Key (KY): None API Enable (AP): 2 API Output Mode (AO): 3
Sorry this is a bit rushed - But I do really want to work with you (as I am hoping maybe you might be able to assist me with PyAlertMe). I may have a look while on holiday, else it may be a week before you hear from me.
p.s Thanks for the _repr_hex() function! Very useful like you I have also been confused at times by the \x802 => \x80\x32 thing!
p.p.s PyAlertMe... I do need to document the library more but yes the SmartPlug example currently just emulates a AlertMe/Lowes Iris SmartPlug, but yes, I would love to make this library a little more generic and emulate standard HW devices also. I am not a Python expert so need some help getting getting the structure of these files/classes in a more appropriate shape.
Right, must pack the car now...
Jim
Hi @jamesleesaunders, thanks for running the script! It looks like sometimes controller doesn't answer in time, so running it again would exhibit the behavior.
But I think I know what was wrong. I had two bugs, which meant that changing just one thing didn't improve anything:
AO
was set to 3 instead of 1 (ZDO specs mention that AO should be 1)With this fixed, I now get:
REQUEST (tx_explicit): {'profile': '\x00\x00', 'dest_addr_long': '\x00\x13\xA2\x00\x40\xA3\xB7\x2F', 'dest_endpoint': '\x00', 'src_endpoint': '\x00', 'cluster': '\x00\x05', 'dest_addr': '\x00\x00', 'data': '\xAB\x00\x00'}
RECEIVED: {'profile': '\x00\x00', 'source_addr': '\x00\x00', 'dest_endpoint': '\x00', 'rf_data': **'\xAB\x00\x00\x00\x02\xE6\xE8'**, 'source_endpoint': '\x00', 'options': '\x01', 'source_addr_long': '\x00\x13\xA2\x00\x40\xA3\xB7\x2F', 'cluster': '\x80\x05', 'id': 'rx_explicit'}
This now makes sense:
\xAB
is the sequence number I supplied in request\x00
is status (presumably "ok", though I couldn't find a reference except for this)\x00\x00
is controller address (always 0x0000)\x02
is number of endpoints\xE6\xE8
are two endpointsActually, if I had followed XBee User Guide more closely I would have noticed that the ZDO frames have the sequence number at the start (note that ZCL header is different - 3 to 5 bytes, as outlined in ZCL specs). But in my defense, code from desert-home has the same bug... ;)
The "invalid" rf_data
in responses now makes sense. First byte is whatever I supplied and next one is status (80 is "invalid request", 81 is "device not found"). What follows is the rest of my response and some additional data, depending on status.
So, I think this is solved! I want to thank you all for your responses and for your kind help!
@jamesleesaunders I would love to keep in touch, maybe I can help out with the library. I'll send you an e-mail, in the mean time enjoy your holiday! :)
@jamesleesaunders Just letting you know that I have sent you an e-mail (but received no response yet), in case it went to spam. GitHub lacks PMs... :) Have fun!
Hi @andyka Got your email thanks and will reply soon. Good news you got things working - glad you figured out the problem :-)
Regarding the ZDO 'Status' - I too was on a little paper chase trying to find these, If any use I did in the end find them in this document: http://www.cel.com/pdf/misc/zic09_zdp_api.pdf (Page 7, Section 3.2.4).
I am sure you have already read it but this is also a good doc: http://ftp1.digi.com/support/images/APP_NOTE_XBee_ZigBee_Device_Profile.pdf
Catch you on email!
I later found the statuses in ZCL specs: http://www.zigbee.org/wp-content/uploads/2014/10/07-5123-06-zigbee-cluster-library-specification.pdf (see chapter "2.6.3 Status Enumerations")
Thanks for the XBee docs! I also found these very informative because they show the packets exactly as they are, which helps debug the ZCL packets: (especially with regards to little/big endian) https://eewiki.net/download/attachments/24313921/XBee_ZB_User_Guide.pdf
Have fun!
First of all, thank you for a great library, it is very easy to use!
I am trying to communicate with ZigBee devices via XBee XStick2 ZB. Communication works perfectly, but the data I receive in
rf_data
field makes no sense (according to ZDO). After additional investigation I figured out that the bytes might be there, but they seem to be rearranged - and not in little endian / big endian kind of way (as far as I can tell).The easiest example to see what is going on is when I send a request with cluster 0x0005. I get response with cluster 0x8005 as expected:
All other fields look OK, but
rf_data
has this value:ZDO documentation says that the fields should be:
Active Endpoints Response Cluster ID: 0x8005 Description: Indicates the list of active endpoints supported on the device.
endpoint.
Network Address in this case should be the same as in field
source_addr
, that is:\xD2\xDD
. Note that I have tried this command with different devices and I always get the bytes fromsource_addr
in the same places ofrf_data
(0 and 2). For instance:So,
source_addr
is there but on wrong address, and I am missing value\x01
(Active Endpoint Count). My guess is that there is either a problem with my USB stick or with parsingrf_data
field (more likely?), but I was unable to find the reason (would need to know rx_explicit format to see what is going on).I have also checked the code from http://www.desert-home.com/2014/10/ok-back-to-zigbee-protocol-and-xbees.html but unfortunately I don't have a ZigBee switch at hand. It seems that author (and commenters) had some problems with inexplicable bytes in rf_data too, though not as much as myself.
I would appreciate some pointers...
(installed via
pip install xbee
)EDIT: Upon further examination I discovered that the problem is not in parsing, but in the raw response as it is passed to
XBeeBase._split_response()
. For instance:Last 5 bytes of raw response data already exhibit the issue with mangled bytes. Is this a problem with firmware? If so, which FW should I upgrade to? Latest? Thanks!
EDIT2: I have checked firmware version via XCTU and it is
21A7
, which is latest "ZigBee Coordinator API" firmware. I have upgraded anyway, but there is no change. Also, XCTU doesn't find any ZigBee devices (though xbee library does, albeit with problems as outlined above).