Cmd 004D/device announce: wrong payload length when no rejoin info

tcharp38 commented 3 years ago

I've found the following unexpected "rejoin info" from cmd 004D/device-announce coming from Zigate (FW 3.1c)

Any clues ? I've checked CRC, sounds good and in line with received message.

fairecasoimeme commented 3 years ago

It's weird ... you get this with any device?

Fred

tcharp38 commented 3 years ago

Looks like I get such unexpected "rejoin info" for 004D,

either from Xiaomi temp (square) or luminosity devices
but not from Xiaomi door sensor. I could not see any "device announce" during inclusion

tcharp38 commented 3 years ago

Some other infos during reincluding a Xiaomi lum device.

Zigate returns [2020-11-03 12:23:28][debug] Reçu: "8048000A4C04CF8CDF3C77164B0000" [2020-11-03 12:23:29][debug] Reçu: "004D000D604E6404CF8CDF3C77164B840000" [2020-11-03 12:23:29][debug] Reçu: "80000007D700BE004502A900" [2020-11-03 12:23:29][debug] Reçu: "004D000CD54E6404CF8CDF3C77164B84B4" [2020-11-03 12:23:29][debug] Reçu: "87010005870000040000"

Abeille sees [2020-11-03 12:23:28][debug] Abeille1, Type=8048/Leave indication, ExtAddr=04CF8CDF3C77164B, RejoinStatus=00 [2020-11-03 12:23:29][debug] Abeille1, Type=004d/Device announce, Addr=4E64, ExtAddr=04CF8CDF3C77164B, MACCapa=84, Rejoin=00 [2020-11-03 12:23:29][debug] Abeille1, Type=8000/Status, Status=00-(Success), SQN=BE, PacketType=0045 [2020-11-03 12:23:29][debug] Abeille1, Type=004d/Device announce, Addr=4E64, ExtAddr=04CF8CDF3C77164B, MACCapa=84, Rejoin=B4 [2020-11-03 12:23:29][debug] Abeille1, Type=8701/Route discovery confirm, MACStatus=00 (ZPS_EVENT_NONE->), NwkStatus=00 (ZPS_EVENT_NONE->), Addr=0400

And if I'm correct with Wireshark sync with log I could not see any "rejoin status/info" in packet

and

So looks like this "rejoin" info is not valid but may come from a corruption somewhere. If there is something I can do to help, let me know.

pipiche38 commented 3 years ago

Correct, Xiaomi and some other devices do not send a Rejoin Packet

On 3 Nov 2020, at 13:17, tcharp38 notifications@github.com wrote:

Some other infos during reincluding a Xiaomi lum device.

Zigate returns [2020-11-03 12:23:28][debug] Reçu: "8048000A4C04CF8CDF3C77164B0000" [2020-11-03 12:23:29][debug] Reçu: "004D000D604E6404CF8CDF3C77164B840000" [2020-11-03 12:23:29][debug] Reçu: "80000007D700BE004502A900" [2020-11-03 12:23:29][debug] Reçu: "004D000CD54E6404CF8CDF3C77164B84B4" [2020-11-03 12:23:29][debug] Reçu: "87010005870000040000"

Abeille sees [2020-11-03 12:23:28][debug] Abeille1, Type=8048/Leave indication, ExtAddr=04CF8CDF3C77164B, RejoinStatus=00 [2020-11-03 12:23:29][debug] Abeille1, Type=004d/Device announce, Addr=4E64, ExtAddr=04CF8CDF3C77164B, MACCapa=84, Rejoin=00 [2020-11-03 12:23:29][debug] Abeille1, Type=8000/Status, Status=00-(Success), SQN=BE, PacketType=0045 [2020-11-03 12:23:29][debug] Abeille1, Type=004d/Device announce, Addr=4E64, ExtAddr=04CF8CDF3C77164B, MACCapa=84, Rejoin=B4 [2020-11-03 12:23:29][debug] Abeille1, Type=8701/Route discovery confirm, MACStatus=00 (ZPS_EVENT_NONE->), NwkStatus=00 (ZPS_EVENT_NONE->), Addr=0400

And if I'm correct with Wireshark sync with log I could not see any "rejoin status/info" in packet

https://user-images.githubusercontent.com/35221038/97983914-47d0b880-1dd6-11eb-9238-386fe3d020dd.png and https://user-images.githubusercontent.com/35221038/97984173-aeee6d00-1dd6-11eb-8ccb-73167d285bd8.png So looks like this "rejoin" info is not valid but may come from a corruption somewhere. If there is something I can do to help, let me know.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fairecasoimeme/ZiGate/issues/325#issuecomment-721081703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7IKWQVBFJEMJ4OR3FYYZDSN7YEHANCNFSM4SWQYIFQ.

tcharp38 commented 3 years ago

So makes sense. It's probably a zigate bug then.

tcharp38 commented 3 years ago

Hi guys. Happy new year to all.

How can we proceed to clean such pending bug ? We are still experiencing some unexplained "rejoin info" in "device announce".. still not documented.

0x54 would mean what ? Any clues ?

Abeille1, Type=004d/Device announce, Addr=43E5, ExtAddr=5C0272FFFEC3EA56, MACCapa=80, Rejoin=54

Thanks

pipiche38 commented 3 years ago

@tcharp38 when you get this Rejoin 0x54 what is the value of the LQI in the message ? Would you mind to give the full decoded frame ? For instance what is the length of the frame coming from ZiGate ?

tcharp38 commented 3 years ago

@pipiche38 I don't have these details yet (I'm gonna reproduce & come back to you) for this case but I gave another example (rejoin=B4) on the 3rd of nov with full zigate trace. Does it help or is tracking 0x54 more interesting than 0xB4 ?

tcharp38 commented 3 years ago

Here are 3 captures with received packet from Zigate. Sounds to me it's not a rejoin info but either invalid data or something else since value is quite changing.

[2021-01-05 11:07:31][debug] Abeille1, Type=004d/Device announce, Addr=43E5, ExtAddr=5C0272FFFEC3EA56, MACCapa=80, Rejoin=60, [Modelisation] [2021-01-05 11:07:31][debug] Reçu: "004D000C5543E55C0272FFFEC3EA568060"

[2021-01-05 12:25:48][debug] Abeille1, Type=004d/Device announce, Addr=43E5, ExtAddr=5C0272FFFEC3EA56, MACCapa=80, Rejoin=57, [Modelisation] [2021-01-05 12:25:48][debug] Reçu: "004D000C6243E55C0272FFFEC3EA568057"

[2021-01-05 12:58:28][debug] Abeille1, Type=004d/Device announce, Addr=43E5, ExtAddr=5C0272FFFEC3EA56, MACCapa=80, Rejoin=4B, [Modelisation] [2021-01-05 12:58:28][debug] Reçu: "004D000C7E43E55C0272FFFEC3EA56804B"

pipiche38 commented 3 years ago

My initial was about the value of LQI in the frame received by the plugin . The reason is the following the 0x004D message can be of 2 différents lenght: 11 or 12 bytes and the way to identify which lenght, is based on the LQI value.

LQI = 0x00 ==> Message lenght 12 bytes and Rejoin flag exist LQI != 0x00 ==> Message lenght 11 bytes (no rejoin flag

# There are 2 types of Device Annoucement the plugin can received from firmware >= 31a
# (1) Device Annoucement with a JoinFlags and LQI set to 00. This one could be issued from:
#     - device association (but transaction key not yet exchanged)
#     - Rejoin request (for an already paired devices )
#
# (2) Device Annoucement with a valid LQI and not JoinFlag (shorter message)
#     - Real Device Annoucement on which the plugin should trigger a discovery (if unknown )
#     - Real Device Annoucement for Devices which do not send a Rejoin Request

PS/ For memo LQI is the last byte before 0x03 in the ZiGate received frame

tcharp38 commented 3 years ago

This now reminds me an old point on which I blocked.

If LQI is part of payload, then we have an issue.
- 11 bytes is just enough for the first 3 fields (shortAddr+extAddr+macCapa). In that case there would be neither LQI nor Rejoin.
- 12 bytes, would mean LQI=4B in the "004D000C7E43E55C0272FFFEC3EA56804B" example.
If LQI is NOT part of payload, we have another issue. First LQI should be present on ALL received messages (is that correct ?). But in the last 3 examples I gave above (for ex received 004D000C7E43E55C0272FFFEC3EA56804B), zigate says "12 bytes of payload" which is the case but there is more bytes after (so NO LQI) and "4B" is still there .. for ?

Do you see why I'm lost or there is something big in front of me which makes me blind ?

pipiche38 commented 3 years ago

@tcharp38 as shown in the 'Trame ZiGate' every frame sent by ZiGate and received by the plugin has the same format

It starts with 0x01 and finished with 0x03, in between you have Msgt Type, Lenght, Chksum, Data (Payload), LQI

And yes LQI is present on ALL messages received.

You probably need to check on your plugin, it might be possible that when reading and decoding the frame, some elements like LQI is not sent to the level above.

tcharp38 commented 3 years ago

I will investigate this clue and come back on that. Thanks @pipiche38

tcharp38 commented 3 years ago

Here is the trace of a Xiaomi temp inclusion... 2 consecutives device announces

[2021-01-08 17:37:43] Type=004d/Device announce, Addr=67E3, ExtAddr=00158D0004657079, MACCapa=80, Rejoin=00 [2021-01-08 17:37:44] Type=004d/Device announce, Addr=67E3, ExtAddr=00158D0004657079, MACCapa=80, Rejoin=CC

Let's focus on 2nd one which is the weird one

[2021-01-08 17:37:44] Raw: 0102104D0210021C7967E30210158D0210021465707980CC03 01 => Start of frame 02104D => 004D 0210021C => 000C 79 => CRC 67E3 => Short addr 0210158D02100214657079 => 00158D0004657079 = IEEE 80 => Mac CC => ?? If LQI, then payload size is wrong. 03 => End of frame

As a conclusion my questions are still there and I don't understand this frame.

pipiche38 commented 3 years ago

01/02104D/0210021C7967E30210158D0210021465707980CC03

Here is how I decode this frame

Start Frame: 01 MsgType: 004D Lenght Payload: 000C - 12 Bytes CRC: 79

Payload: NwkId: 67E3 ( 2 Bytes ) IEEE: 00158D0004657079 ( 8 bytes ) Maccapa: 80 ( 1 byte ) LQI: CC ( 1 byte )

End Frame: 03

the last byte of the frame before 0x03 is always LQI.

So you are in the case of a True Device annoucement where there is no Rejoin Flag in the message. Device Annoucement with a valid LQI and not JoinFlag (shorter message)

tcharp38 commented 3 years ago

You decode is in line with mine. BUT ... what about payload length ? Agree it is wrong ? You told me LQI was not part of payload, then I would expect 11 bytes for this message.

I did a 10sec+ press on my Xiaomi temp sensor. Then reset + full rejoin isn't it ? I've see 2 "dev announce" during that process (first with rejoin=00, second which the case under debat).

pipiche38 commented 3 years ago

You decode is in line with mine. BUT ... what about payload length ? Agree it is wrong ? You told me LQI was not part of payload, then I would expect 11 bytes for this message.

I did a 10sec+ press on my Xiaomi temp sensor. Then reset + full rejoin isn't it ? I've see 2 "dev announce" during that process (first with rejoin=00, second which the case under debate).

I tend to agree as regards to the payload length. But will not comment as this is implemented like that .

If you have received 2 Device Annoucements 1 with LQI and Rejoing = 00 1 with LQI and no Rejoin

this correspond to what I mentioned in the post here : https://github.com/fairecasoimeme/ZiGate/issues/325#issuecomment-754620240 (which is what @fairecasoimeme told me; and how the firmware behave )

tcharp38 commented 3 years ago

Ok then. Im gonna rename this issue as a real bug on payload length in this case. And see what trick to put on our side to not trust payload length to not wrongly interpret a "rejoin info".

Hope we are now in line Thanks

tcharp38 commented 3 years ago

@pipiche38 1 last point. Since payload length is clearly wrong, does it mean that your SW is not using it for this case ? According to your display, it looks like you consider LQI included in "payload length".

pipiche38 commented 3 years ago

I leave it to @fairecasoimeme , you can consider it is a bug, however it is like that from the 1st version of the firmware and I doubt that you can change it now.

fairecasoimeme commented 3 years ago

Hi, LQI is a part of payload. the problem is there a command number with 2 structures. But the length is always OK. To solve the issue, we could add a 0x00 (which will be always 0x00) to the intern device announce packet.

We could change the command number for one of 0x4d command too ...

What do you think about that ?

tcharp38 commented 3 years ago

I reached the same conclusion this morning; length = data size + 1 byte for LQI

There may finally be no pb except the following comment from pipiche

My initial was about the value of LQI in the frame received by the plugin . The reason is the following the 0x004D message can be of 2 différents lenght: 11 or 12 bytes and the way to identify which lenght, is based on the LQI value.

LQI = 0x00 ==> Message lenght 12 bytes and Rejoin flag exist LQI != 0x00 ==> Message lenght 11 bytes (no rejoin flag

My understanding: 2 messages format

The one with LQI only, no rejoin => length = 12 bytes
The one with LQI & rejoin => length = 13 bytes

I need to confirm this but sounds more a misunderstanding linked to comment above and not a real bug now.

fairecasoimeme commented 3 years ago

yes, this is 12 bytes and 13 bytes --> 0x0c and 0x0d

fairecasoimeme / ZiGate

Cmd 004D/device announce: wrong payload length when no rejoin info #325