search

jlusiardi / homekit_python

A python implementation to work as both HomeKit controller and accessory.
Apache License 2.0
221 stars 41 forks source link

Pairing Issues: Eve Energy 1EC0 / LG C9 stuck at #5 ios -> accessory: send SRP exchange request #170

Open jlusiardi opened 5 years ago

jlusiardi commented 5 years ago

In #125 and #158 there are reports that Eve Energy 1EC0 and LG C9 are both stuck at the same problem. This seems to influence both transports.

Citing @jc2k:

To me, this looks like a completely normal M1/M2/M3 exchange, then the TV abruptly returns nothing / closes the connection instead of sending us an M4. There are no extra fields or new reserved flags suddenly being used in the M2 to indicate we should behave differently. I wondered if they rolled out a new pairing scheme (different salt? pub key?) There is no new spec, though. That likely means it failed the SRP proof? But that kind of error should (according to the spec) yield an M4 response with a kTLVType_Error.

python3 -m homekit.pair -d XX:XX:XX:XX:XX:XX -a mytv -f pairing.json --log DEBUG
2019-08-27 05:57:08,812 __init__.py:0106 DEBUG #1 ios -> accessory: send SRP start request
2019-08-27 05:57:08,813 tlv.py:0134 DEBUG sending [
  6: (1 bytes/<class 'bytearray'>) 0x01
  0: (1 bytes/<class 'bytearray'>) 0x01
]

2019-08-27 05:57:08,813 tlv.py:0117 DEBUG receiving [
  6: (1 bytes/<class 'bytearray'>) 0x01
  0: (1 bytes/<class 'bytearray'>) 0x01
]

2019-08-27 05:57:08,813 __init__.py:0060 DEBUG write message: [
  6: (1 bytes/<class 'bytearray'>) 0x01
  0: (1 bytes/<class 'bytearray'>) 0x01
]

2019-08-27 05:57:09,023 tlv.py:0117 DEBUG receiving [
  6: (1 bytes/<class 'bytearray'>) 0x02
  3: (384 bytes/<class 'bytearray'>) 0xa4716b8b96d60a92ff6c28c7d00085cd420aaccea34530f300c42f0d3776bc21ee2885e3abcaf2ef4d8466b45801080968384833463ee96ac72edcff2b3b7d506aa16d50524b203874e52f4ca4ff807d24c39c1807ad95f28c12788a930afe8f33aa19fafe42f99f481d9adae48341c75ce4d1e65b2c9f7a34a6e0bdbe0cecfcb017c373557adb7a5d8d470edea65913b877924b92721774b4e325c9c7e3bb988c8cf366308393f4b1efa70a29feff34587079bf103227e8786a9cd0a89ef871ae17ba84d03c9e88d1929ea4df010cd50bbec6a108b582d3c3123b9b81a15aee636fb7f9d1a637c775966d0331f2338eb600f1e22c0f2b46eb27eec9cc80af12d3ad28de55473283a4673ed661f929d7053318720e00ec233725f89b795a3ef393f441eddf1e6998dc31e032e16a53c8052eec5b3699feb3eb3fac89ffdc7d30bc1bd4f47d7779c0ee3736280785053b0a0e32fb272e9c4972e25d6c73349fb43c1eada186ce0c539b8648858b185880fa6195590d9519f70af5eed1832978e3
  2: (16 bytes/<class 'bytearray'>) 0x691039f6b6c8067498ff6166cc336825
]

2019-08-27 05:57:09,024 __init__.py:0067 DEBUG response: [
  6: (1 bytes/<class 'bytearray'>) 0x02
  3: (384 bytes/<class 'bytearray'>) 0xa4716b8b96d60a92ff6c28c7d00085cd420aaccea34530f300c42f0d3776bc21ee2885e3abcaf2ef4d8466b45801080968384833463ee96ac72edcff2b3b7d506aa16d50524b203874e52f4ca4ff807d24c39c1807ad95f28c12788a930afe8f33aa19fafe42f99f481d9adae48341c75ce4d1e65b2c9f7a34a6e0bdbe0cecfcb017c373557adb7a5d8d470edea65913b877924b92721774b4e325c9c7e3bb988c8cf366308393f4b1efa70a29feff34587079bf103227e8786a9cd0a89ef871ae17ba84d03c9e88d1929ea4df010cd50bbec6a108b582d3c3123b9b81a15aee636fb7f9d1a637c775966d0331f2338eb600f1e22c0f2b46eb27eec9cc80af12d3ad28de55473283a4673ed661f929d7053318720e00ec233725f89b795a3ef393f441eddf1e6998dc31e032e16a53c8052eec5b3699feb3eb3fac89ffdc7d30bc1bd4f47d7779c0ee3736280785053b0a0e32fb272e9c4972e25d6c73349fb43c1eada186ce0c539b8648858b185880fa6195590d9519f70af5eed1832978e3
  2: (16 bytes/<class 'bytearray'>) 0x691039f6b6c8067498ff6166cc336825
]

2019-08-27 05:57:09,024 __init__.py:0118 DEBUG #3 ios -> accessory: send SRP verify request
Enter device pin (XXX-YY-ZZZ): 739-34-736
2019-08-27 05:57:33,755 tlv.py:0134 DEBUG sending [
  6: (1 bytes/<class 'bytearray'>) 0x03
  3: (384 bytes/<class 'bytearray'>) 0x73520bb9c3cd416bdf68955569dd1fb506dd851b8ce3ae34a4dfcc2f48edf953bfceccb9ea02fcf10cab8d24cc94b50835b23672debf0a4bf2b35dc030627c8e7060cd1414af3d89e0fc060aa432924a3c18fbf4b77bb703937fb3b55c7c50469b641ee21c532620a20ed7b5ee1f714fe24a33c988ea63f33e3749f0fe23335fe919ba9bec6c2ffbd911f4ee9d970e2ca1225a69350b403168eae727b71af6ad573b9a7f145a441c6da73b1d29c934598d46debd44d6696eb935d98ee0f0db298f051a487ee37334a77538e5668718a9aeb8121603f9dedbddbfa129a7f153fdb4f97602504eb5dee5094b8dc5123bb6354c319f6e1d2bf7429ca95359177dec0b5afbd11155bd07d66678e59f636355a56111c08e306e082bcf0855aeb51cb965d7de65f3618249a3674780c1d690169e71a7e4d627a11ad1a110514fa7368e0289bd8f0fdee7abec1420b15bcf9e46f0a41a547d9873dc3a42accb0cbb613b5ac33fa0e938ca65c42dd18c71e27735a3f2cef1e9d6b085d9eeca84ec790e8d
  4: (64 bytes/<class 'bytearray'>) 0xba082ff088e539e3176a90ce49eb042f5ec1af70796dd72d9f39fe679b84f99848e3f423bdcadf9d7d28d07d509588c791e4f93f55b0cbea0b4957eec741b628
]

2019-08-27 05:57:33,755 tlv.py:0117 DEBUG receiving [
  6: (1 bytes/<class 'bytearray'>) 0x03
  3: (384 bytes/<class 'bytearray'>) 0x73520bb9c3cd416bdf68955569dd1fb506dd851b8ce3ae34a4dfcc2f48edf953bfceccb9ea02fcf10cab8d24cc94b50835b23672debf0a4bf2b35dc030627c8e7060cd1414af3d89e0fc060aa432924a3c18fbf4b77bb703937fb3b55c7c50469b641ee21c532620a20ed7b5ee1f714fe24a33c988ea63f33e3749f0fe23335fe919ba9bec6c2ffbd911f4ee9d970e2ca1225a69350b403168eae727b71af6ad573b9a7f145a441c6da73b1d29c934598d46debd44d6696eb935d98ee0f0db298f051a487ee37334a77538e5668718a9aeb8121603f9dedbddbfa129a7f153fdb4f97602504eb5dee5094b8dc5123bb6354c319f6e1d2bf7429ca95359177dec0b5afbd11155bd07d66678e59f636355a56111c08e306e082bcf0855aeb51cb965d7de65f3618249a3674780c1d690169e71a7e4d627a11ad1a110514fa7368e0289bd8f0fdee7abec1420b15bcf9e46f0a41a547d9873dc3a42accb0cbb613b5ac33fa0e938ca65c42dd18c71e27735a3f2cef1e9d6b085d9eeca84ec790e8d
  4: (64 bytes/<class 'bytearray'>) 0xba082ff088e539e3176a90ce49eb042f5ec1af70796dd72d9f39fe679b84f99848e3f423bdcadf9d7d28d07d509588c791e4f93f55b0cbea0b4957eec741b628
]

2019-08-27 05:57:33,755 __init__.py:0060 DEBUG write message: [
  6: (1 bytes/<class 'bytearray'>) 0x03
  3: (384 bytes/<class 'bytearray'>) 0x73520bb9c3cd416bdf68955569dd1fb506dd851b8ce3ae34a4dfcc2f48edf953bfceccb9ea02fcf10cab8d24cc94b50835b23672debf0a4bf2b35dc030627c8e7060cd1414af3d89e0fc060aa432924a3c18fbf4b77bb703937fb3b55c7c50469b641ee21c532620a20ed7b5ee1f714fe24a33c988ea63f33e3749f0fe23335fe919ba9bec6c2ffbd911f4ee9d970e2ca1225a69350b403168eae727b71af6ad573b9a7f145a441c6da73b1d29c934598d46debd44d6696eb935d98ee0f0db298f051a487ee37334a77538e5668718a9aeb8121603f9dedbddbfa129a7f153fdb4f97602504eb5dee5094b8dc5123bb6354c319f6e1d2bf7429ca95359177dec0b5afbd11155bd07d66678e59f636355a56111c08e306e082bcf0855aeb51cb965d7de65f3618249a3674780c1d690169e71a7e4d627a11ad1a110514fa7368e0289bd8f0fdee7abec1420b15bcf9e46f0a41a547d9873dc3a42accb0cbb613b5ac33fa0e938ca65c42dd18c71e27735a3f2cef1e9d6b085d9eeca84ec790e8d
  4: (64 bytes/<class 'bytearray'>) 0xba082ff088e539e3176a90ce49eb042f5ec1af70796dd72d9f39fe679b84f99848e3f423bdcadf9d7d28d07d509588c791e4f93f55b0cbea0b4957eec741b628
]

2019-08-27 05:57:33,848 tlv.py:0117 DEBUG receiving [
]

2019-08-27 05:57:33,849 __init__.py:0067 DEBUG response: [
]

2019-08-27 05:57:33,849 __init__.py:0169 DEBUG #5 ios -> accessory: send SRP exchange request
list index out of range
2019-08-27 05:57:33,849 pair.py:0085 DEBUG list index out of range
Traceback (most recent call last):
  File "/Users/blake/src/homekit/homekit/pair.py", line 78, in <module>
    finish_pairing(pin_function())
  File "/Users/blake/src/homekit/homekit/controller/controller.py", line 391, in finish_pairing
    pairing = perform_pair_setup_part2(pin, str(uuid.uuid4()), write_fun, salt, pub_key)
  File "/Users/blake/src/homekit/homekit/protocol/__init__.py", line 173, in perform_pair_setup_part2
    assert response_tlv[0][0] == TLV.kTLVType_State and response_tlv[0][1] == TLV.M4, \
IndexError: list index out of range
niemyjski commented 5 years ago

Is there any updates on this? If I can help test anything please let me know.

Jc2k commented 5 years ago

There are some other pairing variants in the latest spec, maybe this doesn't support the original pairing variant for some reason.

@niemyjski bit of an ask but are you able to get a pcap dump of traffic flowing between an iOS device and your TV whilst pairing?

niemyjski commented 5 years ago

If you can tell me exactly what I'd need to do I'd be more than happy to do it. Thanks -Blake Niemyjski

On Thu, Oct 3, 2019 at 11:20 AM Jc2k notifications@github.com wrote:

There are some other pairing variants in the latest spec, maybe this doesn't support the original pairing variant for some reason.

@niemyjski https://github.com/niemyjski bit of an ask but are you able to get a pcap dump of traffic flowing between an iOS device and your TV whilst pairing?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jlusiardi/homekit_python/issues/170?email_source=notifications&email_token=AAHZFI3MPN2RH7IS72CZ4PDQMYLVNA5CNFSM4ISK7IA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIX74Y#issuecomment-538017779, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHZFI4H5IGVMYFHTUSV7KDQMYLVNANCNFSM4ISK7IAQ .

Jc2k commented 5 years ago

It really depends on your network setup, hardware and is.

If you have a desktop or laptop that can run wireshark you might just be able to run your wlan card in monitor mode and tell it to look for traffic between your iOS ip and your tv ip. This can be very OS and hardware dependent and I haven’t done it in a good long time. As your iOS device will be WiFi based this should work even if the tv is Ethernet. I don’t know how well this works on Windows, but I’ve definitely had success with it on Linux. Not sure about Mac - think I’ve seen it work there but could be misremembering.

If your tv is connected by Ethernet and you have a switch which supports mirroring or you have a dumb hub then you can watch traffic flowing to and from the tv by running wire shark on a physical Ethernet port.

You should start by installing wireshark and seeing if you can see your local WiFi traffic.

niemyjski commented 5 years ago

I'm running all on ubiquity and the tv and Apple TV are hard wired. I'm on Mac and can run Wireshark on the same network. Last time I tried to sniff HomeKit traffic I didn't have much luck so pointers welcomed. Thanks -Blake Niemyjski

On Fri, Oct 4, 2019 at 8:05 AM Jc2k notifications@github.com wrote:

It really depends on your network setup, hardware and is.

If you have a desktop or laptop that can run wireshark you might just be able to run your wlan card in monitor mode and tell it to look for traffic between your iOS ip and your tv ip. This can be very OS and hardware dependent and I haven’t done it in a good long time. As your iOS device will be WiFi based this should work even if the tv is Ethernet. I don’t know how well this works on Windows, but I’ve definitely had success with it on Linux. Not sure about Mac - think I’ve seen it work there but could be misremembering.

If your tv is connected by Ethernet and you have a switch which supports mirroring or you have a dumb hub then you can watch traffic flowing to and from the tv by running wire shark on a physical Ethernet port.

You should start by installing wireshark and seeing if you can see your local WiFi traffic.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jlusiardi/homekit_python/issues/170?email_source=notifications&email_token=AAHZFI3E5UEEKXA52AG5U5DQM45P5A5CNFSM4ISK7IA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEALSNWI#issuecomment-538388185, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHZFI5GVYLHMXFHTOSYAHDQM45P5ANCNFSM4ISK7IAQ .

jlusiardi commented 5 years ago

Hey, I guess your mac, the TV and the Apple TV are on the same wired network, but connected via a switch?

A switch is more intelligent than an Ethernet hub, which simply retransmits packets out of every port of the hub except the port on which the packet was received, unable to distinguish different recipients, and achieving an overall lower network efficiency.
(see https://en.wikipedia.org/wiki/Network_switch#Overview)

Perhaps https://wiki.wireshark.org/CaptureSetup/Ethernet#Switched_Ethernet will explain further and can help solve this issue.

-Joachim Lusiardi

Ulrar commented 5 years ago

I have HA running on a pi, I'm guessing a tcpdump file from the pi while trying to pair the TV is what you're looking for ? You can open that up in wireshark to look at the packets. As mentioned in the other issue though I have an LG C9, not an appel TV, but I suppose it's the same problem.

I'm attaching the pcap file, it was running from the moment I clicked "configure" in HA to after the error. capture.zip

Jc2k commented 5 years ago

We could do with a capture between a iOS device and the tv ideally so we can see if it is using one of the new pairing handshakes.

Ulrar commented 5 years ago

I don't have any iOS devices so I can't help with that, sorry

On Mon 7 Oct 2019, 09:14 Jc2k, notifications@github.com wrote:

We could do with a capture between a iOS device and the tv ideally so we can see if it is using one of the new pairing handshakes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jlusiardi/homekit_python/issues/170?email_source=notifications&email_token=AAGVHT7JSUDKDHIVTBOK3NDQNLVVZA5CNFSM4ISK7IA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAPMPBI#issuecomment-538888069, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGVHT3DT6QGU37M3YNSJ7DQNLVVZANCNFSM4ISK7IAQ .

Ulrar commented 4 years ago

Just remembered my girlfriend does have a mac laptop. Googling it looks like that won't do though, is that correct ? No way to use a macos laptop to initiate pairing and capture what you'd need ?

jlusiardi commented 4 years ago

Hi,

I found https://appleinsider.com/articles/18/06/05/hands-on-controlling-your-smart-home-with-homekit-on-macos-mojave which states from MacOS Mojave onwards this should be possible. But adding new devices is not possible in Mojave. Perhaps this was improved in MacOS Catalina.

@netmanchris, I saw your cool analysis from https://www.youtube.com/watch?v=y5L21gjEyC4 do you have a Catalina enabled Mac? Does it support pairing?

Regards Joachim

jzee commented 4 years ago

Facing the exact same symptoms and logs on an LG TV and looked for the latest protocol spec.

It appears Apple gives out R2 to non-commercial users. I'm not sure what you @Jc2k have been working with but here's an implementation against R14 uploaded a few days ago. Can you spot anything obvious in the handling of M4? (start at line 445)

Jc2k commented 4 years ago

Our pairing code was written against R1, and one thing i've been meaning to dig into was whether the Transient and Split pairing modes that are lightly touched on in R2 of the spec could be at play. In a transient pairing there is no M5 or M6, but that only happens if we set kTLVType_Flags.

In addition, if you trigger split pairing then you can reuse an SRP code. I think you can create a secure transient session and then do additional auth over the transient session to finish the pairing?

Can you paste the discovery output for your device? I'm interested in whether the affected devices have an ff flag set to 2.

https://github.com/apple/HomeKitADK/blob/d1174d26f4986f66402b37b46d7239599e1533b3/HAP/HAPIPServiceDiscovery.c#L86

https://github.com/apple/HomeKitADK/blob/d1174d26f4986f66402b37b46d7239599e1533b3/HAP/HAPAccessoryServer.c#L862

Suggests there are 2 possible flags for ff:

However i'm sure an earlier bug report was showing an '8' in the ff field which i'm struggling to match to the R14 code.

Some other relevant bits here:

Also we need to figure out kHAPPairingMethod_PairSetupWithAuth vs kHAPPairingMethod_PairSetup.

Jc2k commented 4 years ago

https://github.com/apple/HomeKitADK/blob/d1174d26f4986f66402b37b46d7239599e1533b3/HAP/HAPPairingPairSetup.c#L597-L805

I don't think we currently do any of that.

Or this:

https://github.com/apple/HomeKitADK/blob/d1174d26f4986f66402b37b46d7239599e1533b3/HAP/HAPPairingPairSetup.c#L204-L235

jlusiardi commented 4 years ago

There seems to be a new way to pair HomeKit accessories present in R2 of the HAP Specification.

Let's do some comparing of R1 vs. R2.

For easier navigation use the next table:

section Revision 1 Revision 2
start of pairing Chapter 4.7 page 39 Chapter 5.6 page 34
definition of methods Table 4-4 page 60 Table 5-3 page 49
pairing type flags N/A Table 5-7 page 51
TLV Values Table 4-6 page 61 Table 5-6 page 49
BLE Protocol Version Characteristic 02.01.00 (Chapter 6.4.3.1, page 126) 02.02.00 (Chapter 7.4.3.1, page 121)
IP Protocol Version N/A? Chapter 6.6.3 page 61 mentions PV from Discovery must be 1.1.

There seems to be a difference in the values of the methods relevant for pairing:

Value Method in R1 Method in R2
0 Reserved Pair Setup
1 Pair Setup Pair Setup with Auth

Our code still sends a value of 1 but without given a value for kTLVType_Flags. This particular key was introduced in R2 with a value of 0x13.

So I am not sure on how to proceed to be compatible with devices that work with R1 and Eve Energy 1EC0 / LG C9 from this issue. I will do some more analysis later.

Is one of the owners (@jzee, @Ulrar, @niemyjski) willing to try to witch the value of PairSetup to 0:

diff --git a/homekit/protocol/tlv.py b/homekit/protocol/tlv.py
index 153d7b2..b2d3a54 100644
--- a/homekit/protocol/tlv.py
+++ b/homekit/protocol/tlv.py
@@ -32,7 +32,7 @@ class TLV:
     M6 = bytearray(b'\x06')

     # Methods (see table 4-4 page 60)
-    PairSetup = bytearray(b'\x01')
+    PairSetup = bytearray(b'\x00')
     PairVerify = bytearray(b'\x02')
     AddPairing = bytearray(b'\x03')
     RemovePairing = bytearray(b'\x04')
jzee commented 4 years ago

Did that, modified the constant straight in the installation. Here's the log output. If you DM me the instructions of how to get to a debugging setup so I can run python3 -m homekit.pair -d XX:XX:XX:XX:XX:XX -a mytv -f pairing.json --log DEBUG or the like from the console i can maybe do some more investigations.

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 247, in async_step_pair await self.hass.async_add_executor_job(self.finish_pairing, code) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.7/site-packages/homekit/controller/controller.py", line 391, in finish_pairing pairing = perform_pair_setup_part2(pin, str(uuid.uuid4()), write_fun, salt, pub_key) File "/usr/local/lib/python3.7/site-packages/homekit/protocol/__init__.py", line 173, in perform_pair_setup_part2 assert response_tlv[0][0] == TLV.kTLVType_State and response_tlv[0][1] == TLV.M4, \ IndexError: list index out of range

jzee commented 4 years ago

Something just happened, it worked! Without having a lot of information how to debug the install properly, I went into log_support.py and hardcoded the call to getLogger().setLevel(5), then I restarted HA. Note that the "PairSetup" was still set to 0, as @jlusiardi suggested.

The TV shows as paired now and reads out the firmware version properly. I have yet to find out what more I can do with it now but it is definitely paired.

jlusiardi commented 4 years ago

ok, great!

can you execute

I guess we must find a way to get the version of the protocol before starting a pairing...

jzee commented 4 years ago

doesn't find pairing.json. I grep'ed through the HA docker instance and the host, there is no such file. Where do I find it?

bash-5.0# python -m homekit.discover --log DEBUG 2020-01-11 17:12:21,959 init.py:0099 DEBUG candidate data {b'md': b'Home Assistant Bridge', b'pv': b'1.0', b'id': b'XX:XX:XX:XX:XX:XX', b'c#': b'2', b's#': b'1', b'ff': b'0', b'ci': b'2', b'sf': b'0', b'sh': b'eJ71Fg=='} 2020-01-11 17:12:21,962 init.py:0107 DEBUG found Homekit IP accessory {'name': 'Home Assistant Bridge._hap._tcp.local.', 'address': '192.168.xx.xx', 'port': 51827, 'c#': '2', 'ff': 0, 'flags': 'No support for HAP Pairing', 'id': 'XX:XX:XX:XX:XX:XX', 'md': 'Home Assistant Bridge', 'pv': '1.0', 's#': '1', 'sf': '0', 'statusflags': 'Accessory has been paired.', 'ci': '2', 'category': 'Bridge'} 2020-01-11 17:12:21,963 init.py:0099 DEBUG candidate data {b'c#': b'1', b'ff': b'8', b'id': b'YY:YY:YY:YY:YY:YY', b'md': b'WEBOS', b'pv': b'1.1', b's#': b'1', b'sf': b'0', b'ci': b'31', b'sh': b'wu5QmQ=='} Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.7/site-packages/homekit/discover.py", line 51, in results = Controller.discover(args.timeout) File "/usr/local/lib/python3.7/site-packages/homekit/controller/controller.py", line 90, in discover return discover_homekit_devices(max_seconds) File "/usr/local/lib/python3.7/site-packages/homekit/zeroconf_impl/init.py", line 102, in discover_homekit_devices info.properties File "/usr/local/lib/python3.7/site-packages/homekit/zeroconf_impl/init.py", line 180, in parse_discovery_properties d['category'] = Categories[int(category)] File "/usr/local/lib/python3.7/site-packages/homekit/model/categories.py", line 94, in getitem raise KeyError('Item {item} not found'.format(item=item)) KeyError: 'Item 31 not found' bash-5.0#

Jc2k commented 4 years ago

I'm guessing you are using Home Assistant's web front end to pair rather than the homekit_python projects CLI? (HA doesn't use a pairing.json and wont understand category 31 yet)

jzee commented 4 years ago

that's correct, I'm unaware of how to use the homekit's CLI. I can have a look at a clean install of it, any pointers are appreciated.

Jc2k commented 4 years ago

There's a file in configuration/.storage/homekit_controller-entity-map that is similar to (has all the same data as) the output of homekit.get_accessories. Review it before posting - it may have e.g. your TV's serial number.

I can try and tell you how to get the pairing data out of HA when i get back home tonight/tomorrow.

jzee commented 4 years ago

that should be it

homekit_controller-entity-map-cleansed.txt

jzee commented 4 years ago

I just ran the discover from the CLI client after walking through the installation and it coughs at cat 31 as well, with an identical traceback. I checked out master

jlusiardi commented 4 years ago

from homekit_controller-entity-map-cleansed.txt:

{
                                "characteristics": [
                                    {
                                        "ev": false,
                                        "format": "string",
                                        "iid": 18,
                                        "perms": [
                                            "pr"
                                        ],
                                        "type": "00000037-0000-1000-8000-0026BB765291",
                                        "value": "1.1.0"
                                    }
                                ],
                                "hidden": false,
                                "iid": 16,
                                "linked": [],
                                "primary": false,
                                "stype": "service",
                                "type": "000000A2-0000-1000-8000-0026BB765291"
                            },

which states a version of 1.1.0. So far so good. But I think we should have the version information before pairing.

@jzee: I just pushed a new commit with an added category 31 (as TV). Can you pull and try again?

Ulrar commented 4 years ago

I just got home and haven't paired my TV at all, happy to help if I can. Not sure how to pull the latest version of homekit_python into home assistant though, should I just git clone the repo and copy the files or is there an easier way ?

Ulrar commented 4 years ago

I can confirm that switching the PairSetup to 0 and then restarting home assistant makes pairing work. Now unfortunately home assistant doesn't support TVs on this homekit integration, so it doesn't do anything, but at least it got rid of the persistent notification.

Jc2k commented 4 years ago

Off topic for here, but @Ulrar can you share your configuration/.storage/homekit_controller-entity-map too? The more TV's I can see the soon/better the HA TV code will arrive/be

Ulrar commented 4 years ago

There it is. Not sure what I should or shouldn't edit out so I just removed the mac, serial number and what I assume must be the key. Let me know if you need anything else, thanks !

https://gist.github.com/Ulrar/30a9fe78e962895cce65413982e2ceaf

jzee commented 4 years ago

@jlusiardi this commit fixed it. Here's the discovery output

$ python3 -m homekit.discover --log DEBUG 2020-01-11 22:30:26,390 init.py:0100 DEBUG candidate data {b'md': b'Home Assistant Bridge', b'pv': b'1.0', b'id': b'XX:XX:XX:XX:XX:XX', b'c#': b'2', b's#': b'1', b'ff': b'0', b'ci': b'2', b'sf': b'0', b'sh': b'eJ71Fg=='} 2020-01-11 22:30:26,390 init.py:0108 DEBUG found Homekit IP accessory {'name': 'Home Assistant Bridge._hap._tcp.local.', 'address': '192.168.XX.XX', 'port': 51827, 'c#': '2', 'ff': 0, 'flags': 'No support for HAP Pairing', 'id': 'XX:XX:XX:XX:XX:XX', 'md': 'Home Assistant Bridge', 'pv': '1.0', 's#': '1', 'sf': '0', 'statusflags': 'Accessory has been paired.', 'ci': '2', 'category': 'Bridge'} 2020-01-11 22:30:26,391 init.py:0100 DEBUG candidate data {b'c#': b'1', b'ff': b'8', b'id': b'XX:XX:XX:XX:XX:XX', b'md': b'WEBOS', b'pv': b'1.1', b's#': b'1', b'sf': b'0', b'ci': b'31', b'sh': b'wu5QmQ=='} 2020-01-11 22:30:26,391 init.py:0108 DEBUG found Homekit IP accessory {'name': 'LG webOS TV A716._hap._tcp.local.', 'address': '192.168.XX.XX', 'port': 37517, 'c#': '1', 'ff': 8, 'flags': 'No support for HAP Pairing', 'id': 'XX:XX:XX:XX:XX:XX', 'md': 'WEBOS', 'pv': '1.1', 's#': '1', 'sf': '0', 'statusflags': 'Accessory has been paired.', 'ci': '31', 'category': 'TV'} Name: Home Assistant Bridge._hap._tcp.local. Url: http_impl://192.168.XX.XX:51827 Configuration number (c#): 2 Feature Flags (ff): No support for HAP Pairing (Flag: 0) Device ID (id): XX:XX:XX:XX:XX:XX Model Name (md): Home Assistant Bridge Protocol Version (pv): 1.0 State Number (s#): 1 Status Flags (sf): Accessory has been paired. (Flag: 0) Category Identifier (ci): Bridge (Id: 2)

Name: LG webOS TV A716._hap._tcp.local. Url: http_impl://192.168.XX.XX:37517 Configuration number (c#): 1 Feature Flags (ff): No support for HAP Pairing (Flag: 8) Device ID (id): XX:XX:XX:XX:XX:XX Model Name (md): WEBOS Protocol Version (pv): 1.1 State Number (s#): 1 Status Flags (sf): Accessory has been paired. (Flag: 0) Category Identifier (ci): TV (Id: 31)

Jc2k commented 4 years ago

So pv is 1.1 for the LG webOS TV, and we have to use PairSetup==0. But looking at my home network both my Hue and Eve Extend are pv 1.1 and work with the existing code. So I don't know if its enough to key off pv.

Both the hue and Eve Extend have ff of 1 and the TV has an ff of 8.

Looking at all the working IP devices in tested_devices, they are a mixture of protocol 1.0 and 1.1 and all have ff 1.

From reading the code @jzee posted, ff 1 seems to mean there is a hardware security chip.

So maybe:

jlusiardi commented 4 years ago

@Jc2k sadly there is not much information about the ff value of the bonjour record in the R2 Spec. It references a table which is pretty useless.

Jc2k commented 4 years ago

Yes its weird that there is a table in R2 but its empty.

In the official ADK the bonjour data for ff is set here. It uses the function HAPAccessoryServerGetPairingFeatureFlags to get the value. We can see that the public Apple code confirms the values of kHAPCharacteristicValue_PairingFeatures_SupportsAppleAuthenticationCoprocessor and kHAPCharacteristicValue_PairingFeatures_SupportsSoftwareAuthentication for ff and their values are defined here. So we know what the value means, but I guess we don't know what it implies for the protocol. We are none the wiser about what ff of 8 means though.

I think it makes sense that 1 works for PairSetup only with the hardware encryption chip - they are largely the first wave of devices and all we've seen till recently. I guess the "Auth" is that the device proves it has an Apple crypto chip?

I suppose we could make a mock accessory that set ff to 8 and pv to 1.1 and see if that changes what an official client does for the encryption handshake.

Jc2k commented 4 years ago

Another user from HA reported issues pairing. They have a VOCOlinc L3 bulb.

Instead of this exception in their case the pairing process hung after M3 but before the asset / IndexError.

This device has an ff of 2 and pv of 1.1. The same patch as above allowed pairing to work - they were able to pair through the HA front end as well.

This is another case where ff & SupportsAppleAuthenticationCoprocessor was a predictor for whether to use PairSetup or PairSetupWithAuth.

Jc2k commented 4 years ago

Trying to find something we can use in lieu of spec...

This is part of the official Apple HAP ADK for PairSetupWithAuth:

https://github.com/apple/HomeKitADK/blob/d1174d26f4986f66402b37b46d7239599e1533b3/HAP/HAPPairingPairSetup.c#L612-L623

My reading of this code is that an accessory made with that spec would return an kHAPError_InvalidState if you tried to do an M4 on a device that didn't have a Mfi Hardware security co-processor. We also know from the other code I linked to that an ff of 1 means that a device does have such a chip, according to the Apple example code at least.

In other words, if we tried to pair with an accessory written with the ADK we would have to have logic that did:

if ff & SupportsAppleAuthenticationCoprocessor:
    PairSetupWithAuth()
else:
    PairSetup()

I.e. we should only attempt PairSetupWithAuth if we detect a security chip, because we know that ADK devices fail otherwise.

jzee commented 4 years ago

@Jc2k building a mock accessory is actually what the ADK repo comes with out of the box (try “make Target=Linux Apps” on a Linux VM) - so doing that exercise is fairly easy. I spent the better part of Sunday fiddling with it and getting it on the network showing up as a node on iOS is trivial.

I still failed on getting it to use a setup code so I could get further down the auth process - out of the box the example accessories fail since they aren’t automatically provided with a setup code. If someone can shed some light on how that could work, I can dig down a little further.

jlusiardi commented 4 years ago

I think we should collect a table of devices here... Perhaps different (firmware)versions of the same device have different behaviors?

Feel free to add devices: https://docs.google.com/spreadsheets/d/17KzVh7pLqO6R0eqJUiliJCvOMSgqTjuYDYmBdGhaNvs/edit?usp=sharing

Jc2k commented 4 years ago

I used demoaccessory and twiddled the pv and ff bonjour data, and captured the TLV_Method attribute when pairing with an official iOS 13 client.

Results:

pv 1.0
ff 0
method 0

pv 1.0
ff 1
method 1

pv 1.1
ff 1
method 1

pv 1.1
ff 0
method 0

pv 1.0
ff 8
method 1

pv 1.0
ff 2
method 1

An official client will use PairSetup = 0 if ff is 0 (and this is what triggers the uncertified device warning also).

Annoyingly though ff 8 does use the failing version of PairSetup, so i suspect that using PairSetup = 0 is actually working by accident rather than design?