Homekit accessory not showing up in Integrations dashboard

tleegaard commented 5 years ago

Home Assistant release with the issue: 0.96.5

Component/platform: Homekit Controller

Description of problem: From the homeassistant.log:

2019-07-27 21:01:56 DEBUG (MainThread) [homeassistant.components.homekit_controller.config_flow] Discovered device VELUX gateway (VELUX Gateway - 34:0F:45:63:A6:C3)
2019-07-27 21:01:57 DEBUG (MainThread) [homeassistant.components.homekit_controller.config_flow] HomeKit device 34:0F:45:63:A6:C3 ignored as already paired

Home Assistant seems to think the gateway is already paired, but I cannot find it anywhere in Home Assistant, the log files nor the .storage/core.-files.

ghost commented 5 years ago

Hey there @Jc2k, mind taking a look at this issue as its been labeled with a integration (homekit_controller) you are listed as a codeowner for? Thanks!

This is a automatic comment generated by codeowners-mention to help ensure issues and pull requests are seen by the right people.

Jc2k commented 5 years ago

Hi @tleegaard

To be clear - have you gotten to the stage of typing in a pairing code within Home Assistant, or is this the first time you've tried to use the device with Home Assistant?

If the device isn't showing up in "Integrations" as discovered and you have that message in your logs it means the device itself is paired. The device will refuse any further attempts to pair, so there is nothing to reset on the Home Assistant side. You will need to reset the Velux Gateway. There might be an option to reset just the homekit part. But it depends on the device. You might have to to do a full reset. On some thermostats there is a menu on the device, on others there is a long hold reset and a short hold reset and one is homekit data only. On the Hue i think there is a button in the official App to reset homekit over the air.

tleegaard commented 5 years ago

Yes - I entered the pin and got an error that the device was paired with another controller/hub. Then I deleted the device from the iPhone it was paired to and tried readding it in HA - but now I only get the "No unpaired devices could be found" message when trying to add it again in HA.

I will try resetting the gateway :-)

tleegaard commented 5 years ago

I tried resetting and adding it again. In the UI I get this error: An unhandled error occured while attempting to pair with this device. This may be a temporary failure or your device may not be supported currently.

And from the homeassistant.log:

2019-07-29 17:26:58 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/homekit_controller/config_flow.py", line 259, in async_step_pair
    pairing)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/homekit_controller/config_flow.py", line 336, in _entry_from_accessory
    pairing.list_accessories_and_characteristics
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 331, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 244, in _wakeup
    future.result()
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 244, in result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/controller/ip_implementation.py", line 73, in list_accessories_and_characteristics
    self.session = IpSession(self.pairing_data)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/controller/ip_implementation.py", line 416, in __init__
    c2a_key, a2c_key = get_session_keys(conn, pairing_data, write_fun)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/protocol/__init__.py", line 296, in get_session_keys
    ios_key = x25519.X25519PrivateKey.generate()
  File "/srv/homeassistant/lib/python3.6/site-packages/cryptography/hazmat/primitives/asymmetric/x25519.py", line 42, in generate
    _Reasons.UNSUPPORTED_EXCHANGE_ALGORITHM
cryptography.exceptions.UnsupportedAlgorithm: X25519 is not supported by this version of OpenSSL.
2019-07-29 17:26:58 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/homekit_controller/config_flow.py", line 288, in async_step_pair
    start_pairing, self.hkid, self.hkid
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 331, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 244, in _wakeup
    future.result()
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 244, in result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/controller/controller.py", line 370, in start_pairing
    raise AlreadyPairedError('Alias "{a}" is already paired.'.format(a=alias))
homekit.exceptions.AlreadyPairedError: Alias "82:BA:03:CC:C5:8C" is already paired.
2019-07-29 17:27:35 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/homekit_controller/config_flow.py", line 253, in async_step_pair
    self.finish_pairing, code
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 331, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 244, in _wakeup
    future.result()
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 244, in result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/controller/controller.py", line 387, in finish_pairing
    pairing = perform_pair_setup_part2(pin, str(uuid.uuid4()), write_fun, salt, pub_key)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/protocol/__init__.py", line 174, in perform_pair_setup_part2
    'perform_pair_setup: State not M4'
AssertionError: perform_pair_setup: State not M4
2019-07-29 17:27:35 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/homekit_controller/config_flow.py", line 288, in async_step_pair
    start_pairing, self.hkid, self.hkid
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 331, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 244, in _wakeup
    future.result()
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 244, in result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/homekit/controller/controller.py", line 370, in start_pairing
    raise AlreadyPairedError('Alias "{a}" is already paired.'.format(a=alias))
homekit.exceptions.AlreadyPairedError: Alias "82:BA:03:CC:C5:8C" is already paired.

Is the device unsupported or is something else wrong?

Jc2k commented 5 years ago

The most important error is about X25519 - not seen that in a while. If you are not familiar with python packaging some of this might be clear as mud. I will try to help. To help I need to know:

What hardware are you running on
What your host OS (including version) is
Are you using containers
What OS is the container using (including version)

How do you install HASS? Is ir the latest release?

You are using a version of homekit_controller with config flow so its quite recent, but I just want to cover all bases.

If you are somewhat familiar with python and python packaging then this might help you figure out things on your own or get close:

The problem is your openssl library and its python counterpart cryptography are either too old or built without X25519 support. From previous experience this means a couple of scenarios:

There is a way of updating Home Assistant where the code itself was updated by its dependency cryptography was not updated. Force reinstalling cryptography may help.
You are using an Alpine based host OS or container, but not one provided by Home Assistant. Prior to Alpine 3.9 the OpenSSL library was too old to support X25519 and whilst python provide a "wheel" that includes a newer OpenSSL library it is incompatible with musl libc, which Alpine uses.
You are using an older linux distro that has libc and not musl libc, but for whatever reason it doesn't support wheels. When the cryptography library is updated on here it uses the host openssl, and it is old enough to not support X25519. I suppose this could happen on an older raspbian - maybe Jessie? - and it either doesnt use wheels or it uses a piwheels.org wheel that uses the host openssl.

From what i remember, cryptography version 2.3.1 or later should work if:

Using wheels on Ubuntu or Debian
Using wheels on Raspbian (Stretch) or later
Not using wheel and on Alpine 3.9 or later
Not using wheel and on Debian Stretch or later
Not using wheels and on Ubuntu 18.04 or later

If not using wheels the host OS needs to have openssl 1.1.0 or later (possibly 1.1.1). Any earlier and cryptography will install but be missing features.

tleegaard commented 5 years ago

Hi @Jc2k, Thank you for all your help! I'm running version 0.96.5 of HA on an Orange Pi Zero (ARMBIAN 5.60 stable Debian GNU/Linux 8 (jessie) 3.4.113-sun8i) and (as I recall) followed this guide when setting it up long time ago: https://www.home-assistant.io/docs/installation/raspberry-pi/

With pip freeze | grep cryptography I'm getting: cryptography==2.7

root@orangepizero:~# openssl
OpenSSL> version
OpenSSL 1.0.1t  3 May 2016

I'm unsure where to go from here 😅

(Update: Seeing I'm on Jessie I will try do an upgrade to Buster)

Jc2k commented 5 years ago

Buster upgrade definitely worth it! Hopefully that will also fix the problem, let me know how you get on.

Jc2k commented 5 years ago

(Just noticed this is issue #25519 and it’s a problem about X25519. Well timed ticket raising!!)

tleegaard commented 5 years ago

(Just noticed this is issue #25519 and it’s a problem about X25519. Well timed ticket raising!!)

I should probably go buy a lottery ticket 😄

tleegaard commented 5 years ago

The upgrade didn't end well, so I started from scratch with hassio, docker and armbian on the Orange Pi Zero.

Tried pairing the Velux gateway again and I am now getting these errors:

2019-07-30 09:27:42 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 288, in async_step_pair
    start_pairing, self.hkid, self.hkid
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/homekit/controller/controller.py", line 379, in start_pairing
    salt, pub_key = perform_pair_setup_part1(write_fun)
  File "/usr/local/lib/python3.7/site-packages/homekit/protocol/__init__.py", line 113, in perform_pair_setup_part1
    response_tlv = write_fun(request_tlv, step2_expectations)
  File "/usr/local/lib/python3.7/site-packages/homekit/protocol/__init__.py", line 64, in write_http
    connection.endheaders(request)
  File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/site-packages/homekit/http_impl/http_client.py", line 44, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/http/client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
OSError: [Errno 113] Host is unreachable
2019-07-30 09:28:01 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 253, in async_step_pair
    self.finish_pairing, code
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
TypeError: 'NoneType' object is not callable
2019-07-30 09:28:10 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 288, in async_step_pair
    start_pairing, self.hkid, self.hkid
concurrent.futures._base.CancelledError
2019-07-30 09:28:10 ERROR (MainThread) [aiohttp.server] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 447, in start
    await resp.prepare(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_response.py", line 353, in prepare
    return await self._start(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_response.py", line 667, in _start
    return await super()._start(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_response.py", line 410, in _start
    await writer.write_headers(status_line, headers)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/http_writer.py", line 112, in write_headers
    self._write(buf)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/http_writer.py", line 67, in _write
    raise ConnectionResetError('Cannot write to closing transport')
ConnectionResetError: Cannot write to closing transport

Jc2k commented 5 years ago

So it’s seems like it’s failing much earlier this time. Can you restart HA and make sure it shows up in integrations as pairable. The relevant error is the “host unreachable” one.thats a low level network error directly from the OS when trying to connect to the IP of your garage opener.

Is the ip static in dhcp? Can you ping it? Can you leave ping running and try pairing?

The aiohttp error is unrelated, it looks like another integration might also be struggling with maintaining tcp connections. It’s a server, could be the web front end?

tleegaard commented 5 years ago

Well - I tried pairing again - and it's actually working now! 😄 Not sure why it failed before..

tleegaard commented 5 years ago

Thank you for your time and help!

tleegaard commented 5 years ago

Hi @Jc2k, I have tried pairing the Velux gateway with Homekit_python and I noticed that I'm missing the CO2 sensor (2.17) in Home Assistant: https://github.com/jlusiardi/homekit_python/issues/155#issuecomment-516791229 screendump Do you know why that is the case?

Also, I think I'm unable to unpair the gateway from HA. I found that I needed to remove the -i parameter from Homekit_python: https://github.com/jlusiardi/homekit_python/issues/155#issuecomment-516703629 to be able to unpair it. Maybe it's a similar case.

Jc2k commented 5 years ago

Unpairing from within HA isn’t currently implemented - it’s on my todo though!

I don’t think we have implemented CO2 sensors yet either. Have you got it working with homekit_python? You can get me some JSON output from it I’ll be able to add it for you

tleegaard commented 5 years ago

Thanks - i appreciate your help! :-) Is it possible to manually unpair via ssh from the Pi with HA? It takes quite some time to reset the gateway as everything needs to be setup again :-)

I did indeed get it working with homekit_python:

1.1: >accessory-information<
  1.2: VELUX gateway () >name< [pr]
  1.3: VELUX () >manufacturer< [pr]
  1.4: VELUX Gateway () >model< [pr]
  1.5: g37416e () >serial-number< [pr]
  1.6:  () >identify< [pw]
  1.7: 61 () >firmware.revision< [pr]
1.8: >service<
  1.9: 1.1.0 () >version< [pr]
2.1: >accessory-information<
  2.2: VELUX Sensor () >name< [pr]
  2.3: VELUX () >manufacturer< [pr]
  2.4: VELUX Sensor () >model< [pr]
  2.5: p0040fd () >serial-number< [pr]
  2.7:  () >identify< [pw]
  2.6: 16 () >firmware.revision< [pr]
2.8: >temperature<
  2.9: Temperature sensor () >name< [pr]
  2.10: 22.7 () >temperature.current< [pr,ev]
2.11: >humidity<
  2.12: Humidity sensor () >name< [pr]
  2.13: 68.0 () >relative-humidity.current< [pr,ev]
2.14: >carbon-dioxide<
  2.15: Carbon Dioxide sensor () >name< [pr]
  2.16: 0 () >carbon-dioxide.detected< [pr,ev]
  2.17: 903.0 () >carbon-dioxide.level< [pr,ev]

Jc2k commented 5 years ago

There isn't an easy way - you'd have to extract the encryption keys from the config entries file and copy them into your pairing json file.

That should be enough for me to get started. I have a few other things I need to do before I get to it though.

If you want to have a go at adding this yourself (I can guide you) the relevant code is here - its mostly just copying a class and renaming a few things.

And then figuring out what to put in here to pick up the new class.

tleegaard commented 5 years ago

Does this https://github.com/tleegaard/home-assistant/blob/dev/homeassistant/components/homekit_controller/sensor.py look about right?

I'm not sure if carbon-dioxide.level corresponds to CharacteristicsTypes.CARBON_DIOXIDE_LEVEL?

I'm also not sure about naming: https://github.com/tleegaard/home-assistant/blob/dev/homeassistant/components/homekit_controller/sensor.py#L42

Jc2k commented 5 years ago

All the characteristic types homekit_python knows are here. So it looks like we want CARBON_DIOXIDE_LEVEL like you have.

Re: devtype, that corresponds to the name of the service in the snippet you posted (look at 2.14). So It looks like it will be carbon-dioxide.

Other than, I think this will work. If it does and you want to put in a PR (i hope you do, cos it will save me a job) can you also:

Update async_setup_entry to look like this one. The current one duplicates logic and it got flagged up in review last time someone added a binary sensor.
Add a test for it. You should be able to copy the ones that already exist here.

And the linter is pretty strict - it will flag up that you need 2 blank lines before class HomeKitCO2Sensor(HomeKitEntity):.

If you submit a PR it will see you've touched homekit_controller and auto-tag me to review it for you.

tleegaard commented 5 years ago

I have added the two blank lines, and I tried updating async to be similar to binary_sensor.py but I'm not sure if it is done right(?): https://github.com/tleegaard/home-assistant/blob/dev/homeassistant/components/homekit_controller/sensor.py

As for tests I updated https://github.com/tleegaard/home-assistant/blob/dev/tests/components/homekit_controller/test_sensor.py to include the CO2 sensor.

I have just done the editing in the browser and I'm not that familiar with Github. Do I make a PR like this? Skærmbillede 2019-07-31 kl 16 19 30

Jc2k commented 5 years ago

Yes, that will open up a template to fill in to make a PR. It will guide you through the next steps, and run the tests and linters.

The setup function needs a bit more tweaking, try:

ENTITY_TYPES = {
    'humidity': HomeKitHumiditySensor,
    'temperature': HomeKitTemperatureSensor,
    'light': HomeKitLightSensor,
    'carbon-dioxide': HomeKitCarbonDioxideSensor,
}

async def async_setup_entry(hass, config_entry, async_add_entities):
    """Set up Homekit sensors."""
    hkid = config_entry.data['AccessoryPairingID']
    conn = hass.data[KNOWN_DEVICES][hkid]

    def async_add_service(aid, service):
        entity_class = ENTITY_TYPES.get(service['stype'])
        if not entity_class:
            return False
        info = {'aid': aid, 'iid': service['iid']}
        async_add_entities([entity_class(conn, info)], True)
        return True

    conn.add_listener(async_add_service)

That way the next person to add a sensor will only have to add the class to ENTITY_TYPES. If you make that change and open the pull request we can continue the discussion over there.

home-assistant / core

Homekit accessory not showing up in Integrations dashboard #25519