homekit_controller: Manual discovery fails when unknown categories are on network

JamesDenby commented 4 years ago

The problem

Environment

Home Assistant release with the issue: 0.106.0
Last working Home Assistant release (if known): not known
Operating environment (Hass.io/Docker/Windows/etc.):
Integration causing this issue: Adding HomeKit accessory
Link to integration documentation on our website: https://www.home-assistant.io/integrations/homekit_controller/

Problem-relevant `configuration.yaml`

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:
zeroconf:

# Need this even though I have default_config:
frontend:
  themes: !include_dir_merge_named themes

# Add customization information:
homeassistant:
  customize: !include customize.yaml

# Needed to assess health of system
system_health:

# Config for mobile phone app
mobile_app:
cloud:
ios:
stream:

# Required for some configurations
#homeassistant:
#    customize: !include customize.yaml

# Uncomment this if you are using SSL/TLS, running in Docker container, etc.
# http:
#   base_url: example.duckdns.org:8123

# Text to speech
tts:
  - platform: google_translate

# Control and limit size of database
recorder:
  purge_keep_days: 5
  exclude:
    domains:
      - camera
      - fan
      - light
      - media_player
      - sun
      - weather
      - device_tracker
      - switch
      - cover
      - zone
    entities:
     - sensor.last_boot
     - sensor.date

# Lutron Caseta Control
lutron_caseta:
  host: 192.168.4.73
  keyfile: /ssl/lutron/caseta.key
  certfile: /ssl/lutron/caseta.crt
  ca_certs: /ssl/lutron/caseta-bridge.crt

# Chamberlain Garage Doors
cover:
  - platform: myq
    username: !secret who
    password: !secret id

# Local IP sensor 
local_ip:

# HACs add-on to identify potential problems on upgrades
breaking_changes:

# WeMo devices with automatic discovery
wemo:
  discovery: true
  static:
    - 192.168.4.35
    - 192.168.4.38
    - 192.168.4.41
    - 192.168.4.36
    - 192.168.4.48
    - 192.168.4.39
    - 192.168.4.32
    - 192.168.4.40

group: !include groups.yaml
automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml
garbage_collection: !include trash.yaml
sensor: !include sensor.yaml
camera: !include video.yaml
weather: !include weather.yaml
homekit: !include homekit.yaml

Traceback/Error logs

Wed Feb 26 2020 10:59:09 GMT-0700 (Mountain Standard Time)
Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/real_ip.py", line 39, in real_ip_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 72, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 135, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 123, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 130, in post
    return await super().post(request)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 50, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 62, in post
    handler, context={"source": config_entries.SOURCE_USER}
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 109, in async_init
    result = await self._async_handle_step(flow, flow.init_step, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 172, in _async_handle_step
    result: Dict = await getattr(flow, method)(user_input)
  File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 107, in async_step_user
    all_hosts = await self.hass.async_add_executor_job(self.controller.discover, 5)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/homekit/controller/controller.py", line 90, in discover
    return discover_homekit_devices(max_seconds)
  File "/usr/local/lib/python3.7/site-packages/homekit/zeroconf_impl/__init__.py", line 102, in discover_homekit_devices
    info.properties
  File "/usr/local/lib/python3.7/site-packages/homekit/zeroconf_impl/__init__.py", line 180, in parse_discovery_properties
    d['category'] = Categories[int(category)]
  File "/usr/local/lib/python3.7/site-packages/homekit/model/categories.py", line 94, in __getitem__
    raise KeyError('Item {item} not found'.format(item=item))
KeyError: 'Item 33 not found'
Connection lost. Reconnecting…

Additional information

This worked on my Raspberry Pi 3B+ 105.5

probot-home-assistant[bot] commented 4 years ago

Hey there @Jc2k, mind taking a look at this issue as its been labeled with a integration (homekit_controller) you are listed as a codeowner for? Thanks!

Jc2k commented 4 years ago

Cheers @JamesDenby.

For context, https://community.home-assistant.io/t/homekit-accessory-integration-fails/174636/2.

As i said on the forums, the underlying library has never supported category 33 so this is not a regression in the traditional sense.

But given you said this worked before there are 2 main scenarios I see:

You previously paired your device from the "Discoveries" list. This ignores the category and so worked. This time you did a manual add. This uses the discovery from the upstream library. This is more thorough and strict and didn't recognise category 33.
You bought some new equipment (or upgraded some firmware on existing equipment). You know have a HomeKit device that you didn't last time you paired. This new accessory is category 33 and is breaking home-assistant from discovery any HomeKit device.

Interestingly you use eero networking gear, and eero is one of the few providers that do (or plan to) support HomeKit routers. Looking at the JS HomeKit library, category 33 is a HomeKit router. This is a feature that lets HomeKit push firewall rules into your networking gear, and was only released very very recently - see here.

For now i would edit line 180 of /usr/local/lib/python3.7/site-packages/homekit/zeroconf_impl/__init__.py and change:

    d['category'] = Categories[int(category)]

to just

    d['category'] = "unknown"

Home Assistant doesn't use this field so its a safe change and should allow discovery to complete.

We are migrating to a different homekit library, so in parallel to this I will make sure it knows about routers but also make sure it doesn't fail the entire discovery process if its an unknown category.

JamesDenby commented 4 years ago

Jc2k,

Thank you for the rapid response. Outstanding!

But, I cannot seem to get to the file to edit it. I tried through Terminal and get a bash error when I try to go past /usr/local/lib/

Any chance you could guide me to that file so that I may change it?

Jc2k commented 4 years ago

I'll try but theres a limit to how much i can intuit about other peoples environments 😺

The traceback you posted shows that the package is under /usr/local/lib/python3.7/site-packages/ so the fact that you can't get to it really indicates to me:

it's a permissions thing
it's a container or vm thing

Really need to know what OS you are using, what commands you are typing and what errors you are seeing.

In your original post you mentioned Home Assistant Core. But you don't say whether its Home Assistant OS, Raspbian or something else. But looking at the fact that your home assistant code seems to be in /usr/src/homeassistant i'm going to go out on a limb and say you are using Docker and the official container and given its a Pi - potentially via Home Assistant OS?

I have never used Home Assistant OS myself so don't know if these are valid instructions but i use the official Docker container on Ubuntu and can do:

docker ps

To see a list of containers running. I find the id for the home assistant container and then do

docker exec -it abcdef123 /bin/sh

And i've now got a shell inside the container. At that point the commands you were using before might now work.

JamesDenby commented 4 years ago

Thank you. I am using Home Assistant OS on a Pi (I believe it is docker). The problem is since it is Home Assistant OS I do not have permission to change the file. Even though I have "root" access through terminal, the OS does not allow any modifications outside of the config directory.

I guess I am stuck until the newer version is finished.

Jc2k commented 4 years ago

That's weird - i'm sure i've helped people with HA OS before. Are you definitely root inside the container and not just outside it? I.e. what is the output of id inside the container? If you aren't root inside the container you can do something like:

docker exec -u root -it 38aeedf833d5 sh

JamesDenby commented 4 years ago

Yeah, I think I am not really root, and the Home Assistant OS doesn't allow anyone but the chosen developers access. Here is what I get when I try your docker suggestion:

NewMacBook:~ Colonel$ ssh root@192.168.4.106
 _    _                                         _     _              _   
| |  | |                          /\           (_)   | |            | |  
| |__| | ___  _ __ ___   ___     /  \   ___ ___ _ ___| |_ __ _ _ __ | |_ 
|  __  |/ _ \| '_ ` _ \ / _ \   / /\ \ / __/ __| / __| __/ _` | '_ \| __|
| |  | | (_) | | | | | |  __/  / ____ \\__ \__ \ \__ \ || (_| | | | | |_ 
|_|  |_|\___/|_| |_| |_|\___| /_/    \_\___/___/_|___/\__\__,_|_| |_|\__|

Our command line:
$ ha help

~ $ docker exec -u root -it 38aeedf833d5 sh
-bash: docker: command not found
~ $

Jc2k commented 4 years ago

OK looks like you are really root, just that the default shell is the HA cli. At that prompt, type login. Then you should have a real shell and docker ps should work.

JamesDenby commented 4 years ago

It works! ... except it won't take 'root' as the login

Jc2k commented 4 years ago

Not really sure what isn't taking root as the login - do you mean the docker exec? If so try it without, what happens.

JamesDenby commented 4 years ago

Actually, it is root not being taken when I enter Login.


Our command line:
$ ha help

~ $ login
core-ssh login: root
Login incorrect
core-ssh login:

Jc2k commented 4 years ago

Hi @JamesDenby - sorry for the delay in getting back to you. I'm at a loss on how to help with SSH in your environment, but this should be fixed in 0.107.0 when its released (the first beta was just tagged so hopefully not far away). I'm going to close this now, but please do open another issue or ping me if it doesn't resolve it for you.

JamesDenby commented 4 years ago

Thank you for letting me know. I am looking forward to 0.107.0. I will certainly let you know if I still have a problem once that is released

jd1900 commented 3 years ago

Actually, it is root not being taken when I enter Login.
Our command line:
$ ha help

~ $ login
core-ssh login: root
Login incorrect
core-ssh login: 

Did you find a solution to this? I could become root with "login", but now I get "Login incorrect" as well.

Jc2k commented 3 years ago

Hi! This is a closed ticket for a homekit_controller bug that was fixed 8 months ago, and your issue seems seperate to the homekit_controller bug that the issue is for. The community advice is not to comment on closed tickets, especially if its not to say something about the original problem. I don't use HA OS so have no idea how to help you. For something like this you are better asking on Discord or the forums. Thanks.

home-assistant / core