search

home-assistant / architecture

Repo to discuss Home Assistant architecture
317 stars 99 forks source link

Allow Config Entries to be discovered #121

Closed balloob closed 4 years ago

balloob commented 5 years ago

Right now the netdisco package is used for discovery. Routing all discovery through netdisco has the problem that we now have a centralized place that has to support an ever growing list of devices that can be discovered, some of them with custom discovery protocol implementations. We also discover things that Home Assistant doesn't support. Result is that netdisco is getting bloated. I've already stopped approving new discovery protocols and plan to remove all non-standard discovery protocols.

But this will leave a discovery gap. And I do think that discovery is important to get users up and running during onboarding and when they want to set up things later.

Proposal

We freeze adding any custom protocols to netdisco (as per https://github.com/home-assistant/netdisco/issues/230)

We will no longer discover every X minutes. Instead we discover once during onboarding and when the user navigates to the integrations page in the UI. We will initially allow a discovery legacy mode to keep running netdisco each X minutes.

Config Entries will be able to register how they can be discovered: upnp/SSDP, mDNS/zeroconf/bonjour or with their own custom method. Our new discovery mechanism will loop over all config entries and see how they can be discovered.

If integration defines a custom discovery, we will install the component requirements and will call that method, the config entry can now leverage the lib for discovery. Installing requirements seem like a lot of useless installation and loading modules, but it's not that bad: on Hass.io/Docker the packages are already installed. We also will only run discovery during onboarding and whenever the user navigates to the integration page. So during normal operation, these packages won't be loaded.

We will keep a list of popular config entries that will be included when doing discovery. This list contains all that rely on standard protocols and a couple popular integrations that use a custom discovery (ie Philips Hue nupnp). A user can have the option to do an "extended" discovery that will include all integrations with custom discovery.

Eventually I want to deprecate the netdisco package. The code from netdisco that we depend on will be included in Home Assistant as part of the discovery component.

Some code to illustrate

class HueDiscovery(config_entries.ConfigDiscovery):

    async def async_ssdp_discovery(self, ssdp_results):
        entries = ssdp_results.find_by_device_description({
            "manufacturer": "Royal Philips Electronics",
            "manufacturerURL": "http://www.philips.com",
            "modelNumber": ["929000226503", "BSB002"]
        })

        for entry in entries:
            self.hass.async_create_task(self.hass.config_entries.flow.async_init(
                DOMAIN, context={'source': config_entries.SOURCE_DISCOVERY},
                data={
                    'host': entry['host'],
                    'path': entry['path'],
                }
            ))

@config_entries.HANDLERS.register(DOMAIN)
class HueFlowHandler(config_entries.ConfigFlow):

    @staticmethod
    @callback
    def async_discovery():
        return HueDiscovery()

Advantages

Disadvantages

MartinHjelmare commented 5 years ago

What do you mean with the last point?

Only supports config entries when legacy mode dropped from discovery

balloob commented 5 years ago

When we drop legacy discovery mode based on Netdisco, we would no longer discover integrations that rely on passing discovery_info to async_setup_platform. These: https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/discovery.py#L58

rytilahti commented 5 years ago

So it's been a while since I pondered about making the discovery nicer, so I'm definitely :+1: for getting this cleaned. As this topic has been on my thoughts almost as long as I have used homeassistant, I will give my 0.02e on the topic.

In contrast to @balloob's proposal, I think that instead of making it harder to make devices discoverable, we should empower everyone to make their components/platforms more discoverable (adding discoverability to IQS?). Note that I'm not against sunsetting netdisco nor against the idea of enabling the component/platform developers to implement their custom discovery handling where it is supposed to be (that is, near the component/platform code), merely pointing out that we can do better for the regular developers with no special needs.

As this comment snowballed into way bigger than I expected, here's a short overview of what I propose (on top of @balloob's proposal):

I'll start by defining a couple of user stories to show how my proposed changes would improve the situation even further.

User stories

  1. User adds a new (supported) device to the network 1.1. A notification is displayed to guide the user to decide if he wants to configure the device: 1.1.1 If unwanted. Either dismiss (will be shown again later) or disable (no future notifications from this device shall be shown). 1.1.2. If wanted, click on Configure leads to the configuration page, where the all potential platforms/components supporting the device are shown, and can be activated and configured.

  2. User wants to configure integrations 2.1. Discovery process is done as a whole, discovering all (even the disabled) devices and providing the opportunity to configure them. 2.2. If other platforms support the already configured device, the user is given a choice to configure those platforms to be activated for the device.

  3. User wants to remove a configured device 3.1. Go to configure integrations, and simply removes the configuration as it is done already.

Developer stories

  1. Developer wants to make the new platform discoverable 1.1. The developer adds a single file defining how the platform can be discovered using mDNS or UPnP.

    • No need to modify netdisco and/or core (discovery component) and/or convert the platform into a full component.

  2. Developer wants to use a custom discovery protocol 2.1. Developer has to use config entry and implement the custom discovery protocol.

Implementation

In order to making the implementation as straightforward as possible, a new meta-data file called metadata.<FILEFORMAT> shall be stored alongside the component/platform. This metadata.<FILEFORMAT> file MAY be used by developers to define which mDNS service types or UPnP search targets (ST) they claim to support.

The components/platforms who have registered their wish to support the new device are loaded on demand and their respective setup_{component,platform} is called with the full discovery_info as delivered by the discovery module. The developer can read the wanted information directly out from the provided data structure and decide if is capable of supporting the given device and inform the user about it.

The biggest change besides the new metadata file is that the platforms are to be moved into their own directories. For example, light/yeelight.py shall be renamed to light/yeelight/yeelight.py and light/yeelight/metadata.<FILEFORMAT> can be created to enable discovering of Yeelight bulbs.

NOTE: Nothing here is set in stone, these are just to give an idea how different platforms could define what they support. The available configuration keys for the metadata are based on checking the current netdisco implementations, there is a list at the end of what types of checks the implemented discoveries do.

Reasoning for a new file and structure

Adding a new file and modifying the source tree structure should not be taken lightly, so I'm listing here some advantages and disadvantages of this change.

Some of these are completely unrelated to discoverability, but, in my opinion worthy reasons to make the change for the future's sake.

Advantages

  1. Cleaner directory structure and separation, all light.yeelight platform files are under light/yeelight/ and not under light/ among other platforms.

    • Developers can, if they wish so, split their huge platform files into smaller chunks.

  2. No need to have a central file for supported discoveries, nor need modify the core code to add new ones (https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/discovery.py#L58).

  3. The file could also be re-used in the future for various other purposes:

    • Defining REQUIREMENTS and DEPENDENCIES, meaning no more need to load the module to look these up.
    • Defining CODEOWNERS, no need to keep updating /CODEOWNERS, clearer ownership (and thus, PR/bug notifications)
    • Defining CONFIGURATION and SERVICES
    • Defining SUPPORTED_FEATURES (allows for easier IQS, effectively disallowing developers to change features on the fly)

  4. The metadata file will be easier to parse than python code, and can easily be exploited for documentation purposes (generating documentation blocks for services and configuration, ..)

Disadvantages

Backwards compatibility & required changes from developers

The existing implementations SHALL continue working without any changes for some later determined time period, and the conversion shall be made straightforward. The already available discovery_info argument will be re-used to pass the data as provided by the discovery protocol (mDNS, UPnP).

Component developers

Platform developers

Implementation details

SSDP

Note, that the file format is not important here, these examples just try to give an idea how such configuration would look like. Instead of ini files this could be YAML or JSON.

Generic media renderer (dlna_dmr)

[upnp]
st=urn:schemas-upnp-org:device:MediaRenderer:[1-3]

Huawei

[upnp]
manufacturer=Huawei Technologies Co., Ltd.
st=urn:schemas-upnp-org:device:InternetGatewayDevice:1

Deconz

[upnp]
modelDescription=dresden elektronik Wireless Light Control
manufacturerURL=http://www.dresden-elektronik.de

Hue

mDNS

Yeelight bulbs

[mdns]
service=_miio._udp.local.
name=^yeelink-light-

HP printers

[mdns]
service=_printer._tcp.local.
name='HP *'

Custom discovery protocols

Appendix 1. Common discovery protocols

UPnP/SSDP

How does it work

The gruntwork here is (to be) done in the backend library, but I think describing how these work will help to understand the big picture better.

  1. UDP multicast on port 1900 to search for wanted devices (by defining the wanted ST, or simply requesting all of them)

    • Devices respond with information about their offered services (including device type) and the URL to the service description URL
    • In order to get more information about the device (manufacturer, model, ..), it is necessary to perform a HTTP GET request to obtain the given service description file.

  2. Live updates from devices (ssdp:alive, ssdp:goodbye)

    • Over UDP multicast
    • Devices send NOTIFY announcements when they join or quit, which can be used to detect new devices to notify the user about their configurability
    • For a proof-of-concept of a non-async "device tracker" based on this info, see https://github.com/rytilahti/homeassistant-binarysensor-upnp

Implementation

mDNS (multicast DNS)

How does it work

  1. UDP multicast on 5353 to request information about specific service type (or a list of all available service types)

    • Devices respond with their types, which can then be queried to obtain more information.

  2. Devices MUST announce when they come online

    • Can be used to listen for new devices, similarly to UPnP.

Implementation

Appendix 2. current netdisco discoverables

mDNS

SSDP

Custom

fbradyirl commented 5 years ago

Just wondering if a decision was ever finalised around this? Are PRs to Netdisco no longer allowed? I have a very standard PR open to discover Enigma2 based devices. Please let me know if this needs to be done in a different fashion. Thanks.