openhab / openhab-core

Core framework of openHAB
https://www.openhab.org/
Eclipse Public License 2.0
931 stars 429 forks source link

Extend service to suggest addons via generic IP scan #3936

Open holgerfriedrich opened 11 months ago

holgerfriedrich commented 11 months ago

PR #3920 introduced a (partly) generic addon finder, which allows to discover devices by sending single IP frames. This is useful for devices which do not actively announce and require a search request to a multicast address. Maybe the word scanning is a bit misleading, at the moment it is just a single search frame, and only once at start-up in the current implementation.

Note: if case you can use mdns or upnp, there are separate addon finders for this! If you want to detect a service running locally, there is one for processes as well.

The implementation is rather tuned to a single use right now - detecting KNX installations - but is intended to be extended towards release 4.2. This issue is to collect requirements from other addons which can be used when starting a generalization.

The configuration is via addon.xml and looks like this (see openhab/openhab-core#3948 for developer documentation):

    <discovery-method>
        <discovery-method>
            <service-type>ip</service-type>
            <discovery-parameters>
                <discovery-parameter>
                    <name>type</name>
                    <value>ipMulticast</value>
                </discovery-parameter>
                <discovery-parameter>
                    <name>destIp</name>
                    <value>224.0.23.12</value>
                </discovery-parameter>
                <discovery-parameter>
                    <name>destPort</name>
                    <value>3671</value>
                </discovery-parameter>
                <discovery-parameter>
                    <name>request</name>
                    <value>0x06 0x10 0x02 0x01 0x00 0x0e 0x08 0x01 $srcIp $srcPort</value>
                </discovery-parameter>
                <discovery-parameter>
                    <name>timeoutMs</name>
                    <value>5000</value>
                </discovery-parameter>
            </discovery-parameters>
            <match-properties>
                <match-property>
                    <name>response</name>
                    <regex>.*</regex>
                </match-property>
            </match-properties>
        </discovery-method>
    </discovery-methods>

Frames to be sent are specified in the parameter section, and dynamic replacement of source IP and port is possible. Any return frame is matched, only the catch all .* is supported.

addon detection method scan interval dynamic data parsed response comment
knx mulitcast frame (hex data) once srcIp, srcPort .* #3912 (merged)
ipcamera multicast frame (xml) tbd randomNumber .* #3943 openhab/openhab-addons#16090 (merged)
govee multicast frame tbd no .* requires listening on port 4002, openhab/openhab-addons#16109 (merged)
zway unicast, scan local network tbd none http code 200 see below
danfoss broadcast tbd none static content #4036 (merged)

(table to be continued)

Features to be developed (incomplete list):

openhab-bot commented 11 months ago

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-get-an-addon-to-show-up-in-the-new-addon-suggestion-finder-list/152127/13

Skinah commented 11 months ago

To summarize what is needed for the ipcamera addon as discussed in the forum link:

To discover ONVIF cameras it uses port 3702 and IP multicast address 239.255.255.250 using what is called WS-Discovery. It is just the following SOAP/XML packet sent.

<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:randomUUID()</w:MessageID><w:To e:mustUnderstand="true">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To><w:Action a:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action></e:Header><e:Body><d:Probe><d:Types xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dp0="http://www.onvif.org/ver10/network/wsdl">dp0:NetworkVideoTransmitter</d:Types></d:Probe></e:Body></e:Envelope>

Challenges identified are:

  1. In the above XML packet you need to use the function java.util.UUID.randomUUID() to replace the text randomUUID() It needs to be different for each scan.
  2. How to enter in XML contents into the addon.xml file without causing issues.
  3. What about when a reply could match multiple bindings, do we do the scan multiple times and then use a different filter, or can they be combined? Doorbird is a binding for that brand of ipcamera which can also do ONVIF, so it would match two bindings. Do we scan once and then filter the cached results differently in each binding, or do the whole scan multiple times?

Good news is that because the XML states NetworkVideoTransmitter in the request, all replies will only be from devices that the binding can talk to. No filtering needed on replies since this is just for just putting the binding forward to be installed where further logic can be used.

mherwege commented 11 months ago

3. What about when a reply could match multiple bindings, do we do the scan multiple times and then use a different filter, or can they be combined? Doorbird is a binding for that brand of ipcamera which can also do ONVIF, so it would match two bindings. Do we scan once and then filter the cached results differently in each binding, or do the whole scan multiple times?

If both bindings define the same discovery criteria, they will both be suggested. It is for the user to decide what to install. No 2 scans are required for this.

holgerfriedrich commented 11 months ago

@Skinah Would the following string work? (this is the result of a draft implementation) Is UUID fine?

<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:1534bd9d-f9c1-45cc-a747-311d6a76db3b<?xml version="1.0" encoding="UTF-8"?><e:Envelope xmlns:e="http://www.w3.org/2003/05/soap-envelope" xmlns:w="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dn="http://www.onvif.org/ver10/network/wsdl"><e:Header><w:MessageID>uuid:)</w:MessageID><w:To e:mustUnderstand="true">urn:schemas-xmlsoap-org:ws:2005:04:discovery</w:To><w:Action a:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</w:Action></e:Header><e:Body><d:Probe><d:Types xmlns:d="http://schemas.xmlsoap.org/ws/2005/04/discovery" xmlns:dp0="http://www.onvif.org/ver10/network/wsdl">dp0:NetworkVideoTransmitter</d:Types></d:Probe></e:Body></e:Envelope>
holgerfriedrich commented 11 months ago

@stefan-hoehn Mechanism for Govee https://github.com/openhab/openhab-core/pull/3920#issuecomment-1868307029 can be discussed here.

Developer docs are not yet merged, see openhab/openhab-core#3948.

request does only support hex bytes by now, so you could make it work if you use <value>0x7B 0x22 0x6D 0x73 0x67 0x22 0x3A 0x20 0x7B 0x22 0x63 0x6D 0x64 0x22 0x3A 0x20 0x22 0x73 0x63 0x61 0x6E 0x22 0x2C 0x20 0x22 0x64 0x61 0x74 0x61 0x22 0x3A 0x20 0x7B 0x22 0x61 0x63 0x63 0x6F 0x75 0x6E 0x74 0x5F 0x74 0x6F 0x70 0x69 0x63 0x22 0x3A 0x20 0x22 0x72 0x65 0x73 0x65 0x72 0x76 0x65 0x22 0x7D 0x7D 0x7D</value>

Not sure about port 4002.... do we have to listen on 4002 or is if source port of the response?

holgerfriedrich commented 11 months ago

@stefan-hoehn just updated the comment above to include an example....

stefan-hoehn commented 11 months ago

This is how it works:


<img width="690" alt="image" src="https://github.com/openhab/openhab-core/assets/5937600/e167f4dd-8008-479b-aebc-651ae9866712">

But I think noticing that we get ANY response would be enough, I would say.
mherwege commented 11 months ago

@holgerfriedrich Looking at this, and the ipcamera request, I think an alternative (non-hex) request should be supported.

holgerfriedrich commented 11 months ago

@holgerfriedrich Looking at this, and the ipcamera request, I think an alternative (non-hex) request should be supported.

:-)

Looks Like.... But we need to make sure to escape tags - otherwise the XML parser might do something wrong....

mherwege commented 11 months ago

@mherwege I don’t think that is the responsibility of the code. When defining the discovery criteria in the xml, it is sufficient to properly escape the special characters: https://stackoverflow.com/questions/1091945/what-characters-do-i-need-to-escape-in-xml-documents But that would be true for any of the finders.

holgerfriedrich commented 11 months ago

@mherwege Yes, lets go for a requestPlain which uses org.openhab.core.util.StringUtils.StringUtilsunEscapeXml().. Developers of bindings need to escape the characters mentioned here by hand. Better than using the hex editor to transform the query into something unreadable.

stefan-hoehn commented 11 months ago

I can confirm that the implementation works well already for the Govee-Binding. See #16109

holgerfriedrich commented 11 months ago

andrewfg commented Dec 27, 2023

@holgerfriedrich the ZWay binding is probably a candidate to be suggested via your IP addon finder. => WDYT?

https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.binding.zway#discovery

This is about the zway gateways connected to the LAN.

The documentation of the zway binding states the following:

Z-Way doesn't support any discovery protocol like UPnP for this purpose. That is why first all IP addresses in local network are checked on port 8083. If the server answers, a ZAutomation request (/ZAutomation/api/v1/status) is performed to ensure, the found server runs Z-Way.

This would be the first real "IP scan", i.e. iterating over all local IP addresses finding a gateway.

It would require a few new features:

Did I miss something, @andrewfg ?

andrewfg commented 11 months ago

Did I miss something

I think it pings all port 8083 on the subnet, and if the port is open it does a specific HTTP GET for a URL on that host:8083, and if it returns HTTP 200 it would become a suggestion.

holgerfriedrich commented 11 months ago

I think it pings all port 8083 on the subnet, and if the port is open it does a specific HTTP GET for a URL on that host:8083, and if it returns HTTP 200 it would become a suggestion.

Yes, this sounds reasonable, thanks.

A subnet could be quite large, I would limit it a /24 and exclude the broadcast address (such that we have to scan 255 IPs per interface at max).

mherwege commented 11 months ago

I don’t think we should allow scans on all interface. See discussion here, how it can cause drama with Docker: https://community.openhab.org/t/unsuccess-openhab-updated-from-4-0-2-to-4-1-0-on-docker-openhab-unresponsive/152282/24

OH does have a setting for the primary IP. Could we restrict it to thecsubnet of that iP?

I even think we may want to limit the current IP finder implementation to only use one interface (or make that configurable).

mlobstein commented 6 months ago

If regex matching is implemented then the epsonprojector and sonyprojector addons could be detected as many newer models respond to Control4 SDDP discovery probes as described here:

https://github.com/sammck/sddp-discovery-protocol/blob/main/README.md

SDDP is similar to UPNP/SSDP in that a multicast search frame is sent and any supported devices respond back to the multicast address with a packet containing device information. By using a regex these packets would be filtered to determine the device type and manufacturer.

andrewfg commented 6 months ago

@mlobstein I created #4234 for this..