Discovering devices on the local network

[Toshbrown]

There has been some discussion of this in https://github.com/me-box/databox/issues/40#issuecomment-341458494 moved to a new issue to make more visible.

So that the databox UI can suggest drivers to install and to prevent the need to give drivers full access to the local network (so they can do their own scans). Databox needs to support device discovery on the local network.

@jcolley has a prototype docker container that performs this task and was asking how to integrate it into databox.

Currently, we can't do this as a driver and app pair because the discovery service needs access to the host network. We also think it makes sense to integrate this functionality into the databox UI rather than as an app to make the user experience smoother.

Therefore, device discovery needs to be part of the core databox components. It should scan the local network and write to a store what it finds. This datasource can be made available to the container manager UI (by default) and other apps if requested in their manifest.

Questions:

1. Should this be part of the core-network or a separate service started with host networking?
2. @jcolley what discovery strategies need to be supported for this to be useful?

cc// @mor1 @sevenEng @cgreenhalgh any thoughts?

[cgreenhalgh]

I think there is a bit of a chicken-and-egg situation in that discovery for some devices is liable to be idiosyncratic, i.e. you'll need some device-specific discovery. Of course that doesn't have to be a full driver; perhaps it could just be some configuration information in the driver manifest (e.g. what discovery protocol, what service identifiers to search for). But it probably does mean working through some real examples to see what functionality might be needed.

I agree that a discovery component is generally special in needing host network. I also agree that is special in the sense that the UI should routinely be able to access it. If the configuration argument above holds then it will also be special in that the UI and/or CM will have to provide it the driver-specific discovery config. And it should probably be auto-installed and hard to uninstall.

But having said all that I think it could probably be a driver (all be it one with an unusual and high-risk host network access).

And I wonder if it would make sense to support several simultaneous discovery drivers (at least in principle), with each one supporting one or more discovery protocols.

[ghost]

My approach at the moment is to use identifying characteristics that can be used to make a best guess as to what device is what.

Im gathering as much information about each device on the network using node libs for NMap arp and ssdp and then storing this (its basically a glorified network scanner). My next step will be to standardise the fields and to start creating an online directory of device network characteristics that can be used to identify a device. Each feature should have a score and if the feature is high enough then it's probably that device. e.g. MAC address ties to Phillips and host name includes "hue" and has port 80 open and gives response X when a GET request is made to port 80/ on the device then it is probably a hue bridge.

I would recommend that this directory is a databox approved service that can be contributed to and that each entry in the directory would have a field for "default databox driver" so that if need be in the future drivers could be auto installed.

Cheers,

James.

On Tue, Nov 14, 2017 at 9:06 AM, Chris Greenhalgh notifications@github.com wrote:

I think there is a bit of a chicken-and-egg situation in that discovery for some devices is liable to be idiosyncratic, i.e. you'll need some device-specific discovery. Of course that doesn't have to be a full driver; perhaps it could just be some configuration information in the driver manifest (e.g. what discovery protocol, what service identifiers to search for). But it probably does mean working through some real examples to see what functionality might be needed.

I agree that a discovery component is generally special in needing host network. I also agree that is special in the sense that the UI should routinely be able to access it. If the configuration argument above holds then it will also be special in that the UI and/or CM will have to provide it the driver-specific discovery config. And it should probably be auto-installed and hard to uninstall.

But having said all that I think it could probably be a driver (all be it one with an unusual and high-risk host network access).

And I wonder if it would make sense to support several simultaneous discovery drivers (at least in principle), with each one supporting one or more discovery protocols.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/me-box/databox/issues/195#issuecomment-344190935, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQr9NzzGqcOPop1a0Llj1xIRj9WA-Avks5s2VgsgaJpZM4Qc9nb .

[yousefamar]

Just adding a point from another thread for posterity: https://github.com/me-box/databox/issues/40#issuecomment-341493694

[mor1]

I'll talk to @sevenEng about this this afternoon. But off the top of my head, there's two things here:

1. Access by drivers to tx/rx multicast/broadcast packets (such as mDNS etc)
2. Active discovery by a driver using all available means

I believe that core-network ought to allow us to handle (1) effectively, though development may be required.

I believe that what @jcolley refers to above is (2). This seems to me something that should fit into the driver model, possibly requiring support from core-network, and could be provided (and extended, updated) as a single default-installed driver (but still a driver).

@jcolley It sounds like you've got a start on this already-- so please could you add a comment to this thread that lists the devices you're supporting already and how you're discovering them? (Or point to some suitably informative piece of code in your existing setup?) Getting some kind of catalogue of discovery mechanisms will be useful so we can check that we don't block supporting any of them, and hopefully start ticking them off...

@Toshbrown How the results of discovery are surfaced, I haven't a strong view on yet. Fully automating installation of drivers seems dubious wrt accountability, but certainly some kind of "click here to automatically install the following four drivers" seems plausible. Having some store that records what's discovered and what's currently installed on Databox seems quite possible too (though not sure of utility :)

[mor1]

@jcolley (/cc @andyc1 @drdrmac ) We really need information about what you need supported for FACT or we can't factor it into the roadmap.

• What devices/services do you need support for?
• What data do you need to be able to pull off them?
• What actuations do you need be able to initiate?
• What other interactions do you need supported for each device? (Discovery? Sensing?)

Even a pointer to the current container that @Toshbrown mentions above would be helpful.

In the absence of other input, we're just going to have to proceed on the roadmap as currently written. This may or may not produce what you need... :(

[ghost]

Hi mort,

We've just got the applications in for the FACT demo and so will be able to provide more information after i've sat down with those tomorrow.

However without doing that I can say the following:

• What devices/services do you need support for?

Parameter based querying on the datastore e.g. filtering on values (support for <,>,=,!,) Aggregation (temporal) queries on the datastore e.g 5 minute average Support for the new datastore in python and node libraries So far as devices are concerned - I'm writing the drivers that we've promised to them myself

• What data do you need to be able to pull off them?

Tied to response above, I'll be writing the drivers but the existing store is sufficient (once the bugs are ironed out).

• What actuations do you need be able to initiate?

Again i'm writing the drivers so I'll be dealing with the actuation

• What other interactions do you need supported for each device? (Discovery? Sensing?)

discovery and sensing won't be needed for the FACT demo as it will be canned.

I'll update with more detail and examples once we've review the applications (later this week).

On Mon, Nov 27, 2017 at 11:04 AM, Richard Mortier notifications@github.com wrote:

@jcolley https://github.com/jcolley (/cc @andyc1 https://github.com/andyc1 @drdrmac https://github.com/drdrmac ) We really need information about what you need supported for FACT or we can't factor it into the roadmap.

• What devices/services do you need support for?
• What data do you need to be able to pull off them?
• What actuations do you need be able to initiate?
• What other interactions do you need supported for each device? (Discovery? Sensing?)

Even a pointer to the current container that @Toshbrown https://github.com/toshbrown mentions above would be helpful.

In the absence of other input, we're just going to have to proceed on the roadmap as currently written. This may or may not produce what you need... :(

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/me-box/databox/issues/195#issuecomment-347149585, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQr9EexqUPQuF80PUUkZ_RND6EJZBASks5s6pdXgaJpZM4Qc9nb .

[mor1]

Ping-- did we get the updates through other channels? Are things in place or at least in progress as required? Can this issue be closed?