Move to v2 manifests

[mor1]

Once specification finished, move to v2 manifests. Requirements that seem to be arising:

• [ ] Validation of provided manifests by the container-manager and app-store
• [ ] Add explicit reference to the registry and image name
• [ ] [drivers] Add explicit definition of data schema for each data type provided
• [ ] [drivers] Add explicit reference to the external services that should be contactable
• [ ] [drivers] Allow external services to be wildcarded to support the export-service

[Toshbrown]

As part of this work, the container-manager and/or the app-store should validate provided manifests.

[Toshbrown]

When installing app/driver the container-manager make assumptions about where the relevant image for the component is located. It would make sense to add registry and image name.

[yousefamar]

:+1: :+1: :+1: re registry and image name in manifest; something that always irked me. :)

[sevenEng]

and maybe also some fields to include a list of external URLs that drivers need to talk to cause all the apps/drivers will be put into internal networks by core-network by default need this info to enable communications

[Toshbrown]

To aid with development, both manually and with the graphical SDK. Drivers should specify a data schema for each data type that they provide. This would give app devs the information they need to write their apps.

We could also use this to ensure that if, two drivers offer the same data source type that they are compatible.

[mor1]

I've summarised requirements from the thread above into the top comment.

[mor1]

@Toshbrown @tlodge @yousefamar @pooyadav Where does the code that "implements" manifests sit currently?

[pooyadav]

Manifests are posted on app server - https://github.com/me-box/platform-app-server

On Tue, Oct 31, 2017 at 11:44 AM, Richard Mortier notifications@github.com wrote:

@Toshbrown https://github.com/toshbrown @tlodge https://github.com/tlodge @yousefamar https://github.com/yousefamar @pooyadav https://github.com/pooyadav Where does the code that "implements" manifests sit currently?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/me-box/databox/issues/40#issuecomment-340737997, or mute the thread https://github.com/notifications/unsubscribe-auth/AExIaMsfR_uUFFGn5oilKuN_PR1bQgztks5sxwgPgaJpZM4NPcWH .

[mor1]

@pooyadav not quite the question I asked-- where is the code that uses them? Just code in the app-server, or code elsewhere? (I presume elsewhere as well, eg the arbiter?)

[yousefamar]

AFAIK, the only place that actually directly touches raw manifests is the dashboard, which requests them directly from the app server client-side like here. Everything else deals in SLAs which the container manager handles all over here.

The container manager translates the SLA into explicit caveats, and the arbiter exposes a much lower level API to set these directly. The arbiter is immune to SLA spec changes; as long as the CM can turn the SLA into caveats, everything works the same.

Also one thing to note: the app server does not validate uploaded manifests (not yet anyway; big TODO on that). So it's also currently immune to spec changes, as long as it's valid JSON. Depending on how the specs change, currently the only code that would need updating is in the CM.

[mor1]

@yousefamar Ok, to phrase the question the other way-- how are SLAs constructed, by what, and in response to what input (a manifest presumably)?

(Apologies if I'm being dense, but I couldn't immediately see where this was done.)

[Toshbrown]

"It goes a little something like this"

1. UI gets manifest from app-server
2. User goes through install process (selecting data sources, blar, blar ) generating the SLA
3. UI POSTs the SLA to container manager endpoint (/install)
4. container manager parse SLA and installs the required app/driver and store
5. container manager registers each component with the arbiter (using arbiter endpoint /cm/upsert-container-info)
6. container manager setup permissions requested in the SLA (using arbiter endpoint /cm/grant-container-permissions)

so the UI converts the manifest to an SLA based on user input.

[sevenEng]

to Tosh's point: 1 is at https://github.com/me-box/core-container-manager/blob/master/src/www/js/app-install.js#L118 or https://github.com/me-box/core-container-manager/blob/master/src/www/js/app-install.js#L140 2 is done is the same module as 1 3 is https://github.com/me-box/core-container-manager/blob/master/src/www/js/app-install.js#L47 and 4 is https://github.com/me-box/core-container-manager/blob/master/src/server.js#L277

at 3 you see body: JSON.stringify(manifest), and when the request comes to 4, there is const sla = req.body :ghost:

[mor1]

@Toshbrown ...so it's the UI code that actually generates the SLA from the manifest plus some user input? Can you point me at the code? I presume it's server-side UI code that actually does the SLA generation (we can't leave that to something running on a client device)?

(@sevenEng, ah thanks :) message arrived as I was typing 👍

It seems to me then that we need to do validation of manifest in the app-server (when app is published) and the UI, and of the generated SLA in the container-manager on receipt (rather than simply believing what we're given. I guess we may also want to separately register the manifest, the user choices, and the generated SLA if, eg., we want to be able to show back to the user what they picked for that app install so that they can modify subsequently (cf. the "I installed a new lightbulb" case).

(Also, @ktg, git blame points at you for that line. But in general, surely everyone who ever makes a network-accessible webservice must make sure that they groom user-provided input ... :(

[sevenEng]

[drivers] Add explicit reference to the external services that should be contactable

For this one, a first cut could be an external-whitelist field inside the manifest, e.g for driver-twitter:

external-whitelist: [
    {
        url: [  "https://api.twitter.com", 
                "https://stream.twitter.com",
                "https://userstream.twitter.com"  ],
        description: "Enable package 'twit' to retrieve data from Twitter endpoints"
    }
]

From CM, this information will be picked up, and call on core-network to enale communication. I have seen a field network-permissions left blank in some manifests, are they trying to serve a similar pupose to this?

Merging in core-network depends on the availability of such a field, so that drivers that need to communicate with "outside" world could work as expect in a denying-everything-by-default setup, which is what core-network imposes. Once the format is fixed, then I could start patching CM so that it knows how to configure core-network to enable communications.

[Toshbrown]

@sevenEng this looks like a good suggestion network-permissions was meant to fulfil this role but the name can be decided later.

I don't see this as a blocker for getting the core network merged in though. Core-network can just expose an endpoint to the container manager (like the arbiters grant-container-permissions endpoint). Let the container manager handle the SLA and call the core-network endpoint as required to set up the permissions.

For now, the CM could be altered to call this new endpoint with "allow all" (https://.) when a driver is installed.

A couple of questions:

1. some drivers need access to the local network to scan for devices, is this granted by default? if not how is it requested?

2. Are the urls in the external-whitelist permissive or do they need to be specified exactly (https://api.twitter.com vs https://api.twitter.com/statuses/user_timeline) or is it host-level access.

3. Do we limit them at a protocol level GET and POST

4. Do we only allow HTTPS, what about other protocols (wss, IMAP etc)

5. Do we filter inbound traffic?

Lots of questions, maybe we need an Issue to document how it currently works so we can discuss some of this in more detail

[sevenEng]

I just add some documentation to core-network, hope the Internals section could answer most of questions here.

Basically, it only operates at IP level, with exceptions for DNS only, so TCP/UDP and even higher protocols HTTPS, wss, IMAP are all transparent to this component (at the moment, but finer control could build atop if needed). So yeah, it's host-level access for 2.

Inbound traffic are also filtered, in the sense that if the external entity who wants to send inboud traffic not made known to core-network during related app/driver's installation phase

for the scan of local networks, i'm sure that it's not granted by default, but I may need to understand how thoses drivers work before come up with a solution for this, I guess for example driver-philips-hue needs this? @Toshbrown

[mor1]

On 2 November 2017 at 14:42, Qi LI notifications@github.com wrote:

I just add some documentation https://github.com/me-box/core-network/commit/8d4c906b7173510270d84b471d172e637dd74308 to core-network, hope the Internals section could answer most of questions here.

Basically, it only operates at IP level, with exceptions for DNS only, so TCP/UDP and even higher protocols HTTPS, wss, IMAP are all transparent to this component (at the moment, but finer control could build atop if needed)

don't understand:

IP level, with exceptions for DNS only, so TCP/UDP and even higher protocols ...

?

Inbound traffic are also filtered, in the sense that if the external entity who wants to send inboud traffic not made known to core-network during related app/driver's installation phase

for the scan of local networks, i'm sure that it's not granted by default, but I may need to understand how thoses drivers work before come up with a solution for this, I guess for example driver-philips-hue needs this?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/me-box/databox/issues/40#issuecomment-341443230, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKP4KE4Tbjc3rF3BjJ1vxtudA0CqBWRks5sydTJgaJpZM4NPcWH .

-- Richard Mortier mort@cantab.net

[Toshbrown]

@sevenEng thanks, I'll take a look at the documentation

For device discovery, there are a couple of scenarios at the moment

1. the Phillips hue driver is a one-off pairing where the IP of the Phillips hue bridge is provided by the user in the driver configuration interface after install.

2. the tp-link plugs driver actively scans a whole subnet looking for plugs

and in the future, we may want to support service discovery using mdns etc

The basic problem is that at instal time these drivers do not know the IPs of the devices they need to communicate with. Early in the project, there was some discussion around a databox-network-discovery component to handle device discovery but I'm not sure how this was left.

cc @mor1 do we need a databox-network-discovery component?

[mor1]

@Toshbrown we probably need something. i'm not aware of any detailed discussion of a core-network-discovery component. if it's just mdns then it seems to me that this ought to be relatively straightforward either added to core-network or as a new component (preferred, subject to resource constraints). If drivers will need to initiate more complex / chatty discovery protocols, this is more complex and may require more thought.

[sevenEng]

2017-11-02 15:07 GMT+00:00 Richard Mortier notifications@github.com:

On 2 November 2017 at 14:42, Qi LI notifications@github.com wrote:

I just add some documentation https://github.com/me-box/core-network/commit/ 8d4c906b7173510270d84b471d172e637dd74308 to core-network, hope the Internals section could answer most of questions here.

Basically, it only operates at IP level, with exceptions for DNS only, so TCP/UDP and even higher protocols HTTPS, wss, IMAP are all transparent to this component (at the moment, but finer control could build atop if needed)

don't understand:

IP level, with exceptions for DNS only, so TCP/UDP and even higher protocols ...

?

What I wanted to say here is that whenever get an IP packet, it checks if the packet encapsulates DNS queries, if so, it will try to respond to that, otherwise, it just modifies the src and dst fields in the header and forward it to one of the networks it connects to, it doesn't try to recognize transport layer or other application layer protocols

Inbound traffic are also filtered, in the sense that if the external entity who wants to send inboud traffic not made known to core-network during related app/driver's installation phase

for the scan of local networks, i'm sure that it's not granted by default, but I may need to understand how thoses drivers work before come up with a solution for this, I guess for example driver-philips-hue needs this?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/me-box/databox/issues/40#issuecomment-341443230, or mute the thread https://github.com/notifications/unsubscribe-auth/ AAKP4KE4Tbjc3rF3BjJ1vxtudA0CqBWRks5sydTJgaJpZM4NPcWH .

-- Richard Mortier mort@cantab.net

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/me-box/databox/issues/40#issuecomment-341451663, or mute the thread https://github.com/notifications/unsubscribe-auth/AFo_ogHIfWAY5GahZd5yJA999ztVQJ-Nks5sydqpgaJpZM4NPcWH .

-- Qi LI Research Assistant, Computer Laboratory, University of Cambridge (+44) 07719 685333 ql272@cl.cam.ac.uk

[mor1]

@sevenEng Ah, fair enough. I suspect interception of DNS is enough for now, maybe DHCP too? What about mDNS for some of the discovery issues mentioned?

[yousefamar]

(Just a note about Hue Bridge: we could get its IP from the SSDP packets it spams non-stop, although I've read that tweaking network for Docker + UPnP is not that straightforward (something about multicast across subnets).)