change core-export-service to a driver

[sevenEng]

As this has been metioned since plenary meeting, and also related to recent manifest v2 migration so open an issue here to provide a place of discussion and progress tracking

[sevenEng]

Some thoughts about this:

IIUC, current interaction model between apps and drivers are like: drivers provide datasources by writing data to stores for apps to retrieve, apps read from those datasources and may also produce some data in its own store for other apps to consume, things are initiated by drivers and data flow unidirectionally from drivers to apps (then perhaps to other apps)

and for interactions between apps and export-service, the app has to somehow make some data known to export-service at the first place, like url to export to and the payload data, so that export-servcie could start working on that, here the work is initiated by apps. If export-service is being treated like a specific driver, it means data have to be able to flow from apps to drivers too. It seems there is no existed out-of-box mechanism in the platform that supports this? To make this happen, we may have to allow an app to write to a driver's store? What do you think? @mor1 @Toshbrown @yousefamar @haddadi

[yousefamar]

I think that that (apps writing to driver stores) should work already. Like with actuators, they would just need write permissions to that driver's store for a certain URL (and read permissions if they're expecting a reply, the two linked with a UID like with current export service).

As far as access control/ permissions goes, I think we can go about it a bit more elegantly (rather than cramming everything into paths and/or making new rules for different store types). I think we can define constraints inside hypercat rels for "general purpose" caveats, like this. The simplest case is where e.g. we require that a POST param exactly match some string (like the URL of an export destination). Fancier cases are equality operators for numbers, and wildcards and such (/^https*/g).

As a plus, this would also let us generalise the code for APIs like /ts/range such that we don't need to repeat ourselves for every new kind of data/api, i.e. we can just have a hypercat rel for URL or POST param from and to and they must both be greater than some number. Then later, say we wanted to have some caveat for image data, we can reuse the same definition for x-resolution and y-resolution to be lower than some number etc. Very DRY.

[sevenEng]

@yousefamar ha didn't know that we could do this, could you point me to some apps that use/have/act as an actuator? I need to see and understand how they work, also with respect to access control and permissions

[mor1]

@sevenEng @yousefamar yes, AIUIm macaroon caveats entail designing a DSL. If we do this (and I suspect we should-- packing everything into the path is inelegant indeed) then (a) the arbiter gets way more interesting :) and (b) we must be very careful about the completeness of the language we introduce. Allowing a DOS via a non-terminating caveat would be careless...

[Toshbrown]

@sevenEng the driver-philips-hue supports them (here) we have no demo apps that use them but toms SDK apps use them

[cgreenhalgh]

I did some thinking about this a while back (about an RPC/passthrough store type in general). My current demo is also throwing up use cases for it, e.g. a driver to send emails.

I've sketched out a possible job queue store API which I think might do the job.

The main permissions implication would be the need for some kind of client identification/authentication (needn't actually be client authentication; could just be a bearer identity in the macaroon) so that different clients to the same service can't access each other's requests.