Align with JSON-RPC?

[tlrobinson]

I think Matt mentioned this early on: what we have is fairly close to JSON-RPC. Maybe it's worth making it compatible?

JSON-RPC 1.0 uses an array of arguments, but 2.0 supports named parameters, so it would be almost identical to what we have (rename inputs to params, outputs to result).

One nice thing about using JSON-RPC is we could potentially expose WebPipes over more efficient transports like TCP sockets/WebSockets/ZeroMQ. When using HTTP we could drop the id and method field (since the "method" is in the URL). When using a socket, the method would just be the url or path, and id would be used to pair the response with the request.

[progrium]

Not a bad idea.

On Sun, Dec 2, 2012 at 1:19 AM, Tom Robinson notifications@github.comwrote:

I think Matt mentioned this early on: what we have is fairly close to JSON-RPC. Maybe it's worth making it compatible?

JSON-RPC 1.0 uses an array of arguments, but 2.0 supports named parameters, so it would be almost identical to what we have (rename inputsto params, outputs to result).

One nice thing about using JSON-RPC is we could potentially expose WebPipes over more efficient transports like TCP sockets/WebSockets/ZeroMQ. When using HTTP we could drop the id and method field (since the "method" is in the URL). When using a socket, the method would just be the url or path, and id would be used to pair the response with the request.

— Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4.

Jeff Lindsay http://progrium.com

[matthewhudson]

I approve. It's also a bit easier to explain to developers as "Federated RPC".

The id field is nice since it essentially behaves as a token/key which is useful in cases where a Block needs to handle state.

[tlrobinson]

I don't think the JSON-RPC id field is useful for tracking state since it's unique per request. I think we'll need another way to identify the pipeline.

On Dec 2, 2012, at 9:42 AM, Matthew Hudson notifications@github.com wrote:

I approve. It's also a bit easier to explain to developers as "Federated RPC".

The id field is nice since it essentially behaves as a token/key which is useful in cases where a Block needs to handle state.

— Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-10932422.

[progrium]

Headers. Either HTTP or if new JSON-RPC supports some kind of headers.

On Sun, Dec 2, 2012 at 9:45 AM, Tom Robinson notifications@github.comwrote:

I don't think the JSON-RPC id field is useful for tracking state since it's unique per request. I think we'll need another way to identify the pipeline.

On Dec 2, 2012, at 9:42 AM, Matthew Hudson notifications@github.com wrote:

I approve. It's also a bit easier to explain to developers as "Federated RPC".

The id field is nice since it essentially behaves as a token/key which is useful in cases where a Block needs to handle state.

— Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-10932422.

— Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-10932448.

Jeff Lindsay http://progrium.com

[matthewhudson]

A database Block perfectly illustrates the value of a "methods" key in the Block Definition.

{
    "name": "database",
    "description": "Store and fetch persistent data.",
    "url": "/block-endpoint",
    "methods": {
        "get": {
            "inputs": [],
            "outputs": []
        },
        "set": {
            "inputs": [],
            "outputs": []
        }
    }
}

To accomodate this new Block Definition format, a HTTP request would change in the following ways:

• The "inputs" wrapper would take the name of the method invoked.
• As of WebPipe Spec v0.2 we no longer use the 'inputs' wrapper array.

curl -i -X POST -d '{"methodName":{"inputName":"inputValue", ... }}' -H "Content-Type: application/json" http://block-endpoint

Thoughts?

[matthewhudson]

Or, if we went full JSON-RPC:

curl -i -X POST -d '{ "method": "get", "params": ["database_key"], "id": null }' -H "Content-Type: application/json" http://block-endpoint

Using the database Block Definition I mentioned in my last post, this example would return a record for a given database key.

[tlrobinson]

Including a method name in the request body is very un-RESTful, but then again what we have so far isn't exactly RESTful either (and JSON-RPC definitely isn't either).

It would be nice if we could just use GET methods to enable caching, etc, but GET requests don't have a request body for us to include the inputs. Inputs could be mapped to the query string (or even include the JSON in the URL).

The HTTP methods could be mapped to the method field for other JSON-RPC transports. If more than one block were handled by a single JSON-RPC server the method could include the url. GET:/my-database or something.

Also we definitely shouldn't use JSON-RPC 1.0 positional arguments. JSON-RPC 2.0 supports named arguments.

[matthewhudson]

I'm sorry, I wasn't as clear as I could have been with my database example. We aren't passing a HTTP Method as an argument, we're passing the name of the "remote procedure" we want to run. In JSON-RPC, the "method" is the remote procedure you want to invoke.

In my example, the database block exposed two methods/functions/remote procedures - 'get' and 'set'. 'Get' would return a database record, and 'set' would save a database record.

As for REST, we're pretty much eschewing HTTP all together if we decide to go multi-protocol. That makes the following things irrelevant:

1. HTTP Subscriptions
2. CORS
3. HTTP Methods (OPTIONS, POST)
4. HTTP Response Body
5. HTTP Status Codes signaling success/failure

[progrium]

I feel like maybe aligning with JSON-RPC is getting a little out of hand. We did agree these are WebPipes and so it makes sense to focus on web technologies. Using other protocols is going to have very little benefit to the goals of the project.

On Sun, Dec 9, 2012 at 8:05 PM, Matthew Hudson notifications@github.comwrote:

I'm sorry, I wasn't as clear as I could have been with my database example. We aren't passing a HTTP Method as an argument, we're passing the name of the "remote procedure" we want to run. In JSON-RPC, the "method" is the remote procedure you want to invoke.

In my example, the database block exposed two methods/functions/remote procedures - 'get' and 'set'. 'Get' would return a database record, and 'set' would save a database record.

As for REST, we're pretty much eschewing HTTP all together if we decide to go multi-protocol. That makes the following things irrelevant:

1. HTTP Subscriptions
2. CORS
3. HTTP Methods (OPTIONS, POST)
4. HTTP Response Body

5. HTTP Status Codes signaling success/failure

   — Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-11181297.

Jeff Lindsay http://progrium.com

[matthewhudson]

I have to agree with Jeff. I think we ought to leverage JSON-RPC only for inspiration, not total adoption. Like Jeff said, we agreed to focus on HTTP. I think the two biggest inspirations we gain from JSON-RPC are:

1. The 'error' key in output. In addition to HTTP Status Codes, this allows Blocks to provide a precise explanation for failure.
2. The 'method' key in the input. This affords Blocks the ability to expose multiple functions which is crucial for something like a database block.

[progrium]

Why can't a database block be split into different blocks?

On Sun, Dec 9, 2012 at 8:34 PM, Matthew Hudson notifications@github.comwrote:

I have to agree with Jeff. I think we ought to leverage JSON-RPC only for inspiration, not total adoption. Like Jeff said, we agreed to focus on HTTP. I think the two biggest inspirations we gain from JSON-RPC are:

1. The 'error' key in output. In addition to HTTP Status Codes, this allows Blocks to provide a precise explanation for failure.

2. The 'method' key in the input. This affords Blocks the ability to expose multiple functions which is crucial for something like a database block.

   — Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-11181698.

Jeff Lindsay http://progrium.com

[matthewhudson]

Wouldn't that be confusing? How would a block specify a relationship with one or more other blocks?

[progrium]

I don't know what you mean. Give me an example.

On Sun, Dec 9, 2012 at 8:56 PM, Matthew Hudson notifications@github.comwrote:

Wouldn't that be confusing? How would a block specify a relationship with one or more other blocks?

— Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-11181979.

Jeff Lindsay http://progrium.com

[matthewhudson]

So, I'm proposing we adopt the "methods" key from JSON-RPC so that a single block can expose multiple functions. This new "methods" key would map method names to our inputs/outputs.

Here's an example:

{
    "name": "database",
    "description": "Store and fetch persistent data.",
    "url": "/block-endpoint",
    "methods": {
        "get": {
            "inputs": [],
            "outputs": []
        },
        "set": {
            "inputs": [],
            "outputs": []
        }
    }
}

You seem to be proposing that we don't need to allow blocks to expose multiple methods and that my example database block should instead be two separate blocks, like this:

Save Record:

{
    "name": "database-set-record",
    "description": "Save persistent data.",
    "url": "/block-endpoint",
    "inputs": [],
    "outputs": []

}

Retrieve Record:

{
    "name": "database-get-record",
    "description": "Fetch persistent data.",
    "url": "/block-endpoint",
    "inputs": [],
    "outputs": []
}

So my question to you is this: how does "database-set-record" describe its relationship with "database-get-record" and vice-versa? How does an enduser who knows about one of these block know about the other?

[progrium]

Naming convention? Meta-data? Maybe that's a registry/index problem?

There was an operating system called Genera. It was 100% LISP. Think about the complexity of an operating system. Now think about there being a single namespace for every operation. The operating system was amazing in that every function was available to the user to explore and everything in it was inspectable. It was truly an open operating system. If you found a get function and wondered if there was a set function, you'd just look it up. There were indexes and things to help you find functions. But you wouldn't find one function from another function.

In fact, think of Unix today. There are lots of system calls with related functions. It's in the man pages that you find the related functions.

Anyway, I'm getting too far from the point. I think the point is that the reason you'd want a block to serve multiple functions is understandable, but it's not necessary. The big reason why I'm against it is that every graphical programming environment has maintained a simple relationship between their equivalent of a block and a function. And I think it's for a good reason.

What do you call a block with multiple functions? An aggregate block? Wait wait, we already have that concept. So let's introduce another name that's inevitably going to be similar? This is something I'd only want to do as a last resort.

-jeff

On Sun, Dec 9, 2012 at 9:09 PM, Matthew Hudson notifications@github.comwrote:

So, I'm proposing we adopt the "methods" key from JSON-RPC so that a single block can expose multiple functions. This new "methods" key would map method names to our inputs/outputs.

Here's an example:

{ "name": "database", "description": "Store and fetch persistent data.", "url": "/block-endpoint", "methods": { "get": { "inputs": [], "outputs": [] }, "set": { "inputs": [], "outputs": [] } } }

You seem to be proposing that we don't need to allow blocks to expose multiple methods and that my example database block should instead be two separate blocks, like this:

Save Record:

{ "name": "database-set-record", "description": "Save persistent data.", "url": "/block-endpoint", "inputs": [], "outputs": []

}

Retrieve Record:

{ "name": "database-get-record", "description": "Fetch persistent data.", "url": "/block-endpoint", "inputs": [], "outputs": [] }

So my question to you is this: how does "database-set-record" describe its relationship with "database-get-record" and vice-versa? How does an enduser who knows about one of these block know about the other?

— Reply to this email directly or view it on GitHubhttps://github.com/webpipes/spec/issues/4#issuecomment-11182159.

Jeff Lindsay http://progrium.com

[tlrobinson]

I agree the JSON-RPC thing might be too complicated, but I think we should commit to something that's RPC-like or REST-like, but not halfway in between. Personally I favor REST.

If we go REST it might make sense to have the "methods" key in the OPTIONs response map to HTTP methods, so we can do GET, POST, PUT, DELETE on the same endpoint. But I don't like having arbitrary methods like "set" in there.

This is sort of why I was interested in working on Skylib concurrently with WebPipes. If we had a generic way to document REST APIs in a machine readable way then WebPipes APIs would just be a slightly constrained subset of Skylib APIs (a "flat" map of inputs in the request, either as a JSON object or maybe query string parameters, or even XML, and a flat output object, or array of objects. Or maybe they don't even need to be flat objects, but that complicates things).

[matthewhudson]

I was never suggesting implementing a new HTTP Method called "set", or mapping HTTP Methods to an OPTIONS key called 'methods'.

I was proposing we nest our input/output arguments inside something like the JSON-RPC request object member called 'method'.

Here's an example of how the Block Definition would change:

{
    "name": "database",
    "description": "Store and fetch persistent data.",
    "url": "/block-endpoint",
    "methods": {
        "get": {
            "inputs": [],
            "outputs": []
        },
        "set": {
            "inputs": [],
            "outputs": []
        }
    }
}

Jeff argued against that idea and I'm slowly coming around to his way of thinking.

[tlrobinson]

Yeah, I understood that completely and didn't mean to suggest otherwise, but including any kind of method name in either the request body (as you're suggesting) or in the URL (even as a separate endpoint for the same "resource", as Jeff is suggesting) is more "RPCful" than RESTful.

If we want to be RESTful I am proposing mapping different HTTP method names in the OPTIONs request.

If we want to be RPCful then why not just use JSON-RPC over HTTP?

But I'd rather be RESTful.

[tlrobinson]

BTW, WADL looks a lot like what I'm proposing, but it's XML :(

Here's an example: http://www.w3.org/Submission/wadl/#x3-40001.3