API

[mcollina]

What API do we want for Graft? I think something seneca-like is really nice, but built on jsChan (with no concept of request/response).

To some extent, I envision Graft as a router for jsChan, in a similar way to relationship between Express and Connect.

What do you think?

[AdrianRossouw]

yup, you're on the right track. We need some way to filter relevant messages. express uses .use/.METHOD to route requests. gulp uses .src to match the filenames (or other critera).

my recent work on waif is also informed by seneca. Specifically the parts where you separate the service definition from the mapping to hosts or message queues or the like.

the thing i'm most unsure about with seneca is patrun. the rules seem arbitrary and you are bound to end up with __propname.

I think messages need some kind of url like address, and matching can happen on properties afterwards. CDRT's could change that picture though.

The other thing about seneca was that it was is stuck in a situation where it would only work with node callbacks. Gulp and highland (and others prolly) have the situation where they allow you to use promises or generators, or callbacks, or return values, or streams, or whatever.

[ahdinosaur]

hey. :)

i like the concept of thinking about graft and jschan like express and connect.

here are some ideas on a possible graft, i hope it can be useful:

it would great if as much of graft as possible is modular, so we can swap out core modules, re-use the core modules elsewhere, add modules for middleware-like functionality, etc. so the main question for me is what functionality do we want to support in graft and what modules do to support that functionality.

i think routing should be generalized matching of arbitrary objects to the service that should handle it. then for example, HTTP semantics could be achieved with matching objects containing headers, path, method, query, body, etc, and could be built with something like a "translator" from HTTP to an HTTP-like object and a "pattern-matching module" focused on HTTP-like objects.

i think every step in graft (every source, filter, sink, etc) should be it's own jschan "service", which of course could be grouped with others on the same machine and even in the same process. then for example, if i do a GET /people/ahdinosaur, it starts at the step that translates HTTP to an object, then it goes through various "middleware" services, then it goes to the graft router (which could itself have multiple services), then it goes to the people service, then it goes to the people.get service, then it goes to the database service, then goes back up all the layers with modifications as necessary. an open question is how speed will be affected with these layers of services, but ideally speed is only a problem if too many of these services are separated by a network.

i think there should be some way to more easily describe "message patterns", which include but are not limited to request / response, publish / subscribe, push / pull, and even more complex patterns like Conversations for Action. i'm not quite sure what this would look like as an API, but i do think the value of jschan is the ability to support arbitrary message patterns (not just req/res), so would be great to be well-supported in graft as a successor to express-like req/res message frameworks.

sorry if these are more open-ended thoughts than specific implementation details, just on my mind.

much love to everyone here, thanks for your work on graft. :)

[mcollina]

I think we should all start and make some good proposal for the API and what we want it to look like. I did the API for jsChan, and I may be to wired there to design something fresh here, so help me with ideas!

Some stuff to consider:

1. forget HTTP and design fresh
2. Graft will not be request/response, but will follow the jschan way of doing things: the client sends a request, containing the response channel, plus any other streams. Ideally the base case should be simpler than jsChan (have a look).
3. we want a routing system that can be made fast, it should not be fast now, we should just be aware that we might need speed if the number of routes grows.
4. We need to decide if we want a pipeline of middlewares, in express each middleware/app can call next to not handle the request, or something more static, e.g. if a message matches a route, that route should handle it.

I'm ok with using message patterns, I also prefer something more deterministic than patrun, but... I think we should use patrun for the time being. I prefer having an API to let people try it, and we have lots of experience with patrun and seneca. However I'm not fond of patrun, so any other algorithm/data structure is ok for me. I would like to dedicate some time to have a trie for JS objects. Plus, I would like to have something like Joy (from HAPI) to validates the format for the router.

[AdrianRossouw]

I wish I had time right now to talk about this more completely, maybe this is something more for the mailing list?

Firstly, it's important that everyone read through the vision/pitch document:

Graft.js : full-stack javascript through microservices

When I imagine this api, I imagine something similar to gulp, where you have a never ending stream of messages coming in, that you can filter, fork, pause, pipe, do whatever with.

Each of the pipe line'd functions along the way, can in turn generate more messages that are sent across the pipe.

In my mind, the only difference between patrun and express, is that it's a different function that forks the stream of messages/requests.

    graft()
      .fork(patrun(), [more processing])
      .fork(router(), [more processing]);
      .on('error', doStuff);

They can both interact on the same channels, along with completely different types of request filtering. I'd love to get the input from @dominictarr about how scuttlebutt and the like could affect a system like this.

[mcollina]

Adrian, can you sketch a client like the server you just did? I'm not sure exactly how much time I'll have in the next few weeks, but I'd like to have a working prototype by the end of the month. I would like to have a small app example, featuring Graft, jsChan, some live updates, and maybe leveldb.

In my understanding, this is a microservices library, so it has not the goal to replace Express or Hapi. However, we might support remote calling through WebSockets sometime in the future (this is also in the roadmap for libchan, so... :D). Plus, we might want some adapter for HTTP, but it should not be the first goal, or should it?

[AdrianRossouw]

i've been thinking about this a lot, especially after you asked for the client @mcollina.

To design the API for this, we kind of have to have some use cases in mind to test it against. I suspect that the use cases and API design is going to be heavily influenced by how we see this extend to the browser too.

Even just on the server, we would have 2 main categories of use cases / example apps to consider. Familiar things, that are related to what you know now. and things that jschan/graft could make possible/easy that aren't possible now.

In my mind the latter mostly kicks in with the browser implementation, and I have no idea how feasible that is yet.

As for the websockets+http fallback, I am very interested in this, and it might also be something that we can really help the libchan guys with. They will have to integrate with JS if they want to talk to the client, and we can build that part, probably.

[mcollina]

To design the API for this, we kind of have to have some use cases in mind to test it against. I suspect that the use cases and API design is going to be heavily influenced by how we see this extend to the browser too.

I prefer working code, and we can iterate on it very fast (like we did for jsChan). Right now the browser is slightly out of scope as we don't have a transport for it (yet), but we'll probably have in the next few months.

Even just on the server, we would have 2 main categories of use cases / example apps to consider. Familiar things, that are related to what you know now. and things that jschan/graft could make possible/easy that aren't possible now.

Here are my goals:

1. be as good as express/hapi/seneca/* for building internal REST API (request/response)
2. support live updates of a resource (something not possible now)
3. send through files, biggish data in a stream-oriented way

In my mind the latter mostly kicks in with the browser implementation, and I have no idea how feasible that is yet.

Let's give the libchan guys some weeks to settle the SPDY part, their are working on a websocket transport anyway. We can collaborate on that and built a web (and node) JS counterpart. I'll reach them in this regard.

Finally, here is my take for the server:

graft
  .server(opts)
  .through({ ... }, function(req, done) {
    this.push(req)
    done() 
  })
  .fork({ welove: "spoons" }, function(graft) {
     graft
        .through(...)
        .handle(...)

  })
  .handle({ first: "message" }, function(req) {
    var chan   = req.chan
        , msg  = req.msg
        , ret  = msg.returnChannel

    ret.end({ ack: true })
  })
 .on('error', function(err) { ... })

and for the client:

var client = graft.client(opts)
    , returnGraft = client.request({ hello: "world" })

returnGraft
  .through(..)
  .handle(...)

What do you think?

[AdrianRossouw]

Ok. so after some discussion we know the app we want to build to uncover the API some more.

It's a collaborative whiteboard app called AetherBoard.

The thing that really opened this up for us, was the realization that jschan's in-memory transport should mean we can run it in the browser already.

We can also hack together some kind of websocket transport, even though it won't be compatible with libchan directly

We can start by finding stream-like libraries for each of the microservices I identified, and possibly mapping out more that are needed. I will write out a complete app in pseudocode for this soon, but right now I need to crash.

[AdrianRossouw]

@pelger @mcollina

Ok. I am pretty sure how to build this now, and I'm going to be following through over the next few days.

these are some notes i took while working through it : https://gist.github.com/AdrianRossouw/379818652ad6e110f148

this is my first pass at the API, just porting the examples : https://github.com/GraftJS/graft/compare/master...wip-api

I really like where this is going btw.

[AdrianRossouw]

well, for better or worse, there is an API now =)

so I'm closing this as being impossibly vague.

[AdrianRossouw]

future discussions about the api can be handled in future tickets or PR's