D needs an async I/O web core akin to Rack or Express

[truedat101]

Originally Python came up with a simple set of principles for web server interfaces (WSGI), Perl followed, with PLACK : http://plackperl.org/ . Ruby took it to a new level with RACK. And web frameworks florished. And it was good. Then Node.js came along with a nice async-i/o web core (http parser, request/response libraries, and http listener), upon which all modules are built. Express formalized the design of such servers, and allowed for users to create their own middleware stuff (concepts borrowed from WSGI). Frameworks took a huge leap forward.

D has vibe-d. It's awesome. But it's big, too big for small devices. @bithavoc started something interesting with heaploop.io and showed what was possible taking a feather from Node.js, using libuv and http parser. I borrowed those libraries (unsupported) and created a frameworkish hack called ultraviolet.d . I need to formalize this into a library worth releasing. From something like that, one should be able to build new web frameworks or just roll-your-own web server. Everything should of course sit behind a secure front end, like NGINX or apache2. No one is suggesting yet to write a full webserver.

[truedat101]

One thought on this deisgn. To use libuv (and maintain this going forward as uv.d ), or use fibers? Pros and cons to both. It would be interesting if the async design was a configurable aspect of the design. One could just implement the HTTP engine interface, like WSGI's architecture, and we'd provide a fiber engine and a uv.d engine. Then, you can actually start to explore performance and make the case for where to invest, or leave that as an exercise for the community to add plugin modules.

[bithavoc]

heaploop failed because I couldn't find motivation to continue working in a copy of Node.js that doesn't really take advantage of neither the power of D and nor javascript, serious D developers would think it's trash and would not attract other noobies because it had no javascript.

Also, forcing developers to use a single thread across the whole D and suggest trickery to use other threads doesn't feel natural. Esentially forcing developers to use a single thread with the justification of taking advantage of evented IO it's a design mistake, it was an easy call from the Node.js perspective because javascript is single threaded; but D, D is too powerful for such oversights. Among other reasons, this is why I don't like vibe.d.

We can now customize the Scheduler via std.concurrency and whatever solution we come up with has to work for hard 1:1 threads(the default) and fiber scheduler, actually, it should work with any scheduler and the IO layer shouldn't care about it.

So the solution should not come from implementing an specialized Scheduler, it should be an IO layer that you'd be able to use from any thread. The most natural solution to me would be to have an actor to handle the iO loop and communicate with it solely by message passing.

I also vote for not having a wrapper around uv.d this time, uv.d is favoring cross-platform maintenance over perfomance these days by pushing even more and more tasks to the thread pool instead dealing with async API on each OS. We could do that easily from D, we don't need uv for that; we can do better than this.

Since message passing would force us to use inmmutable structures, I don't see why we couldn't serialize them and receive loop control message from the network too effectively making all IO resources of this library remotable over the wire a-la Erlang.

[truedat101]

Interesting points, all valid. I'll think on this. I'm open to different approaches.