Please don't give up on this awesome project!

[antonh]

If someone knows Richard personally, please tell him not to give up on this project. He has such a good head start with his unique coroutine-based approach vs everybody's else callbacks with libuv.

Thank you

[dvv]

+1 +1 +1

[richardhundt]

Guys, that's really encouraging thank you.

I'm not giving up. Not by a long shot. I've this programming language on the go [1] and I want concurrency primitives and a systems library for it, so that's my motivation. The project has a new name though - it's called Ray and currently lives mainly on my laptop, however there is also a branch on github which I push to occasionally when it's worth a snapshot.

Right now I'm reading Joe Armstrong's (he's the main guy behind Erlang) 295 page PHD thesis on Actors and concurrency. It is a difficult thing to do properly, and I'm aiming for a badass concurrency system which is equally easy to add new C bindings for, as well as extending from Lua.

The tricky part of this whole thing is the similarities between process calculi like Tony Hoare's CSP and Actors. On the one hand I don't really want to expose the low level actor stuff to Lua because I think being able to do

local data = socket:read()

is much nicer than

socket:send('read') local data = self:recv()

which means that from Lua, the libuv bindings behave more like CSP channels wrapped around an event watcher. However I'm not so happy with CSP all the way down either, because channels need mutexes in a multi-threaded environment whereas actors have the potential for lock-free concurrency.

Then there's the additional complication of error propagation. Erlang's approach is to create process trees which exit if any one process fails, and then a supervisor can be notified and restarts the processes (optionally).

Finally, I want to get ØMQ out of the core and into an extension because I've got much faster in process message queues which can do around 40,000,000 operations / second on my laptop, so the ØMQ extension can be used when you want to push it over the process boundary.

So I'm going to get back to hacking on Ray now :)

Cheers, Richard

[1] https://github.com/richardhundt/lupa

On Nov 4, 2012, at 5:46 PM, antonh wrote:

If someone knows Richard personally, please tell him not to give up on this project. He has such a good head start with his unique coroutine-based approach vs everybody's else callbacks with libuv.

Thank you

— Reply to this email directly or view it on GitHub.

[antonh]

Thank you! I'm glad and relieved that you're still working on the project. To the best of my knowledge it's the most promising I've seen in the last couple of years. Please keep pushing to the Ray branch as often as possible. It's a joy to see it growing since I've added the project's page to my Morning Coffee.

P.S. Morning Coffee is a Firefox add-on ;)

[miketang84]

Support!

On Mon, Nov 5, 2012 at 3:49 AM, antonh notifications@github.com wrote:

Thank you! I'm glad and relieved that you're still working on the project. To the best of my knowledge it's the most promising I've seen in the last couple of years. Please keep pushing to the Ray branch as often as possible. It's a joy to see it growing since I've added the project's page to my Morning Coffee*.

• Morning Coffee is a Firefox add-on ;)

  — Reply to this email directly or view it on GitHubhttps://github.com/richardhundt/luv/issues/24#issuecomment-10054565.

Nothing is impossible.

[hnakamur]

Hi. I am also creating coroutine based libuv wrapper for lua/luajit. https://github.com/hnakamur/couv My current approach is doing yield/resume in C. Another approach I am interested in is queueing the results of libuv callbacks and pulling those from lua. And I am also interested in what approach you will take in ray.

[richardhundt]

Hi,

I noticed Couv a few weeks ago and saw the ANN. Looks interesting.

Regarding Ray: I've finally given up trying to do anything actor-like at the C level in favor of just using CSP-style channels (so basically implementing your idea of queuing callback results and pulling those from Lua).

This seems to be the simplest and most flexible approach. Actor models can be built on top of CSP primitives (or vice versa, but this way gives one a nice API).

If you want some ideas to play with for the queues, then I've found that you can use a lua_State* as a ring-buffer, so it's pretty efficient and easy to move values between states. However, making queues thread-safe with low contention is going to be trickier. I may need to move to a linked-list with two mutexes (one for read and one for write) and restrict the values to only scalars. We'll see how it goes.

At the moment Ray is hopelessly over-engineered, though, since I was doing lots of experiments and the thing just grew, so it's probably not in a great state for groking the code - let alone borrow ideas from ;-)

Anyway, thanks for the heads-up. I'll keep an eye on Couv to see how you're progressing.

Cheers, Richard

On 11/11/12 6:32 PM, Hiroaki Nakamura wrote:

Hi. I am also creating coroutine based libuv wrapper for lua/luajit. https://github.com/hnakamur/couv My current approach is doing yield/resume in C. Another approach I am interested in is queueing the results of libuv callbacks and pulling those from lua. And I am also interested in what approach you will take in ray.

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/luv/issues/24#issuecomment-10269643.

[hnakamur]

Hi, thanks for your kind comment! Actually, couv is already using queues for stream/udp reads. It has one queue per each handle. I was thinking I may change that couv to have one queue per lua_State and having all libuv callback results in it.

Maybe we can work together in the future, if you remove dependencies on LGPL/GPL libraries. I would like a BSD/MIT style licensed libuv wrapper for lua/luajit.

Please keep up the good work. I will keep trying my experiment too. Cheers, Hiroaki

[richardhundt]

I was thinking I may change that couv to have one queue per lua_State and having all libuv callback results in it.

I tried this and found that sometimes you want a queue for data, and sometimes you want a queue for states. For example when a timer ticks, you may wish to wake up multiple coroutines, so the semantics are more like those of condition vars (which are conceptually just queues of blocked threads). For I/O, of course, it makes more sense to queue the data.

Another issue with using lua_State as a queue is that the LUAI_MAXCSTACK is set at 8000 by default. It might not be a problem in practice, especially if you're okay with unbuffered channels (as in Newsqueak, or Limbo). For buffered channels I'd be more comfortable if I could put 1M items in it, though, so I'm playing with channels which only hold scalars and are allocated outside the Lua heap. This has the added bonus of saving interning strings if there are multiple C components connected in a pipeline which parse input from a stream. However, passing tables and closures requires serialization, so that gets more expensive, but at least it's "shared nothing", which is a good thing, I'm told ;)

Maybe we can work together in the future, if you remove dependencies on LGPL/GPL libraries. I would like a BSD/MIT style licensed libuv wrapper for lua/luajit.

I'm removing the dependency on ØMQ (so the only dependency will be libuv).

[hnakamur]

Thanks again for your great input! It is very valuable to listen to your experience.

As for timers, I am using callbacks (not using queues) to minimize delays because timing is important. I thought it is OK to use callbacks in some places.

I also implemented "sleep" function using a timer. It is a one shot timer and a coroutine yields on start and resumes on timeout. I found It is very handy when I use "sleep". However for repeating timers (not one shot), I couldn't think of an API better than callbacks.

Now I look at your timer API, I think it is easy to use! I think the fiber is the key here, right? I will study your implementation of fiber and understand it.

I am okay with unbuffered channels between lua_State's. I think the start point would be the implementation presented in "Programming in Lua, 2nd Ed." Chapter 30. It uses pthread and a doubly linked list for a channel, outside the Lua heap. Since we are using libuv, I think we can use uv_thread instead of pthread and ngx_queue_t for a queue.

I am glad to hear you are removing the dependency on ØMQ. Maybe I can contribute to ray when you are done.

[hnakamur]

Ah, you are already done removing the dependency on ØMQ. Now I will study ray codebase. Thank you for your great work!

[antonh]

Hi Richard, I would be grateful if you could update more examples to work with the Ray branch. Thanks and keep up the good work.

[richardhundt]

Well, the only things which are compiling right now are the fiber core and timer (not even sure if what's up on github is even doing that tbh), so it's a little premature.

Listen, I know the feeling of waiting on a project to progress (I've also watched projects which I'd built up dependencies on just die never to be resurrected), however - and this is personal - I'm 2 weeks into a new job which is taking some power out of me; makes it hard to come home after a full day's work and still be productive. Add a wife and 5-year old boy to the mix, and time must be found where available.

So, basically, it'll be ready when it's ready. Sorry, but that's how it is. Hang in there. It'll get done.

On 11/16/12 7:05 PM, antonh wrote:

Hi Richard, I would be grateful if you could update more examples to work with the Ray branch. Thanks and keep up the good work.

— Reply to this email directly or view it on GitHub https://github.com/richardhundt/luv/issues/24#issuecomment-10456039.

[antonh]

Richard, sorry if I seemed to be insisting. There's absolutely no hurry.

[antonh]

Thanks a lot for updating the examples ;)

Just had to remove -luv from the LDFLAGS line in Makefile:

LDFLAGS=-luv -lm -ldl -lpthread

to avoid the following error when running make (Mac OS X 10.6.8):

ld: library not found for -luv

Ray is impressively fast! Keep up the good work!

[dvv]

it also requires adding -fPIC to LDFLAGS -- i686, latest archlinux 64-bit

[dvv]

this works for me: LDFLAGS=-L./uv -luv -lm -ldl -lpthread

[dvv]

i beleive linking libuv into all ray modules bloats things

[reliquaryhouse]

I have to echo what everyone else is saying, it was a breath of fresh air to find this project as I think Lua is the prime platform for a web development stack (especially with LuaJIT behind it). It was very frustrating seeing people doing a straight port of libuv and then implementing the node.js callback architecture in a language that doesn't need callback hell. If we could get a Rack-esque server deployment interface and a simple Sinatra/Flask-esque microframework going based off Ray then I think it would awaken the sleeping giant that is Lua and really attract people to the platform, as I (and many others) feel that Lua has heavily stagnated and is now trapped in the same position that Python 3 is regarding going from 5.1 to 5.2, except we have even less people caring about updating their modules to work with it.

I appreciate the effort that the Kepler Project guys put into WSAPI and Xavante but they were written for a different time and a different web, and they cannot match the web stacks provided by other languages. We don't need FastCGI deployment anymore, we just need a fast httpd the app can run itself and reverse proxy to nginx.

I have to give credit to John Abrahamsen for having the same general idea and making the effort to do something similar to what I said above with https://github.com/JohnAbrahamsen/nonsence-ng, I think we need to centralize and work together and things can really take off for Lua web development and Ray.

I'm really impressed with the work you've done so far, Richard. I hope you can find the time to keep it going.

[jvburnes]

Richard, dvv and friends...

I'm about to run my first tests on my OS uber-hack of luv, libuv and inferno called 'node9'. Instead of Limbo it uses the luv/luajit scheduler for the basis of it's process model and shadows these within the inferno process model. Non-blocking syscalls are made from luv and routed through libuv worker queues to resolve the inferno/plan9 service requests. The kernel lives on the primary libuv event loop and portably handles i/o, network and timer requests.

I had to massively refactor the inferno kernel code because the old dis virtual machine model executing compiled Limbo is radically different from most other systems including lua/luv.

I'm testing the basic system calls now and with a little luck I'll have a highly dynamic, portable, hosted, distributed OS that integrates well with web apps and nodejs soon. (thats a mouthful)

Let me know if you're interested. I don't know if anyone's written a Lua OS before.

jim burnes

ps: I had to really tweak the cmake config to build on OSX 10.9. Don't know why, but it seems to miss the build configuration even though it detects darwin. I had to update the final link.txt file, removing '-lrt', adding '-framework CoreServices', adding '-lluajit-5.1', change .dylib extension to .so and adding '-bundle -flat_namespace' so it generates a loadable lua .so. my guess is that these are somehow related to clang being used by default or something I screwed up while tweaking it.

[ludamad]

+1. I'd like very much if the Lua community could get behind a single project like this, and I don't think Luvit is it.