An idea for an interesting nyanga sample

[romix]

Hi Richard,

I have the following idea for a relatively complex example that would show nyanga's power. This example should implement a small workflow engine.

It means:

• It should understand a custom syntax of this workflow engine, i.e. create a small DSL using Nyanga's support for defining and using grammars
• Workflows can be triggered via APIs or via an HTTP request
• Workflows may read/write variables, invoke a small set of predefined actions or invoke external services (e.g. via HTTP/REST)
• It should be possible to execute a huge number of workflows concurrently
• Each workflow's execution is implemented based on coroutines, i.e. it is very easy to suspend/resume the execution of a workflow at any stage.
• If a workflow is doing an HTTP request, it should be non-blocking, i.e. the workflow's coroutine is basically suspended and later resumed, when HTTP response is received.

I think this set of features is pretty easy to implement and it covers a lot of Nyanga's features:

• grammars
• coroutines
• async & co
• non-blocking, async IO which looks like normal sequential code thanks to coroutines and your standard library
• pattern matching could be used for parsing the DSL and performing actions.

I think such an example would be really impressive compared to small "Hello World!" like examples and would show a real power of Nyanga.

What do you think?

[richardhundt]

On 2/14/14 8:09 PM, romix wrote:

Hi Richard,

I have the following idea for a relatively complex example that would show nyanga's power. This example should implement a small workflow engine.

It means:

• It should understand a custom syntax of this workflow engine, i.e. create a small DSL using Nyanga's support for defining and using grammars
• Workflows can be triggered via APIs or via an HTTP request
• Workflows may read/write variables, invoke a small set of predefined actions or invoke external services (e.g. via HTTP/REST)
• It should be possible to execute a huge number of workflows concurrently
• Each workflow's execution is implemented based on coroutines, i.e. it is very easy to suspend/resume the execution of a workflow at any stage.
• If a workflow is doing an HTTP request, it should be non-blocking, i.e. the workflow's coroutine is basically suspended and later resumed, when HTTP response is received.

I think this set of features is pretty easy to implement and it covers a lot of Nyanga's features:

• grammars
• coroutines
• async & co
• non-blocking, async IO which looks like normal sequential code thanks to coroutines and your standard library
• pattern matching could be used for parsing the DSL and performing actions.

I think such an example would be really impressive compared to small "Hello World!" like examples and would show a real power of Nyanga.

What do you think?

I think it's an awesome idea!

I have a project for it as well, and I'll need plenty of network code as well as both HTTP client and server code. For this we'd also need a generic stream api which plays nicely with the scheduler. It should implement 3 semaphores for readable, writable and errors (basically for POLLIN, POLLOUT and POLLERR), as well as read, write, flush and close.

Once you have streams, you can make them composable. Either by using layers:

tcp = TCPStream()
http = HTTPServer(tcp)

http_request = http.read()

Or by using filters and channels to connect them:

tcp = TCPStream()
http = HTTPFilter()

chan = Channel()

tcp.output(chan)
http.input(chan)

-- or perhaps just
http.input(tcp.output)

http_request = http.output.read() -- this pumps

All of this will just magically schedule fibers cooperatively of course :)

On a related note: I've decided to bite the bullet and use libuv for a system core and scheduler. It's a bit of work because I need to wrap it in a queue (essentially a ring buffer of events triggered by libuv callbacks) so that it has a level-triggered pull-style interface much like epoll. I started experimenting with this here [1], but that's got a couple of serious design flaws, so it's getting a rewrite. Eventually I'll also add tasks which will be coroutines which don't shared managed state which can be run on different threads (so M:N threading).

So this is all just something for you to think about while working on your project. I'm happy to collaborate generic pieces like the stream api to add to the std library as needed. You're also welcome to build it and we can merge it :)

[1] https://github.com/richardhundt/libray

Reply to this email directly or view it on GitHub https://github.com/richardhundt/nyanga/issues/33.

[romix]

Cool!

• I started with a grammar and had a lot of fun yesterday. Here is my initial feedback.
• I never used LPEG/RE before, but had a lot of experience with more traditional parser generators like Yacc, Bison, ANTLR, etc. Defining the grammar was easy, but it took me a while to understand how semantic actions are defined and executed in the LPEG/RE model, where they get their parameters from and how ASTs are constructed. This part was a bit unusual compared to other tools I used before. Initially, I missed the ability to refer to the attributes associated with a given named or positional element of the rule (or is it possible? Is it good to have such an opportunity?). It was also a bit unusual, how captures are collected by means of e.g. table captures and so on. Very powerful, but unusual if you use it for a first time. Anyway, I figured out the LPEG/RE way and I was able to produce proper ASTs using my grammar.
• Speaking of grammars: How can I catch parsing errors in Nyanga? In RE, one writes a rule something like %1 => error, but it does not work for me in Nyanga.
• Another thing I was missing is pretty-printing for instances of classes. I even defined tostring methods, but they seem to be called only when you do it explicitly. If I just do: print(myobj) it does not work for me.
• Also, when I created classes for my AST (though it was probably an overkill, one could go with nested tables), I had to type a bit too much, like in Java ;-) May be something like case-classes in Scala or ADT (algebraic data types) constructors in ML would be a nice to have feature? I.e. something like:

class Expr

case class Binop(op, lhs, rhs) extends Expr
case class Unary(op, rhs) extends Expr
case class Var(name) extends Expr

or ML-style

class Expr = 
  Binop(op, lhs, rhs) 
| Unary(op, rhs)
| Var(name) 

The syntax is just an example, of course.

• Further thing that I'd like to have, but it is probably too special: I'd like to have something like automated treewalker or tree-rewriter generation. The idea is to be able to say something like: Find all occurrences of a given pattern anywhere in this tree/graph and perform an action, e.g. compute something or replace this occurrence with something else. It would be nice to be able to say that this tree traversal should be done top-down, bottom-up, etc. Such a feature is useful if you implement e.g. something like algebraic simplifications, term rewriting, etc. What do you think? Even if it is not planned, may be you can give me a hit if something like this exists for Lua already?
• I think I have found a small bug:

class C
    tostring()
         print ("C")
    end
end

c = C().tostring()
print(c)
print(C().tostring())

I would assume that both last prints should output the same value. But this is not the case. The first one prints C, the second one results in

nil
Error: workflowengine.nga:11: attempt to index a nil value

More over, the second one sometimes produces segmentation faults....

[romix]

BTW, one more question, a bit off-topic: Imagine that I want to define a grammar for C/Java-like expressions using Nyanga's grammars. Different arithmetic operations have different priorities (and sometimes different associativity). Of course, one can model this by introducing a new non-terminal per priority level, as it is usually done by means of Yacc/Bison. But may be there's a shorter way when using LPEG grammars, e.g. the priority/precedence of operations is defined in a table and parser dynamically consults it to check what to do next or something like this? I.e. for binary operations one would have only one rule binop <- <exp> <op> (some kind of dynamic predecence check here) <exp> (or may be this check should be here?)

This link seems to describe something that I have in mind: http://angg.twu.net/dednat5/gab-lpeg.lua.html

But I'm wondering if this can be expressed using RE/Nyanga grammar syntax instead of LPEG's low-level syntax.