Switch from existing Lua parser to Emscripten-based one

[MaddieM4]

lua-parser.js sucks, and it's too much work for us to fix it on our own.

My current porting project for Hawkthorne is a nightmare thanks to the multitude of "unfinished business" in the current Lua parser. In order to get vendor libraries to work, someone would have to implement string.match, at least far enough for the following line to work:

local _PACKAGE = (...):match("^(.+)%.[^%.]+")

No. Fuck that. I'm done.

There's a better way, just waiting for us to use it.

People have compiled Lua to JS with Emscripten. This is a legit compilation of the actual C source code into Javascript. The interpreter acts the exact same way as the "real deal," because it pretty much is. There are no surprises, no unimplemented features, no parser-specific craziness to waste days debugging.

We switch our internal parsing and compilation to that, and we get a fast and accurate Lua base on which to build our kingdom.

Will this be hard?

Some parts yes, some parts no. The basic architecture should be easy to switch out, but it's the surrounding stuff (especially our customizations to the old lua-parser.js code) where we will likely run into trouble. All our existing code is built on lua-parser, and examples may temporarily break during the transition.

That said, the change is really, really worth it. A consistent, correct, and uncrippled Lua parser is essential to porting any sort of advanced game to Webplayer.

Who will do it?

Probably me. I'm the most highly motivated, all my work can happen in a toy branch over in my repo while we get it up to snuff for merging, etc. But with the satellite concerns, such as supporting other demos, it's more than a one-man job.

[ghoulsblade]

Sounds good, if you can get lua 5.2 running and provide an example for how to add custom functions i can help out getting webgl graphics to work, if that's what you want to do.

Alternatively it might even be possible to compile the whole löve code directly, hedgewars doesn't use löve but they have an emscripten port, i'm not sure about the details tho : https://code.google.com/p/hedgewars/source/browse/GettingEmscriptenToWork.wiki?repo=wiki I probably wouldn't be much help with that approach tho.

I took a look at emscripten in the beginning, but there are a few drawbacks that'd have to be overcome :

• i only found lua 5.1 rather than 5.2 so far
• i had some concerns regarding performance of compiling c memory management to javascript arrays, but i'm more than happy if you can prove me wrong there =)
• initial size of the generated emscripten .js file is over 1mb compared to a few kb for luaparser.js
• afaik lua code will have to be translated using commandline tools rather than on-the-fly like now, not a killer but something to be aware of.
• initial setup and finding an easily adaptable example for custom lua functions, this should be easily solvable with some effort.

[MaddieM4]

Cool! I'm kinda blocked for the moment, waiting for someone in the emscripten bug tracker to help guide me through making native JS functions available to the Lua environment. If I can at all restrict the translation overhead to just the interpreter, that's the path I'm gonna head down, rather than try to compile Löve into it.

To go through the drawbacks, and how big a problem each one is:

• For anyone willing to invest the effort in setting up an emscripten environment (again, probably me) the actual "building Lua 5.2" part is pretty trivial. It's just the setup that's onerous. So this is very doable, even if we don't have a prebaked 5.2 interpreter.
• I have no idea which will end up being faster. I don't think we'll know until we can do some practical, real-world tests.
• Part of that is translation overhead, part of that is the fact that luaparser.js is laughably far from complete, missing (or misimplementing) a lot of features. With the right compression and caching setup on a server, 1MB isn't bad at all, unless you're on a crappy network, in which case the game download itself will be a long download anyways.
• Nope! Nothing more than the interpreter anyways. Remember, it's the interpreter. It's designed to parse and execute arbitrary Lua code. All your code can be shoveled into it untranslated, provided you've overwritten the libraries in the interpreter environment first :)
• This is basically where the work is, yeah. Like I said, waiting for a response in another issue tracker.

[MaddieM4]

To clarify where I'm at right now, I gave up on trying to wrangle the JSRepl version of Lua JS compilation, and started working on a better-documented alternative that will do the things we need, with the version of Lua we want, without the bloat or the mystery meat.

Still not at the point yet where it's working properly, although I've resolved a lot of the problems. I kinda burned out on it, so it'll only progress as I revisit it, but hopefully it's in a state where anyone else can perfectly reproduce my progress just by cloning the repo and running "make," which was the biggest problem with JSRepl (if it was easy to reproduce their success based on the code they've released, I'd be done by now).

[MaddieM4]

I've made sufficient progress with weblua that I'm ready to try it out as a replacement to our current Lua parser. It effortlessly supports function wrapping from either side, injecting entire modules of JS code at once, etc. It was designed to be the perfect tool for this job, and by coincidence, many others.

I'd love your opinion on multiple return semantics, and will probably need a bit of your help to make sure that the filesystem works. Everything else, SDL and whatever, I think I can figure out on my own over time.

[ghoulsblade]

heya, cool, i'm happy to see you haven't given up =) i don't understand what you mean by the multi-return semantics thread, probably because i don't know how binding native js functions for lua-code works in weblua. As far as i understand there is a problem with the way weblua does that, is it just esthetics or does it actually cause errors ? Maybe an example would clarifying the problem.

an example for multiple return with lua.js in my current code : love-webplayer / js / love.mouse.js : t.str['getPosition'] = function () { return [gMouseX, gMouseY]; } How would that look with weblua ?

i don't think adjusting the few places where the love api uses multiple return is a problem, i can help you with that =)

love-webplayer support for love.filesystem is experimental and not well tested, we use the localstorage if supported by the browser, if needed here is the spec for the "localStorage" attribute/globalvar : http://www.w3.org/TR/webstorage/ If i can help with that just tell me how =)

can you provide a working example for weblua with lua 5.2 with a custom js function being exposed to lua ?

[MaddieM4]

Thanks! It took freaking forever, but once I got it to compile at all, everything else just kinda went fast.

Multi-return stuff is a totally aesthetic problem. It's easy to return multiple values from Javascript:

function whatever () { return Lua.MultiReturn([mouse.x, mouse.y])}

Because there's a semantic difference between returning a table, and returning multiple arguments. The other option I choose against was returning everything as an array, because of programmer burden.

The real question is how Lua functions should return to the Javascript environment. Always a table? A MultiReturn object when args.length > 1? I'm currently in favor of the latter, but wanted an outside opinion.

[MaddieM4]

Oh, forgot to give complete examples.

JS function exposed in Lua

function my_func(a) {
    return a+3;
}
Lua.inject(my_func, 'exposed_func_name')
Lua.eval("exposed_func_name(2)") // Returns 5
Lua.eval("exposed_func_name") // Returns function
Lua.eval("exposed_func_name")(3) // Returns 6

Lua function exposed to JS

We actually show this off in the above example, creating a JS function, injecting it into the Lua globals table as a cfunction, then pulling it back out to JS as a wrapper to a Lua function. Not that you want to be layering this kind of Inception stuff everywhere in practice, but it's fun that you can.

Lua.exec("
function lua_func(a) {
    return a + 3
end
");
var func_from_lua = Lua.eval("lua_func");
func_from_lua(-1) // Returns 2

A whole module written in JS

var mymodule = {
    'dothing' : function () { return 'done'; },
    'donothing' : function () { return 'nothing done'; },
    'barf' : function (thingtobarf) { console.log(thingtobarf); }
}
Lua.inject(mymodule, 'mymodule');

Lua.eval("mymodule.dothing()") // Returns "done"
Lua.eval("mymodule.donothing()") // Returns "nothing done"
Lua.eval("mymodule.barf( {'a','b','c'} )") // Logs object ["a", "b", "c"] to the JS console

[ghoulsblade]

multi-return: i personally like the lua.js way of always returning a js array, even for single return values, makes it more consistent, and [] is unobstrusive in the sourcecode. e.g. a function returning one int would do return [42];
It's not returning a table, that's just js code, from the lua point of view it is returning the value 42 directly. Code has to be lua-aware to use most lua functions anyway, so i don't see the point of trying to make the return values of lua functions more "natural" to javascript. Problems could arise if you make a lua function that usually returns one value, but can return more than one in some cases. What would be needed when calling such a lua function from javascript code ? With lua.js always returning an array, i know what to expect and don't have to think about it twice. With "naturalized" single return values, i'd have to make an ugly if type==multiret else ... construct. That's just personal preference tho, the examples you listed work too =)

When changing mechanics like that please make sure that nil values are handled correctly, e.g. myfun ... return [3,nil,5] in js, should work for local a,b,c = myfun() assert(a == 3 and b == nil and c == 5) in lua.

examples: look good, but the build/weblua.js in your repos is a symlink pointing to weblua-0.1.0.js , which is not in the repos, forgot to add file maybe ? symlink should be relative path btw ;) Would be cool if you can provide a pre-built .js file, having a "binary" available is always better than having to compile, which always carries a risk of something not working.

Do you know if stuff like debug-callstacks, protected calls and coroutines work ? Doing a protected call from javascript to execute the main.lua and handle errors would be cool, i'd appreciate some example code for that if you know how to do that =)

[MaddieM4]

I think you're probably right. The conditional type=="MultiReturn" stuff is just gonna be obnoxious, better to return arrays instead.

When changing mechanics like that please make sure that nil values are handled correctly, e.g. myfun ... return [3,nil,5] in js, should work for local a,b,c = myfun() assert(a == 3 and b == nil and c == 5) in lua.

I'm not totally sure what you mean by that. When injecting JS functions into Lua, or wrapping Lua functions for JS, there is no source code conversion involved. Injecting JS into Lua = using Emscripten helpers to store the JS function as a C function, and then using the appropriate Lua library calls to make that external C function available as a global. Wrapping Lua for JS = copying the function to a Lua global like "_weblua_tmp_3", and constructing a closure that will call Lua global function _weblua_tmp_3 and handle argument passing/returns.

It's fancy as all hell, but there's no source parsing. So I can pass in JS functions that are "native code" (like Array.join), and I can wrap out native Lua builtins, and never have to worry about converting conventions or any other such silliness.

examples: look good, but the build/weblua.js in your repos is a symlink pointing to weblua-0.1.0.js , which is not in the repos, forgot to add file maybe ? symlink should be relative path btw ;)

Heh, you're right, forgot to add that. The symlink is autogenerated by the makefile as an absolute path, but I can fix that as well - should never be a problem for someone building weblua from scratch, but a bit odd for anyone using the prebuilt weblua-0.1.0.js! Once I commit those things, you should be able to clone the repo and play around with example.html.

Do you know if stuff like debug-callstacks, protected calls and coroutines work ? Doing a protected call from javascript to execute the main.lua and handle errors would be cool, i'd appreciate some example code for that if you know how to do that =)

All calls are protected. The traceback behavior is still not very useful, though - kind of a bare minimum of functionality.

[MaddieM4]

The following is now pushed to master:

• Fixed symlink
• Actual precompiled weblua-0.1.0.js
• Multireturn fixes (all arrays, all the time).