Request: Avoid LuaJIT FFI

[elliottslaughter]

I have a request for the standard library that I'm creating an issue for so that it's easier to remember.

Please avoid use of LuaJIT-specific features, particularly the FFI. Wherever possible, use features native to Terra, and if Terra can't do something, let's try to figure out if Terra can be extended so that we can have what we want.

Rationale: For various reasons, it may be necessary to move from LuaJIT to PUC Lua (i.e. the original Lua implementation). The most urgent reason is compatibility: LuaJIT supports a relatively small set of OS and architecture combinations. Notably, PPC64 is missing from this list---and if you want to use the current number 1 supercomputer in the world, it runs on PPC64. In general PUC Lua is much more portable so it less likely to run into issues in the future.

There are some promising developments on this front in https://github.com/raptorjit/raptorjit/pull/199, but who knows how long this will take to mature and/or whether it ever will.

So for the moment, I'd prefer to develop in a way that won't preclude a switch to PUC Lua if we decide we need to go that route.

[Qix-]

Is there any reason to keep LuaJIT around at this point? I thought it was just there since it was, at one point, intended to be used as a sort of runtime for applications that wanted it - but I understand that's not the case anymore (or is it?).

Further, at a high level, what would have to be replaced in order to get Terra working with PUC Lua?

[aiverson]

I like keeping LuaJIT around for the performance. In my mind, Terra is about performance, and compiletime performance is an important part of having a pleasant developer experience for AoT code and for mixed terra/lua code like in ebb or other projects the performance of the Lua engine can be important. I agree with avoiding using LuaJIT specific features as much as possible, or at least wrapping them in guards and providing alternatives.

[Qix-]

Just out of curiosity, does this mean losing the ability to JIT terra functions at compile time and run them?

I remember using this hack once to great effect:

local C = terra.includec('unistd.h')

local compilationCWD = C.getcwd()
print(compilationCWD) -- or however you convert `const char *` to a lua string

[elliottslaughter]

The main advantages I see to LuaJIT are:

• Compile-time performance (at least in theory, in practice we'd have to measure it)
• FFI
• Stability (PUC Lua insists on making breaking changes with every release; while these are usually small they are papercuts that you have to deal with if you write code that tries to be compatible across versions)

Of the three of these, by far the most problematic when porting to PUC Lua is the FFI, because it's really pretty pervasive in the way Terra is implemented right now.

@zdevito did the initial work to port to PUC Lua, which I spruced up and got to a kind-of working state here: https://github.com/zdevito/terra/pull/320

The way I did it was to fork one of the LuaJIT-FFI-for-Lua libraries (specifically this one). Because the original library was x86-specific and the entire point of this was to be cross-platform, I ended up ripping out all the architecture-specific code (the original included dynamically generated assembly for calling FFI functions) which simplified things quite a bit. But the tradeoff is that we do not have a way to call functions.

So @Qix- is right that at least in this implementation we can't call JITed functions. But this isn't necessarily a fundamental limitation it just means we have to be a little creative to get that functionality back. One option would be to JIT Terra code itself to call the Lua API. Since Terra gets generated with LLVM, it should work with all the C calling conventions, and then we don't need to worry about generating platform-specific assembly code. But I haven't personally put any effort into this because the PR is already good enough for what Regent needs at the moment.

[lukego]

There are some promising developments on this front in raptorjit/raptorjit#199, but who knows how long this will take to mature and/or whether it ever will.

Just chiming in to say that if anybody wants to help finish this sooner you'd be most welcome to :).

[aiverson]

@lukego How would I go about helping with that? And what are anyone here's thoughts about writing a port of Lua in terra? LuaJIT's Dynasm seems like it could be replaced with terra macros pretty effectively, and I think there is already an existing implementation of that which was mentioned in the terra papers. Terra's metaprogramming capabilities would probably make maintaining it going forward easier than the C implementation, but it also seems like a lot of extra effort to create it in the first place with no obvious payoff without a larger community of users.

[lukego]

@aiverson RaptorJIT has a branch where the VM is being ported from assembler to C. This is about 20% done at the moment in terms of code -- though the first parts were especially hard due to learning curve. Just now progress is limited by my own ability to make time to continue the work of adding bytecodes on this branch (while also working on a bunch of other things.) If somebody else wants to implement a few bytecodes and send a pull request that would move things forward more quickly.

This is only relevant to Terra in the sense that @elliottslaughter mentioned i.e. as a runtime environment for the Terra compiler. The new RaptorJIT VM is being written in plain-old low-brow no-fuss C code: just a straight port of the LuaJIT assembler code.