Build failing

[patrick-scho]

I discovered this a few days ago and wanted to build it and test it out, but running "make" wouldn't work because https://github.com/fperrad/tvmjit does not exist anymore. I finally managed to get it working, but I wonder if you could update your Makefile so that it works again.

[richardhundt]

Hey, what was the fix? I can only find one fork of it on github and it's 5 years old.

[patrick-scho]

I just assumed the fork you had was going to work and it did. I cloned the shine respository, ran make, which failed when trying to get tvmjit. Then I manually cloned https://github.com/richardhundt/tvmjit.git https://github.com/richardhundt/uthread.git https://github.com/richardhundt/upoll.git and then I had to "make bootstrap" before "make && make install" because there is a version of shinec.raw in boot/src which was incompatible with the version of tvmjit

Got it working now and I gotta say it's a pleasure to use, I've been looking for a nice version of Lua for some time this is exactly what i was looking for

[rversteegen]

Yikes, github's search function is totally broken. It doesn't find clones of repos when the original is deleted. I tried using Google instead and the best I could find was https://github.com/caizongchao/tvmjit which is two years old. I imagine little has happened in the last two years. But if you or anyone else has a more recent copy of tvmjit on their machine, could you please put it on github? (Maybe create a new repo rather than update https://github.com/richardhundt/tvmjit.git, so github forgets it's a fork and doesn't censor it.)

[richardhundt]

I'm using my fork of tvmjit. This really needs a rewrite to target LuaJIT 2 vanilla at some point I guess.

[rversteegen]

How much work would that be? Is it just src/lang/translator.lua that would need to be rewritten? Quickly looking about, I couldn't find anywhere else where you make use of tvmjit features.

Based on an even briefer look, I think it might be much more practical to instead port tvmjit (or almost all of it, minus the C api which seems to have lua equivalents) to vanilla luajit as a library for generating luajit bytecode (with a dependency on a UTF8 library).

(I do think extending luajit2 to be an easier target for languages other than lua 5.2 is a great idea. I imagine it will be very difficult to get anyone to agree to add anything to luajit though, even with Mike Pall retired...)

[richardhundt]

The key issue is the debug segment in the bytecode format. This is used for error reporting and tracebacks. TvmJIT supports line number hints to map opcodes to line numbers, and Shine uses these. However, it is possible to produce Lua source with controlled insertion of newline chars. With goto and labels being supported, that should be everything needed (when I started, vanilla Lua didn't support arbitrary jumps).

[richardhundt]

Another thing I was experimenting with was source maps:

https://github.com/richardhundt/blaze/blob/master/blaze/lang/emitter.lua

That emitter produces plain Lua and an embedded source map. The runtime traps exceptions and rewrites the tracebacks using the sourcemap.

[rversteegen]

That's very interesting, thanks for the link; I'm also working on compiling a language down to lua (or shine!) (and in future, possibly luajit bytecode), but I wanted to try to preserve not just line numbers but also columns if possible. I was doing this in C, but might try to make use of your code instead.

This is pretty offtopic, but what is blaze and how it is related to shine?

I realise I made a huge oversight: TvmJIT's parser is written in C; I thought it was in lua. So "porting" it to luajit makes a lot less sense. Instead, a luajit bytecode generation library could be used, such as https://github.com/franko/luajit-lang-toolkit But I guess you already know about this one, because they write

The bytecode generator is based on the original work of Richard Hundt for the Nyanga programming language. It was largely modified by myself to produce optimized code similar to what LuaJIT would generate, itself.

(Are they referring to your code for emitting tvmjit s-exps, or did you actually write a bytecode generator??)

Re: the debug segment. For reference, I notice that tvmjit and luajit bytecode formats and read/write code are identical, but I assume you know that.

[richardhundt]

Shine has a syntax ambiguity which I can't fix without unbounded lookahead, so I decided to start over with Blaze. The problem is that methods declarations in a class body and function calls share the same syntax:

class Foo
  bar(x) return x + 1 end
  baz()
end

baz() causes problems... it looks like an empty method, but then there's a missing end to close the class body. There's no easy way to fix this. So what I wanted to do with Blaze was move to a C-style syntax, and rethink the linker/loader and target vanilla Lua instead, as well as making it possible to compile to different targets (such as JavaScript)

[richardhundt]

And yes, I wrote a LuaJIT bytecode generator from scratch, which the luajit-lang-toolkit guys used as the basis for their project. It had some bugs in it and they cleaned up the code a fair bit, but yeah, the idea was to be able to write out LuaJIT's binary format directly.