Move functionality into elisp

[minad]

Hey,

this is a nice project. I had also thought before about creating an elisp runtime from scratch. One idea I had was to move as much as possible to Elisp, since Elisp is the substrate which ultimately makes Emacs hackable. The interpreter could be a metacircular interpreter written in Elisp itself (eval.c but in Elisp). The one needs a compiler which turns Elisp into native code, either precompiled or jit (cranelift?). In order to bootstrap one could use Emacs itself to produce a compiled version of the Elisp interpreter. The part in Rust would be as minimal as possible, only GC, basic primitives and maybe some higher level functions of critical for performance. Did you also consider such an approach?

[CeleritasCelery]

I think that is a great idea! I have fallen in love with the idea that emacs should be emacs lisp. I even talk a little about that here. I think especially if you can make the runtime fast, you have an opportunity to move code from core into elisp.

The hard part of doing that is bootstrapping. You need something to run your meta-circular interpreter on the actual hardware. You mention one option of using a previous version of Emacs to compile the interpreter, and I think that has a lot of merit. I would love to have Emacs lisp all the way down, at least as far as it can go. Just the runtime and a jit. If the jit was good enough you could even move most of the runtime (interpreter, jit compiler) into lisp.

[minad]

Yes, one could at least move most of the compiler to Elisp. But for the final lowering it may make sense to rely on gcc, llvm or cranelift. I would get rid of the bytecode compiler completely and go from closure conversion directly to SSA.

[minad]

I hadn't watched Tom Tromey's talk before, but I've seen that he had a similar native compilation project. For me the trend to move slow parts into C feels wrong. Instead we should improve or implement a better compiler and move functions out of C. (One still needs many low level primitives written in C or Rust.)

[CeleritasCelery]

I would get rid of the bytecode compiler completely and go from closure conversion directly to SSA.

That is kinda how the native comp works now. It just uses the elisp compiler to generate LIMPLE (SSA) and sends that to libgccjit. They don't use the bytecompiler VM at all.

However with a JIT you would need the ability to instrument the code as it was running. You could do that in an interpreter (though probably not an elisp one), but I think it would be a lot easier to instrument bytecode because the format is simpler then an AST.

[minad]

That is kinda how the native comp works now. It just uses the elisp compiler to generate LIMPLE (SSA) and sends that to libgccjit. They don't use the bytecompiler VM at all.

No, nativecomp uses the bytecode as IR afaik.

Instrumentation is necessary for advices, but they only occur for function calls. One needs a special dynamic calling convention, but this certainly hurts optimization.