Why elisp src2src compiler over interpreter?

[martintrojer]

I'm sure you have a good answer to this one, but couldn't find it in the README so I thought I'd just come out and ask it here.

Why did you chose to src2src compile elisp over writing an interpreter? An interpreter feels like the natural choice does it not? Especially if you want to work on your elisp modules from within deuce?

[hraberg]

This is a good question, which I actually don't have a good answer to. I think the initial idea of "easily" transpiling Emacs Lisp to Clojure seemed like less work and gives you a lot for free (kind of..), but in hindsight I'm not really sure. I also came to the problem from a shen.clj angle - which was what seeded this madness in the first place - I use Clojure as a bytecode compiler with Lisp tendencies (there are several issues with this).

From a pure run-time point-of-view it doesn't matter too much, source is only generated for loaded Emacs Lisp, things you evaluate on the fly is just fire and forget. It doesn't limit what you can do once Deuce is running, I basically treat Emacs Lisp as a Clojure macro dialect confined to the deuce.emacs namespace.

The core Emacs Lisp is pretty contained in deuce.emacs-lisp, so you can take a stab of rewriting it as an interpreter if you feel like :-).

[martintrojer]

Yeah, the one thing that comes to mind for compiling is speed (by doing clever optimisations) -- but I'm pretty sure an interpreter in clojure would perform well compared with the elisp interpreters in emacs or guile.

So can the macros in deuce.emacs-lisp run elisp without any s2s compilation?

[hraberg]

Yes, the s2s bit is in deuce.emacs.lread (obviously..), which writes a .clj file for any .el file that passes through its load fn. This is also very helpful for debugging, as early on I didn't have this step and just created and large trees of Lisp without any clarity or sanity. The generated Clojure is... Clojure. And the macros in deuce.emacs-lisp are "normal" Clojure macros, so the Emacs Lisp parts of Deuce has the same abilities as that Clojure has, you can use them from the REPL etc, no source needed.

Once Deuce is "dumped" to an uberjar, it doesn't load any new Emacs Lisp at all during the boot, its just a large, weird Clojure program. The rest of Emacs is bundled as .el files, and can be required or autoloaded, in which case they get generated and compiled into ~/.deuce.d.

Speed is an interesting part, the idea is that having everything as basically Clojure should help. But it also makes it pretty opaque at times, and while compiled, Emacs Lisp has some extra indirection (mainly due to dynamic scope and allowing late binding in general). AOT also takes for ever for some reason (there are over 10k classes generated). But it shaves off start-up time considerably for later runs.

[Ericson2314]

I'll agree source to source compilation is generally better. If you find yourself evaluating the same code over and over again, it's a good sign you should compile it ahead of time, not use an interpreter.

[martintrojer]

So it would be possible (atleast in theory) to skip the compile step all-together and just use the clojure reader and the deuce.emacs-lisp macros / functions to run all necessary elisp?

As for the need of compiling -- Emacs itself has done fine with just an interpreter for a number of decades now :)

[hraberg]

So a couple of things, as Ericson says, "Emacs" itself doesn't change between releases, so it can make sense to AOT it, assuming it makes things quicker (which it currently does) to start-up.

There's an Emacs Lisp reader, which lives in deuce.emacs-lisp.parser, which is needed to create the initial in-memory representation of Emacs Lisp-as-Clojure from .el files. You cannot easily read Emacs Lisp using Clojure's reader (believe me, I've tried!). This is then written to (and compiled from) disk by deuce.emacs.lread/load via using pprint, and at this point it's now "normal" Clojure. And there's no real need to AOT it from here on. Initially I was of the opinion that doing so was "to give up", but give up I did.

Emacs Lisp (referred by loadup.el), when running Emacs, is not only compiled to bytecode (the normal .elc files, similar to .class), but actually baked into the emacs binary itself: http://www.gnu.org/software/emacs/manual/html_node/elisp/Building-Emacs.html

Emacs runs "compiled" byte code faster than source. Both are interpreted, but I don't know this part of Emacs well, as it's what I'm getting rid of :-) http://www.gnu.org/software/emacs/manual/html_node/elisp/Speed-of-Byte_002dCode.html

Having said all that, yes I agree, it would be nice to avoid the AOT for all the normal reasons :-)

[martintrojer]

Thanks for sharing this Håkan, very interesting.

I guess the devil is in the details... :)

[hraberg]

Thanks for asking Martin, happy to discuss.

BTW, I'm open for a total rethink and rewrite of the core (it's not big, but quite ... delicate, OK, it's a hack!) once Emacs itself runs respectably under Deuce. Right now I'm living with what we've ended up with (for better or worse) so far, as there are many other battles on the Emacs front to be fought than just the one with Emacs Lisp itself.

[hraberg]

And, just as a final note before closing this - I see one primary goal of Deuce to actually get to the point where someone (potentially myself) can come and say "OMG what is this, let's rewrite it clearly and well", but then not also have to deal with doing the initial port from Emacs C, but can start from a scruffy Clojure version.

[Ericson2314]

God, i worry that the fact they chose a loop for the byte-compilation example means that the code is re-parsed and everything with each iteration. Is emacs lisp even tail recursive?

martintrojer, I would argue emacs itself is over-reliant on interpreting things. Besides that fact the emacs lisp itself is shitty, the other big reason for projects like guile-emacs and deuce is that the elipse implementation and the way it is used are not very efficient at all (relatedly, it doesn't make sense to re-invent the wheel for emacs with regard to this stuff).

I think the most robust goal would be to use the same macros to compile or interpret elisp via clojure, and not have to use any temporary files. Additionally, if clojure itself desugars down to some reduced subset of the language before being converted to class files, it may make sense to target that as an optimization as I'd suspect whatever the macros output now is pretty low-level-looking. That may speed up interpretation but shouldn't affect the compiled the compiled class files much.

Anyways, as you say it will be great to have deuce do all the reverse-engineering for anybody that wants to implement ellipse. I would be interested in cleaning and optimizing this bit come summer.

[hraberg]

Any cleanup and performance improvements (which may actually go hand in hand) will be most welcome once 0.1.0 is out. An obvious problem is test coverage. At the moment Emacs starting acts as a smoke test. I want to get org-mode's test suite running against Deuce eventually (there's a section about this in the README).

[Ericson2314]

Ah, I just realized ERT was just a library for testing and not a series of Emacs tests in itself. Yeah org-mode should help a lot.