Make a survey of languages with syntactic macros

[masak]

Starting from the languages listed in https://en.wikipedia.org/wiki/Macro_(computer_science)#Syntactic_macros, make a table with the languages on the rows and capabilities on the columns.

After the table, also summarize each language in prose, pointing out tradeoffs and limitations; details that the table is too coarse to contain. Include plenty of links to resources on the web such as reference documentation and papers.

(Do not assume Wikipedia's list of languages is complete; if there are other languages that seem to fit the description of having syntactic macros, include them. For example, I'm pretty sure we also want to have SweetJS in there.)

[masak]

The intent here being, of course, to explore the prior art in the field, both in order to be able to relate 007 to other things in some kind of ontology, and in order to loot and pillage find inspiration from other languages, especially those which have walked the same path and faced the same foes.

[vendethiel]

I see sweetjs in the Wiki article (though perhaps you added it?). I’m also sad it’s now defunct.

What are the points we’re most interested aboutin other impls? :)

[ghost]

Looking forward to seeing this table! 👍

[masak]

Just want to link this one in here: Dylan and the Lisp family tree's central core. The list of languages in that post seems worth following up on:

• Common Lisp
• Dylan
• EuLisp (ftp://ftp.bath.ac.uk/pub/eulisp/definition/defn-0.99.ps.gz)
• ISLISP
• Goo
• PLOT

[masak]

It seems I might have to include Wat in my list, too.

[masak]

Linking this syntax-rules primer from here, just to have it linked from somewhere. It also relates somewhat to #491.

[vendethiel]

You cannot write an infix' macro, nor can you write a macro in a nested position', i.e., an expression like ((foo x) y z) is always considered a procedure application. (The subexpression (foo x) will be rewritten of course, but it will not be able to affect the processing of subexpressions y and z.) This will be important later on.

The first part is untrue, for us. The second part is at odds with some other features 007 wants.

[masak]

Kernel.

[masak]

And also Virtua, by the Wat author (and "the Axis of Eval" blogger).

Needless to say, some of these languages (especially the fexpr ones) are not mainstream, and I won't be able to write a lot about them. But there will still be something to say about "how does the language do macro-ish things?".

[masak]

@vendethiel, I forgot to reply properly. Doing so now.

I see sweetjs in the Wiki[pedia] article (though perhaps you added it?).

No, but I think I might have missed it, or forgot it was there.

I’m also sad it’s now defunct.

Any idea why?

What are the points we’re most interested aboutin other impls? :)

For the table: approach to macro hygiene/gensyms, etc. Type of macros (procedural/pattern-matching).

For the prose, some details about how the macro system "feels", what limits it sets up for itself wrt syntax and semantics, maybe what known weaknesses it has (such as Common Lisp and accidental capture), but balance it out with what things it provides (such as anaphoric macros).

[vendethiel]

I’m also sad it’s now defunct.

Any idea why?

Probably a collection of factors: Disnet used to work at Mozilla, and they might've had time for open-source work (after all, sweetjs used to be under the mozilla github organisation). Maybe just lack of interest. Nate Faubion, early contributor + big part of the ecosystem (made the modules for ADT, pattern matching, etc) has joined the PureScript movement. As for the latest contributor, I don't actually know. Maybe a simple lack of time is the answer.

[masak]

Also perhaps include Z in the survey, although it seems a bit buggy awesome.

HN discussion.

[masak]

Also throwing this HN discussion in here.

I feel I need to get somewhat of a handle on syntax-rules, syntax-case, and syntax-parse.

[masak]

I think Filter::Inject deserves to go on this list also.

[vendethiel]

The use of a string reminds me of BlackCoffee somewhat.

[masak]

Also macropy.

[masak]

MetaOCaml. To the extent that I've looked at it, it has quasiquotation and unquotes, and these interact with the type system.

[masak]

Maybe D is worth mentioning in this context, because their official stance seems to be that they ended up not needing macros.

D doesn't have a preprocessor; but D provides a more scalable means to solve the same problems.

Granted, it's preprocessor/text macros they're talking about here, not real AST macros. But still.

I think the most important part of that page is the last one, about template mixins. Haven't looked deeply, but it seems that they do merit the inclusion of D in this list.

[vendethiel]

Granted, it's preprocessor/text macros they're talking about here, not real AST macros. But still.

They do, a bit, using their compile-time reflection.

[masak]

Scala has def macros (still experiemental) and deserves at place in the survey.

Rust also has macros, and the way to define them reminds me of syntax-rules a bit. (Later edit 2020-03-12: And here's a crate that implements quasi-quoting, impressive.)

[masak]

The T reference manual (and by the way, I don't know yet if T deserves a slot of its own in this survey) contains this interesting tidbit: "T's quasiquote facility, inherited from the Maclisp family of Lisp...".

Maclisp goes back a long, long way. It originated in the late 60s, and influenced Interlisp and later Common Lisp. But the claim that quasiquotes come from that language family is not backed up by the Maclisp reference manual, Revision Ø, from 1974. Maybe it was added later? Or there were variants/dialects of Maclisp, and quasiquotes weren't in the main one?

[masak]

Nor does this Interlisp reference manual (also from 1974) mention quasiquotes.

[vendethiel]

Not sure if Scala def macros should « count » still, since they were deprecated for Scala 3/Dotty.

[masak]

Wow, a Big Rewrite. Where have I heard that before? 😄

Anyway, unless I've missed a big announcement, it's not a question of "they were deprecated", and more a question of "they wioll haven be deprecated". We'll cross that Subinverted Plagal Past Subjunctive when we get to it.

[masak]

As to how/where quasiquotes originated, The History of Lisp has much of the story. (See pp72-75.)

Of note, they were originally called "pseudoquotes". (That is, the relation to Quine's quasiquotation is not there from the start.) One of the early practitioners of the art, including of nested quasiquotes, was Alan Bawden. Searching for "alan bawden mit nested quasiquote" gives this .ps.gz, which seems on skimming to be a very nice read. (PDF version.)

On the way I also found this 1993 email where Jonathan Rees recounts Alan Bawden opining strongly that quasiquote expansion should be a part of the reader. (I don't know about other implementations, but it isn't in Bel; it's a macro. That is, you can write either `(a ,b) or (bquote (a (comma b)), and they result in the same macro-expanded expression.)

This text mirthily quotes from Bawden's article, as well as provides a nice example of double-unquote, a defun macro:

(defmacro defun (name params &rest body)
    `(defmacro ,name ,params
        `((lambda ,`,params ,`,@body)
        ,,@params)
    )
)

I use this macro especially because something tells me @vendethiel used it as an example years ago, and now he'll smack me over the head with it. Either for being a slow learner, or for having a memory like a colander, or both.

[masak]

A later reference work about MacLisp is apparently called "The Revised MACLISP Manual" but nicknamed The Pitmanual after its author, Kent M. Pitman (famous Lisp implementer/documenter, and co-star in acclaimed TV Series Breaking Bad).

This one does mention backquoting and splicing. So all in all it would seem that quasiquoting and the backquote syntax was indeed invented during the evolution of the MacLisp family of LISPs.

[masak]

something tells me @vendethiel used it as an example years ago

Ah yes, here: https://github.com/masak/alma/issues/30#issuecomment-225585820 — 3.5 years ago.

[vendethiel]

and now he'll smack me over the head with it. Either for being a slow learner, or for having a memory like a colander, or both.

I don't think I do either, if anything, I'm always surprised at how much you remember was mentioned around. :-)

[masak]

I guess I am my own harshest judge. 🙂 It's taken me this many years to wrap my head around all this stuff. To the extent I understand it even now, that is.

[masak]

Groovy has macros. Apparently, they grew out of their AST Transformers, and are documented together with them here. From what I can see, they don't address hygiene at all. (I might be wrong.) Groovy's macro keyword is for quoting code, and seems to correspond to Alma's quasi keyword.

[ALANVF]

I think that Crystal, Julia, and Nim could also be mentioned here, as they all have very powerful macro systems.

[masak]

@ALANVF Agreed, after checking the documentation of all those three. Hum, somehow I was aware of those languages, but forgot to consider them for this survey. Thanks!

Also, dammit, Crystal's macros look like an impossible paradise version of Perl 6's macros. I should check them out some more.

[vendethiel]

I think they've been mentioned quite a few times over the years on IRC. Crystal's macro system is basically a pre-proc; that lives in another stage; and doesn't have access to everything else. Nim macro I remember us discussing (the delaying of type resolution while in expansion etc), but Julia maybe not.

[masak]

F# has code quotations, which look a lot like quasiquotes-done-right to me. Even the syntax somehow feels less horrible than par for the course.

[masak]

Huh. On page 18 of the "Resugaring" paper (recommended to me by @jnthn) there is actually a table not entirely unlike what I imagined for this issue. Probably would serve as a very good starting point.

[masak]

Agda has "reflection" (a way to turn code into AST objects inside the program itself), as well as quoting/unquoting and macros, as can be seen in this paper. I'm surprised and intrigued, even as I don't see the whole picture. There's something called "the elaborator", and a TC monad to maintain the elaborator's context.

[vendethiel]

I think Idris has a similar mechanism as well

[masak]

I just found Axel, a Haskell with Lisp syntax and macros, quoting, and quasiquoting+unquotes.

[masak]

And also Virtua, by the Wat author (and "the Axis of Eval" blogger).

Whose name is Manuel J. Simoni, and... I just found yet another interesting implementation of his, called "ell" (short for "Executable and Linkable Lisp", it seems).

It's interesting even as a toy because, judging by this LtU comment and its child, it implements "Dybvig's expansion-passing style". Judging from this commit, that seems to be based on the paper "Writing Hygienic Macros in Scheme with Syntax-Case" — one of those many papers I'm vaguely aware of but haven't actually read yet.

[masak]

Katahdin, of which I'm mighty surprised I hadn't heard before. There's a web page, a thesis, and an unpublished paper. It runs on PEG, which is interesting in itself.

I have a language design brewing in a private repository which is not entirely unlike this one. (Although the differences are salient, too.) My language is code-named "Pear" (because I suck at naming and I used an online project name generator to pick an essentially random name), and uses Dylan's pattern macros as a jump-off point, but then mixes together one-pass parsing (à la Raku), custom operators with operator precedence (à la Raku and Alma, but better; I didn't think I had more to say about this but I did — short story — the operator precedence can form a partial order as long as that fact is never observed), and Krishnamurthi's idea of micros (no doubt my own interpretation of them; this gives us "hygienic labels" inside code quotes, and the ability to define if and while in userspace — some of you will recognize this idea as the asm idea from Alma). In short, Pear is a collection of ideas that Alma seems to have been pulling itself towards, but it "couldn't get there from here", so Pear spontaneously arose at point B in language design space instead, out of a kind of frustration. It's not impossible that Pear will succeed as a design, and then absorb itself into Alma at some point. (Yes, I guess that means I started an experimental language in order to be able to iterate quicker than my experimental language.)

Anyway. I will read the Katahdin thesis with interest, comparing it to what I already have in Pear.

Maybe we're pushing the boundaries here on what's considered "languages with syntactic macros". That's fine, I think. It's the magic that counts.

[masak]

Passerine, which has a respectable pattern macro system that reminds me of Dylan's, but with metasyntax that looks more like Scheme's. Claims to be hygienic.

[masak]

Nemerle. Here's a paper: Meta-programming in Nemerle.

(Edit: Here's another Syntax-extending and type-reflecting macros in an object-oriented language. Also, Nemerle is mentioned as an inspiration in an early paper on Scala macros.)

[masak]

Wyvern.