Higher-order macros and macro definitions with explicit delimiters

[fpottier]

Hello,

This PR proposes:

• Higher-order macros. This does not break any existing code: the formal parameters of a higher-order macro must be annotated with their types, and there is a new syntax for this purpose.

• Macro definitions with explicit delimiters, #def and #enddef. This allows multi-line macros (without backslashes) and nested macros. This is unlikely to break any existing code, but it does introduce two new directives.

• (minor) #define on the last line of a file, without a newline, is now accepted. (It used to be an error. Accepting it allows me to produce a reliable error message when #def does not have a matching #enddef.)

• (minor) Better locations for some syntax error messages, by systematically using long_loc instead of loc.

The new features are documented in README.md.

Some examples of the use of higher-order macros appear in the example file test/higher_order_macros.cppo. Similar examples, which also take advantage of #def ... #enddef, appear in test/def.cppo.

Here are some more ideas, about which I would welcome the opinion of the cppo maintainers. I believe that each of these ideas would be easy to implement:

• I am tempted to introduce #scope and #endscope delimiters, so as to allow limiting the scope of a macro definition. I believe that this would often remove the need to use #undef, which is quite painful.

• The use of higher-order macros is currently somewhat heavy, because the actual argument must be a named macro, so, at the call site, it must be locally defined (and possibly undefined). I am tempted to introduce a syntax for macro-abstractions, which would be allowed to appear as actual arguments to higher-order macros. The syntax would be #lambda(formals) ... #endlambda. Perhaps, for greater conciseness, it would be necessary to allow #lambda and #endlambda to be recognized as directives even if they are not placed at the beginning of a line.

• In order to improve hygiene in higher-order macros, I would like to introduce a new directive #fresh x y, which behaves like #define x <some fresh name based on y>. A globally fresh name would be obtained by combining the prefix y with a numeric suffix obtained by reading and incrementing a global counter.

• Because #undef is heavy, I have been tempted to add a command-line switch --allow-shadowing, which allows a new macro definition to shadow an existing definition. That said, perhaps #scope ... #endscope would lessen the need for this feature.

[pmetzger]

A minor comment:

(minor) #define on the last line of a file, without a newline, is now accepted. (It used to be an error. Accepting it allows me to produce a reliable error message when #def does not have a matching #enddef.)

Not that C has much to do with this, but I'll note that in C, a translation unit (aka a file) without a terminating newline is documented as UB.

[pmetzger]

Also, not that Lisp is a precedent we need to care about...

In order to improve hygiene in higher-order macros, I would like to introduce a new directive #fresh x y, which behaves like #define x . A globally fresh name would be obtained by combining the prefix y with a numeric suffix obtained by reading and incrementing a global counter.

In Lisp, this would be a gensym.

[pmetzger]

So a lot of this looks neat. I am obligated to ask, of course, what the anticipated use case is. I've traditionally viewed cppo as a sometimes needed hack that one generally wants to avoid. Also, since I started doing a lot of work on refactoring C code I've come to hate cpp in that context because it makes it harder to automate refactoring. However, some of this seems like a very cool experiment.

[fpottier]

So a lot of this looks neat. I am obligated to ask, of course, what the anticipated use case is.

Good question! I don't really have a clear use case at this point. Lately, for performance reasons, I have been playing with unrolling loops and specializing higher-order functions (which the OCaml compiler does not do), so cppo came in handy. This led me to think about the potential uses and current limitations of cppo, and I thought that it could be useful to remove some of these limitations.

[fpottier]

In Lisp, this would be a gensym.

Sure. I would be happy to use whatever syntax you think is most convenient or elegant.

[pmetzger]

Lately, for performance reasons, I have been playing with unrolling loops and specializing higher-order functions (which the OCaml compiler does not do), so cppo came in handy. This led me to think about the potential uses and current limitations of cppo, and I thought that it could be useful to remove some of these limitations.

I have never used metaocaml, but I understand it is also useful for such things. Would this serve a distinct need?

Also, I have no particular brief for gensym, I just am noting it's an analogous piece of functionality.

[pmetzger]

What do we need to do next to move this pull request forward?

[pmetzger]

Seems fine by me.

[fpottier]

Is there something I can/should do?

In particular, I haven't written an entry in Changes.md.

[liyishuai]

Given the time length for re-running the CI, maybe worth editting the changelog in a separate commit.

[fpottier]

I have pushed two commits to fix a couple typos and add an entry in Changes.md. If the CI passes then I believe it is ready to release as 1.7.0.

[fpottier]

I believe that the CI has succeeded. Is it OK if I release by running make publish1.7.0? Or should I wait for a maintainer to do it?

[liyishuai]

Please fire at will. You might need to create a tag for v1.7.0 before running the Makefile script.

[pmetzger]

@fpottier Just FYI, you're as much a maintainer now as the rest of us. :)

[fpottier]

It's an honor :-)