Proposal: Go2: add hygienic macros

[beoran]

In #32437, a proposal is made for error handing based on a built-in function. However all that is proposed is in essence, much like append(), simply a special case macro for code that can be implemented in Go manually.

One problem I often encounter in Go is that there quite a bit of boilerplate to implement certain functionality, not only in error handling, but in general. Generics have been proposed as a solution to this, but, as can be seen from the proposal I mentioned before, if ever implemented, will be unlikely to be powerful enough to allow the go programmer to implement such error handling boilerplate themselves.

Also we have go:generate, which I use often to generate go code from text/template, but is not part of the language itself, and allows me to use all sorts of C-like preprocessors agnd generators that use text/template go code, with all the downsides of this kind of preprocessors and generators.

Therefore I propose that Go would be enhanced with hygienic macros. They would have to be powerful enough to functions such as try() and append() to be implementable in the Go language itself. They would then also allow to reduce boilerplate many other cases as well. Probably they would have to be based on AST rewriting much like hygienic macros in other languages.

I don't even want to start discussing syntax, but perhaps something like https://github.com/cosmos72/gomacro would be a starting point.

I opened this issue to see if others and the Go designers feel this idea could be useful and acceptable. It seems a better idea to me to introduce a more generally useful hygienic macro feature in Go, than, like has been done before, introduce one off macros for particular cases only. Hygienic macros are a well known feature of many programming languages, for which efficient implementation algorithms exist, which might be more useful than generics to reduce boilerplate in Go considerably, and which can be taught and explained relatively easily. So I think they would have many benefits that would outweigh the cost of implementing them.

Edit: link for more details:

https://en.wikipedia.org/wiki/Hygienic_macro

[beoran]

Yes, I admit the [] idea is not going to work. A prefix, then would a be nice marker, but it should be backwards compatible, so any of ^!|, which are already accepted by the Go lexer, cannot be used. One of \#$~could work, since they now give an invalid character error, and they are in the ASCII range, and available on may keyboards. Any one of them would be fine, but I'll just arbitarily go with the \ since that subjectively looks nicest to me, and not to keep bikeshedding a relatively minor issue.

So now, the design becomes:

1. I propose that the syntax for macro expansion is identical to that of a function call, however, prefixed with a \ marker. So a macro named Foo defined in a package named macro will be called as \macro.Foo(arg1, arg2, ...). This makes it easy to distinguish macro calls from function calls, and is backwards compatible as currently the use of a \ statement leads to a invalid character error.

2. A macro expansion may be called at the top level of a file, or inside a function. The expansion of the macro is an ast.Node which is inserted at the point of expansion. Macros are expanded at compile time, so their arguments must be fully determinable by the compiler at compile time.

3. As we can see from the several examples macro above, a macro should be able to take several arguments to be powerful enough to use namely: 3.1. An expression, to allow for things like Try(). 3.2. A type. 3.3. An identifier or variable. 3.4. A constant string. 3.5. A constant numeral (int or floating point). 3.6. A block of code. 3.7 A function literal. 3.8 ... Anything else I forgot?

[beoran]

Next would then be how to define a macro. Since we are already taking \ as a prefix for macro expansion, why not then also use the same prefix for all compile time statements? This would also make it easy to implement this proposal as a preprocessor.

So, a macro definition would then be something like \define Foo(arg1 argType2, arg2 argType2) { .... }.

[creachadair]

Although I generally like hygienic macros, I strongly oppose their addition to Go, on the grounds that they harm the readability of code and thereby the ease of its reuse.

As with many proposed language features, macros are understandably appealing to the writer of code, whose effort is saved by their application. Even medium-size programs often contain meta-patterns not easily represented in the syntax (e.g., type dispatch) and therefore feel tedious to write out longhand. For the reader, however, who does not already understand the code, I have found macros make the learning process much worse. I think this depends less on whether or not the macro expander is hygienic, and more on whether the authors have moved on to bigger and better things.

For obvious reasons I can't describe any direct experience with macros in Go, but a couple decades ago I spent about six months doing some heavy-duty maintenance on a large (~20K lines or so) Common Lisp codebase. It made heavy and very clever use of macros throughout. As you probably know, CL macros are not hygienic by default, but that didn't really matter in this case: The authors made no extensive use of captures, and had carefully inserted gensyms or used naming conventions to avoid problems. For the reader, however, their extensive macrology was disastrous: The program relied on an elaborate library of control-flow constructs—via macros—riddled with subtle and poorly-documented assumptions. It took me nearly a month to gain a rudimentary working understanding of the program's architecture, and much longer to become productive in it. My knowledge of Common Lisp was largely useless in this research: Beyond the bits you could learn in a ten-minute reading of the Hyperspec, this program was effectively a new language of its own.

I acknowledge that my anecdote does not prove anything: You may justly argue that Common Lisp is not Go, and that perhaps I am not the brightest candle in the menorah, and that Lisp programmers are well-known to fetishize cleverness, and that Go programmers would only use such a facility judiciously and with taste. The first three, at least, are probably true. I've had to read and maintain enough complex projects, however, that I do not believe we should privilege the convenience of the writer of the code (typically one person, or a small handful) over the needs of the reader of the code (often many people, over a long period of time).

One of Go's virtues is that it is boring. "Boring" is annoying when you have to write a bunch of similar code with small tweaks—but it is a virtue when you are trying to figure out what went wrong after the fact. The obvious response is that "repetitive boilerplate is also hard to read". That is true, but it has also been my experience that, the majority of cases where there's a lot of boilerplate (and where macros would be useful) can be served as well—and maybe better—by writing a little code generator in the original (boring) language.

Arguably a macro is just a code generator: But it's an implicit one. With explicitly-generated code, the relationship is clear during the build process, and the reader can clearly see the boundary between the code that was "generated by X from Y", and the parts a human wrote out longhand. A minor tweak to an innocent-looking data parameter is less likely to destroy the entire build.

It's easy enough to go look up the provenance of generated code later, if you need to. I have found that archaeology on with the language equivalent of gcc -E or MACROEXPAND to be much more tedious. Plus: It slows down the compiler quite a bit, in ways that can surprise someone who didn't write the macro. And: It makes diagnostics much harder to surface in the UI (a problem familiar to anyone who's built tools around clang).

In summary: While the benefits of macros to the writer of a program are obvious, their costs to the reader seem too great to consider. I do not believe the benefits are sufficient to justify working around the costs.

[Chillance]

@creachadair I see where you are coming from but how about thinking of ways to make macros more "boring" for Go? This is also why we are having this discussion, so we can figure out proper ways to do macros. "Anything that can be used, can be missued." is a quote someone said too that came to mind here. Meaning, you can use macros to make things harder for yourself, but that goes with anything really. And, if macros was a bit "dumb down" for Go, it might actually be more harder to complicate things with macros and push using macros for simpler and less convoluted cases.

One thing that could make it annoying would be if macros were overused. For cases like implementing "try" using macros, macros could be quite neat. You learn how that works in your code, and then every time you see it you know what it does (check err and return something if not nil). Obviously, if macros were overused, you would have to keep more in your head, which could make code less readable and annoying with more overhead in your head. But again, this comes down to just using macros for cases where it makes more sense. And use more rarely compared to functions I suppose.

[beoran]

Yes, I'd agree that certainly flow changing macros should be used sparingly, and documented well. But use does not prevent abuse, and perhaps we can think of good ways to limit the potential for abuse.

As for using code generators, that's what I use all the time in my day job. As I stated above, these are normally based on text/template, and the experience is not so comfortable, because text/template is not Go, but it's own little language, and does no syntax checking of the Go code it generates. I modify the template, run the generator, then run the compiler and ... I get an error if I made a mistake in the template. A macro system in Go would alleviate that problem.

[creachadair]

As for using code generators, that's what I use all the time in my day job. As I stated above, these are normally based on text/template, and the experience is not so comfortable, because text/template is not Go, but it's own little language, and does no syntax checking of the Go code it generates. I modify the template, run the generator, then run the compiler and ... I get an error if I made a mistake in the template. A macro system in Go would alleviate that problem.

It seems clear that you could benefit from a better code generator tool. What is less obvious (to me) is why macros would be the correct solution to this problem, as opposed to (say) a library with better support for the output language—perhaps based on the go/parser package, since it sounds like you are maybe generating Go—with support for writing and checking Go syntax, formatting, type-checking, etc.

It would take some work to get this right, but if Go is the output language much of that work is already well-supported by existing libraries, and a well-designed API for generating Go could be a nicely reusable package. I don't think this needs to be baked into the core syntax of the language, though, which affects all users of the language rather than only those who need to generate Go code.

[beoran]

Yes, I am generating Go. The problem is that I have to use certain tools, such as gqlgen where I don't get to choose the language that is used for the templates. The reason why text/template is used so widely to generate Go code is because it is part of the standard library.

As an alternative to this proposal, perhaps a go/template library could be implemented that works as you suggest, but then it should probably become part of the standard library, otherwise it will not be widely used. That's why I still think it would be nice to have macros in the language in stead of as an external tool. It makes the feature more widely accessible.

[beoran]

Seeing the latest progress with generics, I would like to see what are the results of this endeavour. If successful it could reduce the need for macros.

[beoran]

As it seems generics will fill some of the gap, the best approach for macros in go is to build an external macro pre-processor at first. Since I was unable to come up with one in a reasonable delay, and seeing interest has died dout, I will voluntarily close this issue to allow the Go team to focus on more pressing issues.