Proposal: Go2: add hygienic macros

[beoran]

In #32437, a proposal is made for error handing based on a built-in function. However all that is proposed is in essence, much like append(), simply a special case macro for code that can be implemented in Go manually.

One problem I often encounter in Go is that there quite a bit of boilerplate to implement certain functionality, not only in error handling, but in general. Generics have been proposed as a solution to this, but, as can be seen from the proposal I mentioned before, if ever implemented, will be unlikely to be powerful enough to allow the go programmer to implement such error handling boilerplate themselves.

Also we have go:generate, which I use often to generate go code from text/template, but is not part of the language itself, and allows me to use all sorts of C-like preprocessors agnd generators that use text/template go code, with all the downsides of this kind of preprocessors and generators.

Therefore I propose that Go would be enhanced with hygienic macros. They would have to be powerful enough to functions such as try() and append() to be implementable in the Go language itself. They would then also allow to reduce boilerplate many other cases as well. Probably they would have to be based on AST rewriting much like hygienic macros in other languages.

I don't even want to start discussing syntax, but perhaps something like https://github.com/cosmos72/gomacro would be a starting point.

I opened this issue to see if others and the Go designers feel this idea could be useful and acceptable. It seems a better idea to me to introduce a more generally useful hygienic macro feature in Go, than, like has been done before, introduce one off macros for particular cases only. Hygienic macros are a well known feature of many programming languages, for which efficient implementation algorithms exist, which might be more useful than generics to reduce boilerplate in Go considerably, and which can be taught and explained relatively easily. So I think they would have many benefits that would outweigh the cost of implementing them.

Edit: link for more details:

https://en.wikipedia.org/wiki/Hygienic_macro

[networkimprov]

At least one of these links is a general purpose macro system https://github.com/golang/go/wiki/Go2ErrorHandlingFeedback#inlining

@gopherbot add Go2, LanguageChange

[beoran]

Yes, this ...

https://medium.com/@phlatphrog/handling-more-than-just-errors-in-go-f97c5aa2eac4

... Looks like it could fit the ticket, with some refinement. Thanks for the suggestion.

Op vr 14 jun. 2019 17:07 schreef Liam notifications@github.com:

At least one of these links is a general purpose macro system https://github.com/golang/go/wiki/Go2ErrorHandlingFeedback#inlining

@gopherbot https://github.com/gopherbot add Go2, LanguageChange

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/32620?email_source=notifications&email_token=AAARM6OKHW5VLDN7ZZCD4O3P2OX4BA5CNFSM4HYHEYT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXXB4ZQ#issuecomment-502144614, or mute the thread https://github.com/notifications/unsubscribe-auth/AAARM6MNC3IHQMVF2QR3OWDP2OX4BANCNFSM4HYHEYTQ .

[ianlancetaylor]

If the try builtin proposal can be implemented as a macro, then macros can change control flow. That does not strike me as a good feature. A number of people have reasonably objected to try on the grounds that it changes control flow, and therefore makes the program harder to understand. If macros can change control flow, then macros can similarly make a program harder to understand. Much harder. That seems like a problem.

[urandom]

@ianlancetaylor catch from the draft proposal can also change control flow. There's nothing inherently wrong with that, and as with other language constructs that do the same, I assume people will either learn how they work or not use them.

[beoran]

@ianlancetaylor Personally I don't think the problem with try is that it changes flow control. Actually, to be useful and save on boilerplate, it should do so. The problem is that we can't implement such flow control changing try, cond, etc, macros ourselves, and customize them to our own liking.

And if flow control changing special case built in macros like try are desirable to reduce boilerplate, then moreso can be said for programmer defined macros, when used judiciously like @urandom suggests. The Go standard library could contain a few standard macros that are well documented and of common use, such as try.

If you read the link to phlatprog's medium you will see how he suggests a macro similar to try could be implemented. He suggests that a macro is like a func, but the body of the macro is inserted hygienically inline in the macro call site with the parameters substituted in the body. I really like this idea, because it is simple to explain, but extremely useful.

[iand]

Rust implements it's try feature as a macro: https://doc.rust-lang.org/1.9.0/std/macro.try!.html

[deanveloper]

I personally agree with @ianlancetaylor's sentiment here. Maybe if we had a distinction between macros and functions (such as rust's try! and println!), this would seem a bit better. Then we can at least clearly see where control flow may be happening.

[beoran]

Well, why not? I didn't propose any syntax yet, but maybe some marker like ! or so could be nice. Good to see that try can be implemented as a macro in a language that has hygienic macros. I'd say that this demonstrates the advantages of this idea.

[ianlancetaylor]

I think the advantages of hygienic macros are fairly clear.

What I'm personally concerned about is the disadvantages. In particular, it seems to me that it can easily become much harder to read Go code written by somebody else using a different set of macros. I'm concerned that it could become much harder to share Go code, not in the sense that the shared code doesn't run, but in the sense that it is hard to understand and debug the code you didn't write.

[urandom]

I think the advantages of hygienic macros are fairly clear.

What I'm personally concerned about is the disadvantages. In particular, it seems to me that it can easily become much harder to read Go code written by somebody else using a different set of macros. I'm concerned that it could become much harder to share Go code, not in the sense that the shared code doesn't run, but in the sense that it is hard to understand and debug the code you didn't write.

Is there any such precedent in languages with hygienic macros? Granted, that might be hard to observe, but it could be a nice use-case nonetheless.

[beoran]

During the discussion try(), some people also worried that it makes the code more difficult to read. But that argument was refuted, because it is not because try() can be abused, or because try() changes the flow control, that it becomes so complex that it cannot be used safely. People will learn to use this feature correctly and understand how it works.

The same goes for hygienic macros. While potentially they could make program more difficult to understand, the Go users will learn how to use them, and when use correctly they will provide us with better abstractions and less boilerplate. If you think about it, even a function is an abstraction that hides away details, but that doesn't stop anyone from defining functions. The same goes for macros, we will have to read the documentation of a particular macro to learn how it works, just like for any unfamliar function that we want to use.

As suggested above a way to distinguish them from function calls could make it even more clear to the reader what is going on. Perhaps, simply the convention that any macro that is not builtin to Go, should be defined in a package named macro. In the source code then the call will look like macro.Foo(), which makes it pretty obvious what is going on.

As @urandom suggests, there are many languages with hygienic macros. We can take a look at large projects in such languages and evaluate the benefits and downsides of macros through them. One example might be Mozilla Servo, which is written in Rust and defines quite some macros: https://github.com/servo/servo/search?q=macro_rules%21&unscoped_q=macro_rules%21. In that project, the macros are not well documented, but even just by their name, it is often quite obvious what they do, and they cut back on large amounts of boilerplate. There are many other possible projects in other languages as well we could look at.

[fishedee]

I agree this, I write a tool use golang AST tree to generate concrete code. If have hygienic macros,I think it will more easy to use and more easy to write

[beoran]

I cheked @fisedee 's tool and his experience with Go is very common. I had a similar experience. It goes a bit like this:

1. Write everything manually, leading to a lot of boring boilerplate copy pasted everywhere.
2. Switch to libraries that use interface{} and reflection. These are much more convenient, but have mediocre performance and no type safety, leading to many difficulties.
3. Switch to code generation using go:generate + a custom tool. Now, we have better performance and type safety, but it comes at the cost of having to use an external tool. These tools often use text/template as a template language for generating go from. While that is OK, now it means that go programmers have to now both Go and text/template's language, and get all the downsides of having to use a text based preprocessor.

What we need is:

4. Use hygienic macros. Like this we can generate code directly in the Go compiler itself, without the need for external tools or preprocessors, and without having to learn additional template languages.

[beoran]

I added a link to this issue on the https://github.com/golang/go/wiki/Go2GenericsFeedback, because hygienic macros are an alternative to generics as well.

I think hygienic macros would be better for go than generics, because experience from other programming languages shows that hygienic macros have all the power of generics, and are furthermore relatively easier to understand, implement and use. Furthermore, they tend to work similarly between different programming languages, making it a "portable skill", which cannot be said about the highly different way in which generics are implemented. Finally hygienic macros are a well understood feature, with well understood implementation algorithms available, making them far less experimental than the dozens of various generics proposals that have been floating around.

[networkimprov]

What does a macro-defined type look like? Or a macro method?

[beoran]

I'm not proposing any syntax yet, but we can take a look at other languages that have hygienic macros, like Rust, and it looks a bit like this:

• Macro defined method in Rust: https://github.com/servo/servo/blob/799490a02e9bea575bf34c39f045ef0883539f05/components/script/dom/macros.rs
• Macro defined type in Rust: https://stackoverflow.com/questions/41391522/how-to-use-a-type-ty-within-a-macro-to-construct-a-struct-instance-in-rust

Altough admitted, Rust combines generics and macros, I feel that with powerful enough hygienic macros, generics will not be needed.

[jsjolen]

@beoran, @networkimprov

To implement generics through macros you could define a defining macro along with an instantiating macro.

Something like:

defineGeneric!(T) {
    func foo(a T) {
      // ...
   }
}

// At call-site
myFoo := instantiate!(int, foo)

This would be similar to C++ templating system.

This is a `80% solution' though because the type system would have no idea about parametric polymorphism.

EDIT: And of course the guy who wrote gomacro did this in Common Lisp: https://github.com/cosmos72/cl-parametric-types

[beoran]

@jsjolen Yes, although I'd make it so that the macro is "typed", like in rust or other languages. With Typed macros the definition specifies the AST types that the macro takes as arguments, and also specifies the AST type of the object that must be injected at the point of invocation. Like this, macros don't only become hygienic but also meta-type safe.

Admitted it is an 80% solution, but Go is the language of good 80% solutions that work well with each other. Hygienic macros are orthogonal to interfaces and other Go language features, something that cannot be said of generics, where concepts such as contracts have to be introduced to make a distinction with interfaces.

[Chillance]

I did this as an example in #32620 and #32811

define returnIf(err error, desc string, args ...interface{}) {
    if (err != nil) {
        return fmt.Errorf("%s: %s: %+v", desc, err, args)
    }
}

func CopyFile(src, dst string) error {
    r, err := os.Open(src)
    :returnIf(err, "Error opening src", src)
    defer r.Close()

    w, err := os.Create(dst)
    :returnIf(err, "Error Creating dst", dst)
    defer w.Close()

    ...
}

Edit: Also added colon in front of the macro to suggest that maybe that can be done to clarify it's a macro and not a function call.

[beoran]

Yes, a way to mark macros is desirable. However, macros will be defined in packages, so the package name will need to be prefixed when using the macro. So in stead of : or ! As a prefix or suffix, as I said before, just the convention that the package must be named macro is enough. Then above, :returnIf becomes macro.returnIf which is longer but even more clear.

[deanveloper]

@beoran How would importing macros from different packages work in that case?

[beoran]

Like it works now, for packages with the same name, by specifying the name to import. Something like

import (
  "builtin/macro"
  macro2 "github.com/chillance/returnif/macro"
)

func useMacro(){
    macro.try(foo())
    macro2.returnIf(foobar(), "error")
}

[rodcorsi]

Maybe the syntax could be based on text.Template, the users are familiar and is very powerful with conditions and loops

define returnIf(err error, desc string, args ...interface{}) {
    if ({{$err}} != nil) {
        {{if $args}}
        return fmt.Errorf("%s: %s: %+v", {{$desc}}, {{$err}}, {{$args}})
        {{else}}
        return fmt.Errorf("%s: %s", {{$desc}}, {{$err}})
        {{end}}
        }
    }
}

edit: Reading the comments below the text.Template isn't the best alternative for macros in Go, because could bring an invalid syntax and is hard to read

[beoran]

While text/template is very powerful, it is definitely not Go language but it's own little language. I didn't propose a syntax yet, but I would like it to be as close as possible to normal Go.

[turtleDev]

I totally agree with @ianlancetaylor ; one of the great features of go is that it's a carefully defined small set of features that have reliable behaviour, which makes it easy to read and understand. Adding user defined control flow via macro's is a big no for me, since it would let you define different behaviours that people may or may not expect, not to mention the added complexity of dealing with the macro's themselves.

[natefinch]

Macros are the worst thing in C and C++. You literally stop writing the language and start writing some other language that is defined by whoever decided they didn't like typing out some boilerplate. When I was writing those languages I was literally just copying and pasting how other people wrote macros because they were so awful to understand.

I like that any random line of Go code I look at, I can instantly understand its computational complexity purely based on the text in this file. The most you have to wonder about is if append will have to reallocate an array.

There's never a time when I want to look at a line of code and have to wonder if it's the first part of an if statement, or if it has a return statement inside, or if it's actually a loop.... and then go look at the content of the macro and see some weird other language that isn't go... No no no, 1000x no.

[urandom]

@natefinch The proposal here isn't about C style macros, so I'm not sure how relevant your first part is. As for your second part, besides a return statement, the rest would be exactly the same as a regular function. What your are suggesting is that functions be removed then?

[natefinch]

There's very little proposal here, since there's no syntax, so I just went off the macros I'm familiar with. You can't say it's not like C macros if you don't actually say what it is like. Sure, it's hygenic, so it won't clash with existing variable names, but that doesn't mean it wouldn't have all the other problems of C-style macros, i.e. you end up with functions that look like this:

func (h *handler) handleUserPOST(w http.ResponseWriter, r *http.Request) {
    DO_SETUP(h, w, r)
    CHECK_AUTH(h, r)
        // log to sentry
        h.sentry.AuthFail(r)
        AUTH_FAIL(w)
    AUTH_FINALIZE(h)
   // now actually do user post stuff
}

[Chillance]

Do note that the point here is also to discuss what to do, so we can agree on something that is better than what is macros in other languages. I don't think it's wise to dismiss macros because it has negative impact on other languages. The reason to use macros and why I pasted my code is for the discussion to take place where macros can be made better. This is how Go was created. Take good parts from other languages and make it simpler and easier in Go.

[beoran]

@natefinch

The situation which you describe, where I have to work in a weird language that isn't Go is part of my daily work. I have to use text/template, which isn't Go, to generate my go code, because that is currently the best solution for type safety and performance. Hygienic macros with a correct syntax would allow me to get back to programming Go.

As for your example, this is how it probably would look now, in well factored code:

func (h *handler) handleUserPOST(w http.ResponseWriter, r *http.Request) {
    err := doSetup(h, w, r)
   if err != nil {
       return err // or handle the error somehow.
    }
    err = checkAuth(h, r)
    if err != nil {
         return err
    }
     // log to sentry
     h.sentry.AuthFail(r)
     err = authFail(w)
     if err != nil {
         return err
     }

     err = authFinalize(h)
     if err != nil {
         return err
     }
     // now actually do user post stuff
}

So you still need to lookup what the called functions do to understand what this function is doing. Macros don' t add much more of a difficulty, apart from the fact that they might return. You will have to read the macros to see if they do, which is not more difficult that reading functions to see what they do. Furthermore, thinking about implementation, perhaps we could mandate that a package defines macros can only do that and nothing else, to create a clear separation.

More likely than the example you posted, with macros it would look like this:

import error_macro "example.com/error/macro"

func (h *handler) handleUserPOST(w http.ResponseWriter, r *http.Request) {
   error_macro.Try(doSetup(h, w, r))
   error_macro.Try(checkAuth(h, r))
   h.sentry.AuthFail(r)
   error_macro.Try(authFail(w))
   error_macro.Try(authFinalize(h))
     // now actually do user post stuff
}

Which is more explicit that the current try() proposal. And you'll be able to look up the documentation and definition of the macro easily with go doc example.com/error_macro Try, or through your IDE, something you can't do with C.

As @Chillance says, I don't propose C macros like or C++templates. I think that thanks to all comments above I can home in on the desirable features of macros a bit more finely, namely:

1. They must be hygienic
2. They must have a definition syntax that is a Go like as possible.
3. They must be type checked at compile time, both the parameters and the body.
4. A macro invocation expands to the syntax tree of it's definition with the macro parameters expanded and replaced.
5. In the call site they will be prefixed by the package name, or maybe also with some kind of sigil like ! or : to make it clear that a macro is invoked.
6. Macros may only be defined in separate macro packages that may only contain macro definitions.

[urandom]

@natefinch

Indeed, one of the pain points of C style macros that are more or less text expansion, is that expanding a macro might leave you with an invalid syntax. I assume this is what at least I've if the things you wanted to show in your example, and is something I've seen a lot in C.

I'm hoping that if Go ever gets macro support, said macros will not be just some text substitution, and will always produce valid syntax when expanded.

[Chillance]

@beoran That handleUserPOST looks rather nice when using macros, don't you think? :) So, with the macro you can get the code smaller and using less rows. If needed, one can always take a look at the macro to understand what it does, and then you know what it does for all the rows.

[ianlancetaylor]

I've already expressed my reservations about this proposal. I'm skeptical that this will be adopted.

That said, I don't see how we can fairly evaluate it without a syntax. Without a syntax we can't tell how hard or easy it is to use or implement, we can't tell what complexities might arise when using it, we can't tell what existing code would be simplified. I'm going to mark this proposal as being on hold until there is something more complete to examine.

Thanks.

[sylr]

I'd love to have macro in Go. One thing it could solve for me is duplication of code when casting back objects stored in a cache as interface{} back to their original type, i.e.:

https://github.com/sylr/prometheus-azure-exporter/blob/v0.5.0/pkg/azure/batch.go#L85-L91 https://github.com/sylr/prometheus-azure-exporter/blob/v0.5.0/pkg/azure/batch.go#L132-L138 https://github.com/sylr/prometheus-azure-exporter/blob/0.5.0/pkg/azure/batch.go#L181-L187

So far that's the only time I felt the urge to have macros and although I would take go over rust any day for code readability reason (far too many symbols for my taste), I'm really envious of rust's macro system.

I'm not very fond of the idea of having to segregate macro in their own package though. I'd like to be free to define right next to where it would be useful.

Would be nice to be able to make them public or private the same way we do with func (first letter capitalization).

So what now ? Here a cheap attempt at porting some of https://doc.rust-lang.org/stable/rust-by-example/macros/designators.html examples go style

macro createFunc($funcName ident) {
    func $funcName() {
        fmt.Println("You called: %s", expand fmt.Stringify($funcName))
    }
}

// Not sure about $(x:type)
macro FindMin($x ...expression) $(x:type) {
    var min $(x:type)

    for k, y := range $x {
        if k == 0 || y < min {
            min = y
        }
    }

    return min
}

func main() {
    expand createFunc(foo)
    expand createFunc(bar)

    foo()
    bar()

    x, y, z := 2, 0, 4
    i := expand FindMin(x, y, z)
}

[Chillance]

@sylr Almost like it's also generics. :)

[beoran]

@ianlancetaylor The reason I did not come up with a syntax yet was exactly because I wanted to test the waters first. If you feel that the feature of hygienc macros itself will be rejected then of course I don't have much incentive to propose a syntax.

On the other hand, whilst the response to this proposal has been modest, in balance it has been more positive than negative, so for the benefit of the people who encouraged me, I will go to the next step and also propose a syntax. Please give me some time to consider which syntax would be best.

For the moment, I think that perhaps @sylr's idea, but then with 2 built in functions in stead of keywords, namely macro() to define a macro and expand() to expand one seems like an interesting approach, but I will write it out more in detail.

[networkimprov]

I wouldn't expend the time on the basis of 20-some upvotes. You already heard that the Go team isn't interested.

[sylr]

@beoran If your idea of macros rely on built-in go functions then I withdraw my interest.

[ianlancetaylor]

@beoran As I said, I'm skeptical that this will be adopted. If you don't want to waste your time devising a syntax, and want to withdraw this proposal instead, that seems like an appropriate choice.

But I'm just one person. I'm not making the decision here. And the other things I said are also true. A proposal like this cannot be fairly evaluated without details of the syntax and a clear understanding of the power of the macros.

[beoran]

Well, I'm not going to quit without trying, so I will work on designing a syntax and specify the exact possibilities I have in mind. I started looking at it but admittedly, it is harder than I thought, so it will probably take a while before I have something that can be reviewed. If anyone else would like to make some suggestions in the mean time, they would be welcome.

[networkimprov]

If you have time to burn, use it to build a following. You'll need ~1,000 supporters I'd guess, enough so that "I can has macros?" starts appearing as a leading request in the annual surveys.

See also Zig, which offers "comptime" blocks -- code that executes at compile time.

[turtleDev]

@networkimprov I'm not sure if supporters will actually help. What Go team wants (or expects) is a proposal that can help improve the language in the vision that they carry. Even now, when you look at the language, there are dozen of features that the authors could have added (methods on slices come to mind, or data structures like set et al) but they instead elected to keep the very minimum of features that they thought were necessary.

Same more or less goes macros. Just because it's a thing that solves a problem doesn't mean it also aligns with the vision of Go. There are many languages that already have these features, yet a lot of people prefer Go to those languages (this is just conjecture on my part, as I prefer Go over more advanced languages)

[networkimprov]

The reason for the try() proposal is that, year after year, ~5% of survey respondents identified error handling as a problem. Then, ironically, the community decided that proposal didn't align with the vision of Go :-)

Supporters, and detractors, count.

[beoran]

Yes, and I think it shows that Go now become so widely used that it will be difficult to have everyone agree on how to do error handling. Now try() is just a macro, but it's a one off macro, implemented by the language itself. I think it would be better for the language to have it's own macro system in stead, so everyone can define the error handing, and other similar things, like, e.g. cond() they like.

Go has a certain vision, but in the end practicallity often won out over purity. Look, e.g. to struct tags, or //go: pragmas. Even when hygienic, macros are not a "pure" solution, but they are definitely practical, moreso than ad hoc solutions like try().

If this issue is not accepted, then I will probably end up writing a preprocessor, so devising a syntax is useful anyway. So, in stead of arguing the social issues, I'd like to focus on that in stead. If we work together we might come up with a great syntax for this issue.

[Chillance]

@beoran I think that is well said and also why I posted my little quick code sample above. To get syntax and discussion started. We make the macro work the way we want it to work in Go. Possibly use inspiration from other languages, and also what we don't want to do from other languages. Think outside the box.

[beoran]

I think that first I'd like to think about the syntax for invoking a macro. From that we might then understand how macros should be defined.

1. I propose that the syntax for macro expansion is identical to that of a function call, without a marker, however, as usual, for functions, with a package prefix, but using '[]' in stead of () for the argument list. So a macro named Foo defined in a package named macro will be called as macro.Foo[arg1, arg2, ...]. This makes it easy to distinguish macro calls from function calls, and is backwards compatible as currently such a statement leads to a syntax error'.

2. A macro expansion may be called at the top level of a file, or inside a function. The expansion of the macro is an ast.Node which is inserted at the point of expansion. Macros are expanded at compile time, so their arguments must be fully determinable by the compiler at compile time.

3. As we can see from the several examples macro above, a macro should be able to take several arguments to be powerful enough to use namely:
   3.1. An expression, to allow for things like Try(). 3.2. A type. 3.3. An identifier or variable. 3.4. A constant string. 3.5. A constant numeral (int or floating point). 3.6. A block of code. 3.7 ... Anything else I forgot?

Please discuss! :)

[urandom]

How would you distinguish between macro expansion and map/slice indexing?

[beoran]

The arguments are comma separated, while slicing uses : . Only for a single integer argument there could be some confusion like macro.Foo[1]. But yes, maybe that could make it hard to implement it for the parer, so a better suggestion would be welcome.

[Chillance]

Not sure I like the [] usage to distinguish a macro from a normal function call. It might look like an array/map access and be a bit confusing. Especially if you are coming from php/javascript. :)

@beoran What are your thoughts on my idea of prepending with something like a colon like this:

    math.SomeFunction()
    :macro.SomeMacro()

to distinguish from a function call? Skimming the code, I kinda like the idea of something prepended like that as it makes it rather clear and easy to see what is a normal function call and a macro usage.

[Chillance]

Obviously doesn't have to be a colon, although I don't think this will be confused with labels. Maybe use |?

    math.SomeFunction()
    |macro.SomeMacro()

or ~?

    math.SomeFunction()
    ~macro.SomeMacro()

or \?

    math.SomeFunction()
    \macro.SomeMacro()