Hygienic Macros

[renatocf]

Encouraged by our Generics discussion in #17, let's talk about our previously chosen metaprogramming technique: hygienic macros.

[renatocf]

Marking as opened to #18

[igorbonadio]

I was thinking that a macro is like a 'compile time function' #22, but the difference is:

• compile time functions' arguments are values (and they are evaluated by eager evaluation)
• macros' arguments are programs. We can convert those 'programs' to ASTs which is a data structure and then we can manipulate them inside the macro.

For example:

# This macro receives an AST that represents a class template like:
#
# class Box<T>
#   var element: T
#   def get() -> (value: T)
#     value = element
#   end
# end
#
# And generates a classe like:
#
# class BoxInt
#   var element: Int
#   def get() -> (value: Int)
#     value = element
#   end
# end
macro class_template(klass: ClassTemplateAst, type: Type) -> (ast: ClassAst)
  let basename = klass.basename() # => Box
  let typename = type.name() # => Int
  klass.replace_name(name: basename + typename) # => BoxInt
  klass.replace_template(type: T, by: Int)
  ast = klass.build() # returns:
                      # class BoxInt
                      #   var element: Int
                      #   def get() -> (value: Int)
                      #     value = element
                      #   end
                      # end
end

Obs1: All AST functions (klass.*, type.name) are 'compile time function' #22.

We could call this macro as:

class_template {
  class Box<T>
    var element: T
    def get() -> (value: T)
      value = element
    end
  end;
  Int
}

Obs2: I think we have to differenciate function call of macro_call... My proposal is that we use macro_name{p1; p2; p3}

Obs3: Positron compiler should have a sugar syntax to express generics. For example, when a programmer write a template class like class Name<T> ... the compiler knows that to build a new class using this template it should call class_template macro.

[igorbonadio]

Other example:

macro attr_reader(id: IdentifierAst, type: Type) -> (ast: AST)
  var variable = Variable.new(name: id.name(), type: Type) # => var x: Int
  var body = Attribution.new(to: "v", value: id.name())    # => value = x
  var getter = Method.new(name: id.value(), input: [], output: [v: Type], body) # => def x() -> (value: Int)
                                                                               #      value = x
                                                                               #    end
  ast = getter
end

class User
  attr_reader{name; String}
  attr_reader{friends; Vector<User>}
end

user = User.new
user.name()
user.friends()

What do you think about it?

[igorbonadio]

We could simplify it using quotes:

macro attr_reader(id: IdentifierAst, type: Type) -> (ast: AST)
  ast =  `
  var #{id}: #{type}
  def #{id}() -> (value: #{type})
    value = #{id}
  end
  `
end

Where #{} is like a string interpolation... but for programs. This quotes are like programs inside programs.

But I'm not sure if we need the return type of a macro...

[vinivendra]

Wow, ok! These ideas go way beyond what I initially though the macros would be like. I really like them :)

I like the idea of having "functions" that manipulate the ASTs, that seems to give a lot of power to this metaprogramming strategy. I also like the syntactic sugar that the quotes mechanism provides.

I don't usually use macros in C because it can be a pain in the ass to debug them, since it's pretty much all just text replacement. However, with this approach, I think the type checking provided by the compiler helps a lot.

[Kazuo256]

I also really like this kind of feature. It would be much more syntax-highlight friendly if we could avoid using quotes for manipulating code strings, though. How could we do that?

[vinivendra]

I don't really like the way we call macros, though. It seems a bit unclear, and maybe too troublesome to be something we would like to insert in the middle of other normal code. For instance, would we have to call the class_template macro on Box every time we wanted to create a Box with a new type?

[vinivendra]

(I'm brainstorming here, bear with me.)

Maybe there's a way to automate some of these macros, a bit like regexes: they run through the entire code and, every time they find an AST that matches what they want, they change it accordingly.

This might be good to allow easy ways of using macros anywhere in the code. For instance, one might create a macro to transform every variable fitting a certain description into a getter and a setter, or every class followed by <Type> into an instance of that template, etc.

Also, we might want to allow macros to create new keywords. As a simple example, one might create the keyword until, which when found in an AST gets transformed into a while not AST; or unless into if not. Perhaps someone else can get more creative and give better examples :)

[vinivendra]

@Kazuo256, you're probably right. Maybe if we used something that resembled other languages, it would be more universally accepted:

ast = {
    var {id}: {type}
    def {id}() -> (value: {type})
        value = {id}
    end
}

I used {} only because I don't think we use it anywhere else. Also, I removed the #s because they turned everything into comments.

[Kazuo256]

The idea of using pattern matching might be good. I also remember a tool called OpenC++ that did a few nice metaprogramming things for C++, but it probably does not work anymore.

[igorbonadio]

I think macro is our general metaprogramming technique. We could have specialized syntax for:

• Generic #17 :

Without sugar syntax:

macro class_template(template: ClassAST, type: Type)
  #...
end

class_template {
  class Box<T>
    var element: T
    def get() -> (value: T)
      value = element
    end
  end;
  Int
}

class Main
  def main() -> ()
    let x = BoxInt.new
  end
end

With sugar syntax:

class Box<T>
  var element: T
  def get() -> (value: T)
    value = element
  end
end

class Main
  def main() -> ()
    let x = Box<Int>.new
  end
end

• Annotation #23 :

Without sugar syntax:

class TestSuite
  test {
    def test_something() -> ()
      # ...
    end
  }
end

With sugar syntax:

class TestSuite
  @test
  def test_something() -> ()
    # ...
  end
end

[igorbonadio]

@vinivendra , we can implement your example of regex with something like:

match_macro { 
  some_pattern,
  some_expr,
  # your program here
}

Where match_macro will look for some pattern and change it to some_expr.

And again, we could have a sugar syntax for it.

[igorbonadio]

@Kazuo256 I think we can highlight quoted code too... And maybe it is better to just allow interpolation of nodes of an AST. For example:

`
class #{prefix}Blah # error... prefix is not a node
end

class #{prefix.join("Blah")} # ok
end
`

[igorbonadio]

Now, about syntax. I agree that the syntax is not good... We can think about it together.

We need a syntax to:

• define a macro
• call a macro
• quote code
• (maybe) unquote code

[vinivendra]

Ok then, let's think about it then.

If we really take macros to be like AST-regexes, it would probably make sense to store them in their own files. This is because they probably should be applied to all the code across a few files, instead of being "called" like they might be in C. For illustration's sake (it's way too early to define this permanently), one might have a "myPositronMacros.macro" file in some parent folder that defines a series of macros to be applied to the files in that folder.

In this situation, it makes sense to have a good way of declaring macros, but there's no need to create a way of calling them (since they get run automatically). Also, since there would be no mixing macros with code, we would be able to create a syntax similar to normal functions, like the sytaxes suggested a while back.

Alternatively, one might want to "call" a specific set of macros to use for each file (I can see how that would avoid unexpected behavior and conflicts). In this sense, we could just "include" the desired macros files like we might do with other positron files.

If we did decide to adopt a strategy like this one, do you think C-like macros would still be useful (macros created inside the file itself and then called as needed throughout)?

[igorbonadio]

I think users should choose when and where to call a macro. So I think we should not use AST-regexes-like macros as default.

My suggestions are:

• function-like macros These macros are like normal functions, but they receive N expressions (instead of N values).
• annotation-like macros Are like function-like macros, but they receive only one expression: the next expression.
• generics Like java generics or c++ templates

Examples:

class VoightKampff
  def create() -> (being: Being)
    being = Decard.new
  end

  @test # annotation that manipulates the following method
  def test_name() -> ()
    being = create()
    assert{being.name(); "Decard"} # function-like macro
  end

  @test
  def test_replicant() -> ()
    being = create()
    assert{being.is_replicante(); false}
  end  
end

or

@testcase # it changes all test_* methods... something like @vinivendra's regex-like macros.
class VoightKampff
  def create() -> (being: Being)
    being = Decard.new
  end

  def test_name() -> ()
    being = create()
    assert{being.name(); "Decard"}
  end

  def test_replicant() -> ()
    being = create()
    assert{being.is_replicant(); false}
  end  
end

We could add an annotation at the begining of the program too...

Now I will comment my own examples hahahaha:

• I like annotation syntax. It is clean and clear.
• I didn't write example of generics but I like generics with <T>
• I don't like the syntax of function-like macros... I think we have to differentiate function call of macro call... but {} and ; doesn't seem good options... Maybe:

macro!(expr1, expr2) # it is not so different
macro[expr1, expr2] # array???
macro{expr1, expr2} # without ;
macro expr1, expr2;
macro expr1; expr2;

@vinivendra rose a question about sourcefiles. I think it is very important, and we should open an issue about it... Not only about source files... but also about program organization... will we use modules? packages? etc...

[vinivendra]

I like the idea of using them only for the next expression or for a series of expressions is interesting and might solve all cases, though I don't like the @ notation.

I think macros that need to encapsulate several lines of code might do so the same way as other language constructs:

@doSomethingAwesome
    object.method()
    object.otherMethod()
end

And I think this way we could simply do something like adding macro calls at the start of the file to apply them to the whole file.

Also, I don't think we need a special case for generics. If we create a macro that applies a string regex to a class name (recognizing classes with names like class<type>), we can simply apply that macro to the whole file and it's done. This is the kind of "language improvement" we mentioned a while ago that I think makes sense: using our main metaprogramming technique to add a new feature to the language itself. Also, customizing the language to better fit one's needs.

[igorbonadio]

Ok. But how whould you express a test case (like the one I showed)?

Now, I think the generics syntax is just a sugar syntax... If we can write expression with class<type> I dont think we need to surround it with another macro.

[vinivendra]

Hmmm I'm not sure... I don't like using @, but all the alternatives I can think of are worse.

As for generics, I think I expressed myself poorly. I was thinking that generics would normally just be a normal class with a funny name involving < and >. When the generics macro processed classes with funny names like those, it would transform those classes in what we want them to be. This way there would only be one macro involved, and it could be a normal macro. No need for special cases for macros, no need for special syntax sugar for generics, I think it simplifies the language.

[vinivendra]

Ok, I think our vacations have gone on long enough :) How about we pick it up from this issue?