Macros by Example

[dwwoelfel]

I was in a conversation recently about the power of macros, and the use of syntactic abstraction in building simpler systems.

We quickly realized though: it’s tough to convey in a conversation what’s so special about macros. What can you do with macros that you couldn’t do with functions?

In this essay, we’ll use 2 examples, alongside some imaginary javascript syntax and lisp [1] to explore that question!

Note: This tutorial assumes you have a light understanding of lisp syntax. Go through this tutorial to brush up if you haven’t gotten to explore lisp yet.

Note: I’ve been meaning to write this for weeks, but was worried that it would be confusing. I am going to apologize now if that’s how you end up feeling when you read this. There may be a better way to explain it, but I needed to get this out of the head. If you have any feedback on how I could make this simpler, please let me know 🙂

[1] The specific language is Clojure, but anything done here can be done with any lisp

Example 1: nullthrows

With this example, let’s gain an intuition for when macros run and why that can be powerful.

Context

In any language with nulls, there’s a nullthrows abstraction: If some value evaluates to null, throw it.

Here’s how we could implement that as a function in javascript:

function nullthrows(result) {
  if (result === null || result === undefined) {
    throw new Error("uh oh");
  } 
  return result;
}

So if we run it, and it evaluates to null, we’ll throw an exception

nullthrows(getUser(db, 'billy'))
// if it's null, throw Exception

This works great…but there’s a problem. What would our stacktrace look like?

index.html:700 Uncaught Error: uh oh
    at nullthrows (index.html:700)
    at someStuff (index.html:1325)
    ...

When some value is null, the stacktrace won’t have much helpful information. It will say which line threw, but we’d have to do some digging each time to find out where the code was.

One way we can fix that, is to pass in a message argument

nullthrows(getUser(db, 'billy'), 'expected billy');

function nullthrows(result, message) {
  if (result === null || result === undefined) {
    throw new Error(`uh oh: ${message}`);
    ...

This could work…buut

Challenge

What if I told you: I don’t want to have to pass in a message.

Instead, when the source code that nullthrows wraps is specific enough, I’d be just as happy if the error printed the offending piece of code.

For example, with nullthrows(getUser(db, ‘billy')), nullthrows is wrapping the source code getUser(db, ‘billy’))

If the error printed out “Uh oh, this returned null: getUser(db, ‘billy’)”, it would be specific enough, and I wouldn’t need a custom error message.

Problem

Well, by the time nullthrows is run, getUser(db, ‘billy’) will be long gone: all the function will see is the evaluation of getUser(db, ‘billy’). Since the evaluation will be null, there’s not much information we can gain.

Javascript Solution

To actually capture and work on source code, we need a new kind of abstraction.

This would be some kind of function, that does two things:

1. It would take snippets of source code as input, and return new snippets of source code as output
2. This abstraction would be called at the build step, and replace the source code snippets that it takes in, with those new source code snippets.

Let’s say Javascript had that. instead of function nullthrows, we would have macro nullthrows. It could look something like this:

macro nullthrows(sourceCodeSnippet) {
  return `
    const result = ${sourceCodeSnippet}; 
    if (result === null || result === undefined) {
      throw new Error("Uh oh, this returned null:" + ${stringify(sourceCodeSnippet)});
    } else {
      return result;
    }
  `;
}

Here, the input would be the actual source code.

Whenever it’s called, we would replace that piece of code, the source code snippet that this abstraction generates.

For example, during the build step nullthrows(getUser(db, ‘billy’)) would be replaced with:

const res = getUser(db, 'billy'); 
if (res === null || res === undefined) {
  throw new Error("Uh oh, this failed:" + "getUser(db, 'billy')");
} else {
  return res;
}

Now, you might see some potential problems here:

Snippets are just text! It’s really had to programmatically add/remove/edit text without causing a bunch of syntax errors. Imagine if you wanted to change the source code, based on what it was — is it a functional call or a value? — there would be no way to tell with just text.

With javascript, you can work on the abstract syntax tree itself with babel transforms, but that will make the implementation quite different from what we want our code to be doing.

We really want to use some better data-structures to represent our code. Turns out, this idea isn’t new: there’s a whole family of languages — the lisp family — that wanted code to be represented with data-structures so we could read/generate code snippets more easily.

It seems like a lot of work just to make a built-in code-snippet-generator for a language, but let’s see what our challenge looks like if we use lisp’s approach:

Lisp solution

Since all code in lisp, are just lists, our lisp macro takes in a list of code, and returns a new list of code

We would write nil-throws like this:

(nil-throws (get-user "billy"))

The function variant would look like this:

(defn nil-throws [res]
  (if (nil? res)
    (throw "uh oh")
    res))

Now, I’m going to show you how the macro variant would look like: (don’t worry about a few of the symbols you’ll see, they’re all simple and I’ll explain them in just a few words below)

(defmacro nil-throws [form]
  `(let [result# ~form] ;; assign the evaluation of form to result#
    (if (nil? result#)
      (throw
        (ex-info "uh oh, we got nil!" {:form '~form})) ;; save form for inspection
      result#)))

Here’s how we can think about it:

1. Similar to how we wrote ` in javascript, the backtick here does the same thing: it says, hey, here’s the code I want to return, don’t evaluate it right away.

2. # is a handy way to generate some symbol, that won’t interfere with any other symbol when this code gets replaced in a different scope.

3. ~ is like our interpolation ${} in javascript, but for lists

4. ‘ is a way to say: hey, I want to treat something as a list, and don’t want to evaluate it

This would make it so when we write: (nil-throws (get-user db “billy”))

It would be replaced (~approximately) with:

(let [result# (get-user db "billy")]
  (if (nil? result#)
    (throw (ex-info "uh oh, we got nil!" {:form '(get-user db "billy")})) 
    result#))

Wow…we just wrote code that wrote more code…that’s pretty cool

Lessons learned so far

Macros take code as input, and return code as output. They run during the build step

Example 2: pipe syntax

Now, let’s explore the kind of power this can give us.

Context

The pipe operator is quite common in a bunch of languages.

It takes code that you would normally write like this:

createBill(addToCart(cart, updatePrice(item, 100)))

And let’s you invert the flow visually:

item |> updatePrice($$, 100) |> addToCart(cart, $$) |> createBill

Challenge

What if our language didn’t have this, and we wanted to implement it? Maybe we’d want our syntax to look like this:

|> [
  item, 
  updatePrice($$, 100), // updatePrice(item, 100)
  addToCart(cart, $$), // addToCart(cart, updatePrice(item, 100))
  createBill, // createbill(addToCart(cart, updatePrice(item, 100)))
]

Problem

Now, we could do this by implementing a pipe function:

pipe(item, (item) => updatePrice(item, 100))

But we would need to introduce anonymous functions, and the code would be less concise.

The only way to do this to spec, would be to change the syntax itself.

Javascript Solution

Now, with our imaginary javascript syntax, we could write something like this:

macro |> (listOfForms) {
  return listOfForms.reduce(
    (lastForm, thisForm) => {
      if (isFunctionalCall(thisForm)) {
        return `
          let $$ = ${lastForm};
          ${thisForm};
        `;
      } else {
        return `${callFunction(thisForm, lastForm)}`;
      };
  });
}

Here, would start with a list of the forms, un-evaluated:

[item, updatePrice($$, 100), addToCart(cart, $$), createBill]

Reduce would start with the arguments lastForm = item, thisForm = updatePrice($$, 100)

Now, we would need a way to know: is this a form of a function call updatePrice($$, 100), or just a function: createBill

If it’s a function call, we can create new code, which defines $$ as the last form, and evaluate the function call within that scope.

Otherwise, we can create new code, which calls that function with the last form.

Lisp Solution

What we want would be something like this:

(|> item
    (update-price $$ 100)
    (add-to-cart cart $$)
    create-bill)

And our macro could look like this:

(defmacro |> [form & forms]
  (reduce
    (fn [last-v form]
      (if (seq? form) ;; am I being called: (update-price $$ 100)
        `(let [~(symbol "$$") ~last-v]
           ~form)
        `(~form ~last-v))) ;; or am I just a function: create-bill
    form
    forms))

Our lisp code would follow the same idea as our Javascript solution. Let’s see the code we didn’t have to write:

(create-bill 
  (let [$$ (let [$$ item]
             (update-price $$ 100))]
    (add-to-cart cart $$)))

…that’s pretty cool. We get the best of both worlds: efficient, well-erroring code that’s short to write and — most importantly — clear to read.

Lessons learned so far

Macros let you change the language itself. You can transform code and change the syntax.

Conclusion

Macros let you change your language to suit your problem. This is extremely powerful: You can build up your language so you can express your problem as clearly as possible. This makes your code more concise and simple, which in turn makes your system more malleable.

At the same time, macros…change the language itself. There are few moments where this level of abstraction is warranted, so if you use them when simpler abstractions would do, you risk adding unnecessary complexity.

Yet… when they are warranted, having them as an option can change the game.

Further Reading and Practice

If this got you interested, here’s some reading and practice you may enjoy:

• Read Norvig’s Paradgims of AI Programming, and do the homework
• Read Clojure for The Brave and True’s Macro Guide, and do the homework
• Look into how Clojure uses macros to define the language itself (when, and, or, etc)
• Write async await syntax for promises

Credits

Shoutout to Daniel Woelfel: I saw his nil-throws macro years back when we worked together, and it opened my eyes to the power of syntactic abstraction.

Thanks to Sean Grove, Daniel Woelfel, Martin Raison, Alex Reichert, Mark Shlick for a beautifully deep review of this essay.

Thanks to Paul McJones, perfunctory and tzs, for their feedback on the code examples.

{"source":"medium","postId":"6ddbc8f3d93b","publishedDate":1571762653033}