Procedural macro reimplementation of `quote!` to resolve longstanding limitations

[dtolnay]

Originally filed as https://github.com/dtolnay/quote/issues/82 but I would like this to begin its life as a separate library.

The current macro_rules-based quote macro has the limitation that duplicate interpolations inside of a repetition are not allowed. quote! { #a #a } works but quote! { #(#a #a)* } does not work. The reason boils down to macro_rules macros having no way to determine that two identifiers are equal.

Some background on the quote macro:

• quote! { #a #a } expands to:

  {
      let mut _s = TokenStream::new();
      let _span = Span::call_site();
      ToTokens::to_tokens(&a, &mut _s);
      ToTokens::to_tokens(&a, &mut _s);
      _s
  }

• quote! { #(#a)* } expands to:

  {
      let mut _s = TokenStream::new();
      let _span = Span::call_site();
      for a in a {
          ToTokens::to_tokens(&a, &mut _s);
      }
      _s
  }

• quote! { #(#a #b)* } expands to:

  {
      let mut _s = TokenStream::new();
      let _span = Span::call_site();
      for (a, b) in a.into_iter().zip(b) {
          ToTokens::to_tokens(&a, &mut _s);
          ToTokens::to_tokens(&b, &mut _s);
      }
      _s
  }

• quote! { #(#a #a)* } expands to:

  {
      let mut _s = TokenStream::new();
      let _span = Span::call_site();
      for (a, a) in a.into_iter().zip(a) {
          ToTokens::to_tokens(&a, &mut _s);
          ToTokens::to_tokens(&a, &mut _s);
      }
      _s
  }

That last expansion is illegal so the quote invocation fails.

error[E0416]: identifier `a` is bound more than once in the same pattern
  --> src/main.rs:12:17
   |
12 |         for (a, a) in a.into_iter().zip(a) {
   |                 ^ used in a pattern more than once

In an implementation as a procedural macro we would want that invocation to expand instead as:

{
    let mut _s = TokenStream::new();
    let _span = Span::call_site();
    for a in a {
        ToTokens::to_tokens(&a, &mut _s);
        ToTokens::to_tokens(&a, &mut _s);
    }
    _s
}

[WildCryptoFox]

I do agree we need a procedural macro alternative. However macro_rules can determine if two tokens are identical. Playground

I discovered this trick a few years ago and created a lisp-style language using it. Unfortunately, I can't find that code so I reconstructed this. Edit: I found it!

macro_rules! tt_eq {
    // remove the extra block to run this in item-context
    ($a:tt $b:tt) => {{
        macro_rules! __tt_eq_internal {
            ($a) => { true };
            ($b) => { false };
        }
        __tt_eq_internal!($b)
    }}
}

macro_rules! tt_match {
    // wrap the result in extra block to run in expression-context 
    ($left:tt in $( ($($right:tt)|*) => $result:tt );+ $(; _ => $default:tt)* $(;)* ) => {
        macro_rules! __tt_match_internal {
            $( $( ($right) => $result; )* )+
            $( ($left) => $default )*
        }
        __tt_match_internal!{ $left }
    }
}

tt_match!{ c in
    (a | b | c | d) => {
        struct Foo;
    };
    (e) => {
        struct Bar;
    };
    _ => {
        struct Baz;
    }
}

fn main() {
    let _foo = Foo;
    assert!(tt_eq!(a a));
    assert!(!tt_eq!(a b));
}

[Goncalerta]

I created the repository proc-quote in order to work on this new library.

[Goncalerta]

Besides the already existent #ident and #( #iter )* patterns, I'm adding a new one in my crate: #{ expr }.

I created this because sometimes I feel the need to call a function or access an inner element of a struct inside quote!, which is not normally allowed. This means I'd need to create a new variable outside the macro just to use it inside.

With #{ my_struct.inner } or #{ call_function() } this issue is solved. The #{ ... } pattern accepts any block of code that evaluates to ToTokens, even though it is supposed to be used in simpler cases, whereas more complex ones it can still be done outside the macro.

@dtolnay what are your thoughts on this feature?

[dtolnay]

I strongly prefer not to have that feature and I believe the library is worse off for providing it.

Some existing discussion in https://github.com/dtolnay/quote/pull/88 and numerous other threads on the quote repo.

[Goncalerta]

I'm sorry, it was silly on my part not to check if this had been proposed before. It seemed such a simple and straightforward addition that I felt compelled to add it.

Thanks for pointing me to the existing discussion. I will remove this functionality from my crate.

[dtolnay]

Thanks! To be clear, plenty of people think I am wrong on this.

Let me know once your existing TODO items are either implemented or have issues filed to track them, and the crate has been published to crates.io, and then we can close out this thread and mark it off the list. :)

[eddyb]

For the record, this is implementation of the (unstable) proc_macro::quote!.

It's very simple, but it could be used as a starting point for someone else.

However, it can't just be copied verbatim to a proc macro, as it uses the unstable Span::def_site() and it relies on a different part of the compiler special-casing proc_macro::quote! so that the paths it generates resolve in proc_macro.

[Goncalerta]

It is up! https://crates.io/crates/proc-quote

:)

[dtolnay]

Nicely done! I will go ahead and close this thread. I added a link to your crate in the readme.

If you'd like to explore ways to address the other limitation (involving non-repeating variables inside of a repeating block) I would be interested to see what you come up with.

[WildCryptoFox]

If the only thing blocking this in quote is the incorrect assumption that macro_rules cannot filter out duplicates, here you go.

https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=0bb9df8fd1d555d84c1720e66f8a044b

macro_rules! tt_match {
    ($left:tt in
        $( ($($right:tt)|*) => $cb:tt!$cba:tt );+
        $(; _ => $de:tt!$fault:tt)?
        $(;)?
    ) => {{
        macro_rules! __tt_match_internal {
            $( $( ($right) => { $cb!$cba }; )* )+
            $( ($left) => { $de!$fault } )*
        }
        __tt_match_internal!{ $left }
    }}
}

macro_rules! uniq {
    (($($xs:tt)*) () $cb:tt!($($tt:tt)*)) => {
        $cb!( $($tt:tt)* $($xs)* )
    };

    (($($xs:tt)*) ($y:tt $($ys:tt)*) $cb:tt!$cba:tt) => {
        tt_match!{
            $y in
                ($($xs)|*) => uniq!(($($xs)*) ($($ys)*) $cb!$cba);
                _ => uniq!(($($xs)* $y) ($($ys)*) $cb!$cba);
        }
    };
}

fn main() {
    println!("{}", uniq!(() (a a b c c) stringify!()));
}

[dtolnay]

Thanks!

The relevant performance comparison would be between proc-quote vs quote-with-uniq-trick, not quote without uniq trick..

[WildCryptoFox]

@dtolnay Naturally. I already removed the note for speed as the remaining limitation regarding mixing non-repeating with repeating args remains unresolved and may be the selling point for proc-quote that this would have been.