dtolnay / quote

Rust quasi-quoting
Apache License 2.0
1.32k stars 90 forks source link

Optimize parse+extend to make only one proc macro bridge call #203

Closed dtolnay closed 2 years ago

dtolnay commented 2 years ago

proc_macro::TokenStream provides both Extend<TokenTree> and Extend<TokenStream>. They are implemented in libproc_macro as:

impl Extend<TokenTree> for TokenStream {
    fn extend<I: IntoIterator<Item = TokenTree>>(&mut self, trees: I) {
        self.extend(trees.into_iter().map(TokenStream::from));
    }
}

impl Extend<TokenStream> for TokenStream {
    fn extend<I: IntoIterator<Item = TokenStream>>(&mut self, streams: I) {
        *self = iter::once(mem::replace(self, Self::new())).chain(streams).collect();
    }
}

For our use case in quote, Extend<TokenTree> is worse because it involves taking the TokenStream we parse (which is the impl IntoIterator<Item = TokenTree> passed into the Extend impl), iterating it to get a sequence of TokenTrees, and converting those TokenTrees back to a TokenStream. That's circuitous! We had a TokenStream already.

This PR tweaks quote to use Extend<TokenStream> instead, which skips the needless round trip through TokenTree.

This optimization is also important for https://github.com/dtolnay/proc-macro2/pull/320. If the TokenStream holds its contents as a CallSite-spanned string buffer instead of trees, using Extend<TokenTree> forces proc-macro2 to reparse that buffered representation in order to iterate its tokens, whereas Extend<TokenStream> allows the whole buffer to be efficiently appended to the lhs all at once.