For our use case in quote, Extend<TokenTree> is worse because it involves taking the TokenStream we parse (which is the impl IntoIterator<Item = TokenTree> passed into the Extend impl), iterating it to get a sequence of TokenTrees, and converting those TokenTrees back to a TokenStream. That's circuitous! We had a TokenStream already.
This PR tweaks quote to use Extend<TokenStream> instead, which skips the needless round trip through TokenTree.
This optimization is also important for https://github.com/dtolnay/proc-macro2/pull/320. If the TokenStream holds its contents as a CallSite-spanned string buffer instead of trees, using Extend<TokenTree> forces proc-macro2 to reparse that buffered representation in order to iterate its tokens, whereas Extend<TokenStream> allows the whole buffer to be efficiently appended to the lhs all at once.
proc_macro::TokenStream
provides bothExtend<TokenTree>
andExtend<TokenStream>
. They are implemented in libproc_macro as:For our use case in quote,
Extend<TokenTree>
is worse because it involves taking the TokenStream we parse (which is theimpl IntoIterator<Item = TokenTree>
passed into the Extend impl), iterating it to get a sequence of TokenTrees, and converting those TokenTrees back to a TokenStream. That's circuitous! We had a TokenStream already.This PR tweaks quote to use
Extend<TokenStream>
instead, which skips the needless round trip through TokenTree.This optimization is also important for https://github.com/dtolnay/proc-macro2/pull/320. If the TokenStream holds its contents as a CallSite-spanned string buffer instead of trees, using
Extend<TokenTree>
forces proc-macro2 to reparse that buffered representation in order to iterate its tokens, whereasExtend<TokenStream>
allows the whole buffer to be efficiently appended to the lhs all at once.