Pre-compile macro output as `include!`

[overlookmotel]

@rzvxa and I have been discussing various ways to make macros cheaper on compile time, for example the #[ast] macro.

I had another idea in this vein. How about this:

• For each type marked #[ast], ast_tools would output a separate file.
• If the type is in oxc_ast/src/ast/js.rs, then the codegen would write to oxc_ast/.macro_expansions/src/ast/js.rs/<line number>.rs.
• Content of those files would be the complete output after processing (i.e. alter the type def in same way #[ast] macro does now).
• #[ast] macro would expand to:

include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));

Example

Original type def:

// oxc_ast/src/ast/js.rs line 123
#[ast]
enum Foo<'a> {
    Bar(Box<'a, Bar>),
    Qux(Box<'a, Qux>),
    @inherits Donkey,
}

ast_tools writes to file:

// oxc_ast/.macro_expansions/src/ast/js.rs/123.rs
#[repr(C, u8)]
enum Foo<'a> {
    Bar(Box<'a, Bar>) = 0,
    Qux(Box<'a, Qux>) = 1,
    BigDonkey(Box<'a, BigDonkey>) = 2,
    SmallDonkey(Box<'a, SmallDonkey>) = 3,
}

#[ast] macro's implementation is very simple indeed:

#[proc_macro_attribute]
pub fn ast(_: TokenStream, _: TokenStream) -> TokenStream {
    // No `syn` here!
    r#"include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));"#.parse().unwrap()
}

Spans

I've not tested this out, so I don't know for sure it'll work. In particular, I wonder if spans/hygiene will get mashed up so Rust Analyser may lose "jump to definition" etc.

Maybe we can work around that by outputting a macro_rules! macro. e.g. ast_tools writes this to the file:

// oxc_ast/.macro_expansions/src/ast/js.rs/123.rs
macro_rules! macro_123 {
    (
        #[ast]
        enum $Foo:ty $life:lifetime {
            $Bar:ident($Box1:ident<'a, $Bar:ident>),
            $Qux:ident($Box2:ident<'a, $Qux:ident>),
            @$inherits:ident $Donkey:ident,
        }
    ) => {
        #[repr(C, u8)]
        enum $Foo $life {
            $Bar($Box1<'a, $Bar>) = 0,
            $Qux($Box2<'a, $Qux>) = 1,
            macro_123!(@$inherits $Donkey,)
        }
    }

    (@inherits Donkey,) => {
        BigDonkey(Box<'a, BigDonkey>) = 2,
        SmallDonkey(Box<'a, SmallDonkey>) = 3,
    }
}

Then #[ast] macro would output a token stream:

include!(concat!(".macro_expansions/", file!(), "/", line!(), ".rs")));
macro_123! {
<original input TokenStream>
};

That's a bit more involved, but still can be done just by concatenating TokenStreams, which is much cheaper than syn.

Actually I'm not sure how to get proc macro to output macro_123!, as the proc macro doesn't know the line number. Maybe proc macro has to hash the input stream and use that as the UID instead of line number. Hashing with FxHash is fairly cheap.

Or maybe it searches through the input TokenStream for the token struct or enum, and then the UID is the token that follows (which will be the type name).

What do you think, @rzvxa?

[rzvxa]

This one is fairly complex. Give me some time to digest the ideas. But I'm kind of on board with the "include" already; Just need some time to process it and come back with comments.

Edit

I'm mostly worried about the manual import of these files. We should make them internal and unexposable to the users somehow. Maybe we can have them outside of the src directory so nobody can use them as valid Rust modules. But having source files outside of src makes me worried about crate and how the publish works.