Tracking issue for procedural macros and "hygiene 2.0"

alexcrichton commented 5 years ago

This is intended to be a tracking issue for enabling procedural macros to have "more hygiene" than the copy/paste hygiene they have today. This is a very vague tracking issue and there's a very long and storied history to this. The intention here though is to create a fresh tracking issue to collect the current state of play, in hopes that we can close some of the much older and very long issues which contain lots of outdated information.

petrochenkov commented 5 years ago

To give some background, feature(proc_macro_hygiene) tracked by this issue was merged from multiple features (proc_macro_mod, proc_macro_expr, proc_macro_non_items, proc_macro_gen) in https://github.com/rust-lang/rust/pull/52121.

So, it's pretty heterogeneous and is not even entirely about hygiene.

petrochenkov commented 5 years ago

Copypasting what I wrote in https://github.com/rust-lang/rust/pull/52121:

None of these features is blocked on hygiene, except for proc_macro_gen.

proc_macro_mod is, IIRC, about expanding inner attributes and expanding non-inline modules (mod foo;). I'm not sure why #[my_macro] mod m { ... } with outer attribute and inline module is still gated, I'm not aware of any reasons to not stabilize it.
proc_macro_expr/proc_macro_non_items have no reasons to be gated as well and can be stabilized after some implementation audit. Proc macros have well-defined "copy-paste" Span::call_site() hygiene, this includes "unhygienic" use of local variables. IIRC, @nrc assumed that additional feature gates would prevent interactions of that copy-paste hygiene with local variables, but that's not true - you can easily expand an item and inject tokens from the outside into its body (e.g. https://github.com/rust-lang/rust/issues/50050), causing exactly the same interactions.

What is really blocked on hygiene is Span::def_site() and decl_macro (which is also blocked on a ton of other stuff).

petrochenkov commented 5 years ago

proc_macro_gen is more about interactions between Span::call_site hygiene (entirely unhygienic) and macro_rules hygiene (unhygienic except for local variables and labels).

CodeSandwich commented 5 years ago

This looks like the right branch for this question. Sorry about copy-paste, but I really need to know.

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

petrochenkov commented 5 years ago

@CodeSandwich This may be a regression from https://github.com/rust-lang/rust/pull/52319 perhaps?

cc @tinco

petrochenkov commented 5 years ago

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code.

From what I remember, this is an open question. The transformation mod m; -> mod m { ... } is an expansion of sorts, and we may treat it both as an early expansion (like e.g. #[cfg], which is also expanded before passing the item to attribute-like macros) and as a late expansion (the item is not expanded before being passed to macros).

The "early expansion behavior" looks preferable to me, and this change in behavior looks like a bug rather than a fix.

tinco commented 5 years ago

I lack some context, so I can't be certain, what exactly is "process"? If it relies on rustc's pretty printing, then yes this would be a consequence of #52319, and it should be simple to fix by changing the invocation I think. #52319 does two things:

Add information to the AST (a single bool, tracking if a mod is inline)
Make pretty print no longer print the external file if that bool is false

If you need the expanded file, then that should be made explicit in the invocation of the pretty printer.

petrochenkov commented 5 years ago

Yes, the change is likely due to the pretty-printing change. So it can be fixed either by reverting the change, or making it more fine-grained so it prints the contents regardless of the inline flag if pretty-printing is done "for proc macros" (whatever that means).

tinco commented 5 years ago

If you can point me at the line of code that invokes the pretty printer, I can suggest a fix.

CodeSandwich commented 5 years ago

By "processing" I mean parsing TokenStream with Syn to an Item, modifying its content and converting it back to TokenStream. The precise spot of parsing is here and content traversal is there. I haven't dug into implementation of Syn yet, so I can't tell for sure if it uses pretty printing. I've set nightly feature in Syn, which as far as I know skips step of printing and parsing the string again.

alexcrichton commented 5 years ago

I personally feel that what a proc macro receives as input tokens when attached to a module is an open questions, as @petrochenkov said. I do not think that one answer is clearly correct over another (mod foo; vs mod foo { ... }) at this time. That's why it's unstable!

eddyb commented 5 years ago

I'm confused why proc macros expanding to macro_rules! definitions is gated. (especially when the new macros are only invoked by other expanded items)

petrochenkov commented 5 years ago

@eddyb Well, the answer is "not enough confidence in correctness of hygiene interactions between transparent (procedural on stable) and semi-transparent (macro_rules) macros".

I hoped to at least get rid of the "context transplantation trick" before stabilizing it, but got distracted by the edition stuff.

It generally works though, I'm not aware of any specific bugs manifesting in normal cases.

petrochenkov commented 5 years ago

$crate in generated macros concerns me in particular. (I don't remember whether it works correctly now, need to test.)

dbeckwith commented 5 years ago

rustc pointed me to this issue because I tried to use a proc macro in a type context (procedural macros cannot be expanded to types), but I don't see anything about this usage in the discussion. Is this the right issue to track that feature?

jebrosen commented 5 years ago

@dbeckwith proc_macro_non_items was originally introduced in #50120, and I renamed it to be part of proc_macro_hygiene in #52121. I reviewed several of the discussions on both PRs and discussions that led to them and can't find any actual discussion of expanding types in particular. There has been some disagreement on exactly what should still be gated or not and why, but mostly with respect to statements and expressions.

@alexcrichton -- do you remember a reason for gating expansion to types, other than "types are not items"? @nrc -- do your concerns from https://github.com/rust-lang/rust/pull/52121#issuecomment-422311424 also apply to types, or only to expressions, statements, and presumably patterns?

petrochenkov commented 5 years ago

I'm not aware of any issues preventing stabilization of fn-like macros expanding into types.

alexcrichton commented 5 years ago

@jebrosen IIRC it's just "types weren't items" at the time, so if others agree seems like we could stabilize it!

dtolnay commented 5 years ago

Since this tracking issue touches on procedural macros involving mod and spans -- I filed https://github.com/rust-lang/rust/issues/58818 which seems to involve mod and spans. Basically the following two currently behave differently, which seems wrong:

mod parent {
    mod child;
}

mod parent {
    noop! { mod child; }
}

where noop! is a procedural macro implemented as:

#[proc_macro]
pub fn noop(input: TokenStream) -> TokenStream {
    input
}

awygle commented 5 years ago

Is there a plan to get function-like macros to work in expression or statement positions? From looking at https://github.com/rust-lang/rust/pull/52121#issuecomment-422311424, I understand that there are hygiene concerns, but is there a plan to either resolve or dispel those concerns?

jebrosen commented 5 years ago

Full list of what the proc_macro_hygiene feature controls (see ~~src/libsyntax/ext/expand.rs~~ src/librustc_expand/expand.rs):

Applying an attribute proc_macro to:
- Any input containing non-inline modules (mod x;)
- Gated because of unclear or undecided expansion behavior of non-inline modules https://github.com/rust-lang/rust/issues/54727#issuecomment-426192109
- A statement or an expression
- Aside from the expansion concerns below, I vaguely remember a question along the lines of "how much of the expression the macro applies to" in something like
```
#[expr_macro] a / b;
```

Update (2019-10-02): Link to relevant stabilization PRs. Update (2020-01-31): Update links and stabilization statuses. Update (2020-06-18): Reworked the list entirely, since proc_macro application and expansion is now stabilized in most positions.

SergioBenitez commented 5 years ago

I believe we should stabilize almost all of proc_macro_hygiene with the exception of the following (for the reasons outlined in @jebrosen's post above):

Applying attributes to non-inline modules.
Applying macros to expressions.
Expansion to macro definitions containing $crate.
Expansion to patterns.

In particular, we should stabilize the following:

Applying attributes to inline modules.
Applying attributes to statements.
Expansion to types, expressions, statements, and macro definitions without $crate.

The first two cases have clear semantics with no notable concerns for stabilization; their lack of stabilization seems to be largely due to a lack of demand. The lattermost case is more interesting: there are valid concerns regarding hygiene with expansion to some of these kinds, in particular statements and expressions, but they are largely resolvable post-stabilization via hygiene defaults in external libraries (i.e, syn and quote). What's more, the gated functionality is already possible today (i.e, proc-macro-hack): the instability of the lattermost case is not preventing what it aims to, and as such, does not benefit from being unstable. As a result, stabilization only makes what is possible today more convenient.

Again, I believe we should stabilize almost all of proc_macro_hygiene. This is the final feature gate preventing Rocket from compiling on stable.

petrochenkov commented 5 years ago

Some status update.

Macro expansion infrastructure is being steadily simplified and bug-fixed right now, filling in the "20%" part of the implementation. I want to complete some of this infrastructural work before further increasing the stable surface.

The current blocker is https://github.com/rust-lang/rust/pull/63468 worked on by @c410-f3r. After this PR lands I'm going to rewrite something called invocation collector, and after that we'll be able to start stabilizing confidently.

Everything in https://github.com/rust-lang/rust/issues/54727#issuecomment-523307027 is stabilize-able, with exception of attributes on expressions/statements which have a number of unresolved questions.

petrochenkov commented 5 years ago

Regarding the timing - I'd say vaguely this year. Note, that these features are not on the roadmap, so we are unlikely to get some quick-and-dirty stabilizations like Macros 1.1.

STATUS UPDATE: Due to a job change I have several times less time to work on this, but the progress is still happening.

SergioBenitez commented 5 years ago

@petrochenkov

Everything in #54727 (comment) is stabilize-able, with exception of attributes on expressions/statements which have a number of unresolved questions.

Are you saying that everything in my comment is stabilize-able now or after the changes you're hoping to make? I am advocating for stabilization of the subset of features now on the basis presented in my comment, in particular, that this functionality is largely already possible today and thus stabilization does not significantly expose a greater surface.

petrochenkov commented 5 years ago

@SergioBenitez Some subset can be stabilized right now, I've opened https://github.com/rust-lang/rust/pull/63931 for it.

Proc macros expanding into macro_rules can be stabilized right now too, but for a different reason. Unfortunately they were never gated in derive macros (and we couldn't do that during Macro 1.2 stabilization as well due to some real breakage), so people in practice just use derives to ignore the feature gate and generate them. So, if we ever change some hygiene rules in them it will affect people to the same extent anyway.

I'll make a separate stabilization PR for them, I need to check a couple of things before that.

petrochenkov commented 5 years ago

Stabilization PR for proc macros generating macro_rules - https://github.com/rust-lang/rust/pull/64035.

SergioBenitez commented 5 years ago

@petrochenkov Given the existence of proc-macro-hack and proc-macro-nested, why not also stabilize expansion to expressions? This would be the final blocker to Rocket on stable.

petrochenkov commented 5 years ago

why not also stabilize expansion to expressions?

https://github.com/rust-lang/rust/pull/63931#issuecomment-526367570

(I'd personally be ready to stabilize after some implementation audit and gating of sub-features with known issues, as I already said back in October - https://github.com/rust-lang/rust/issues/54727#issuecomment-426008711.)

petrochenkov commented 5 years ago

@SergioBenitez

This would be the final blocker to Rocket on stable.

By the way, what prevents Rocket from using proc-macro-hack for migrating to stable? What are the cases in which unstable macros are qualitatively more powerful than proc-macro-hack?

zbraniecki commented 5 years ago

@SergioBenitez

This would be the final blocker to Rocket on stable.

By the way, what prevents Rocket from using proc-macro-hack for migrating to stable? What are the cases in which unstable macros are qualitatively more powerful than proc-macro-hack?

I can only speak from my own experience - I use proc-macro-hack in unic-locale and it's a serious PITA to maintain - I need 4 crates in place of one. I'd say that maintaining that combination is the top of my burdens when working with any of my crates.

talglobus commented 4 years ago

@Centril why cc this issue?

jhpratt commented 4 years ago

With regard to an implementation audit for expansion to expressions, where would one get started? I'd be interested in helping.

petrochenkov commented 4 years ago

@jhpratt I think, the most useful thing right now would be to try addressing https://github.com/rust-lang/rust/issues/55414#issuecomment-554005412.

First, that work's eventual goal is to simplify macro expansion by removing two largely inappropriate features - expansion in non-terminal tokens and doc(include).
Second, it would free my hands, so I could do something in the direction of the invocation collector rewrite instead (mentioned in https://github.com/rust-lang/rust/issues/54727#issuecomment-523374617).

jjpe commented 4 years ago

Just bumped into this issue by trying to have a proc_macro expand to an expression. Are there currently any plans to enable this feature?

petrochenkov commented 4 years ago

@jjpe

proc_macro expand to an expression. Are there currently any plans to enable this feature?

Short version: I think we can stabilize it for fn-like macros now.

Longer version:

The main issue with them is rather ideological than technical - you almost always want hygienic local variables in expression macros, but the only stable kind of hygiene is Span::call_site which is unhygienic. (You can workaround that by using some uglified names or temporary macro_rules, like proc-macro-hack does, though.)

To resolve it we can stabilize Span::mixed_site (https://github.com/rust-lang/rust/issues/65049), which then will be used by proc macro authors for expression macros. Span::mixed_site was implemented almost 4 months ago, but unfortunately didn't get much adoption so far (https://github.com/alexcrichton/proc-macro2/issues/210), presumably because it's unstable.
There are some minor technical issues with the expression position macros, with trailing semicolons in particular (https://github.com/rust-lang/rust/issues/61733), but they are shared with already stable macro_rules, so we can ignore them right now.

So, with Span::mixed_site stable, fn-like proc macros can be stabilized in all positions where macro_rules macros can be used, basically. I think I'll start sending the stabilization PRs this weekend then, after checking a couple more things.

jjpe commented 4 years ago

@petrochenkov This is amazing to read, thanks for the quick response :smiley:

I understand that you probably don't unilaterally decide whether or not Span::mixed_site and expanding proc-macros to expressions are stabilized, but I would like to send a signal to the team that does make that decision that I would greatly appreciate having those features available on stable as soon as that is humanly feasible.

petrochenkov commented 4 years ago

The stabilization PRs for https://github.com/rust-lang/rust/issues/54727#issuecomment-580647446 are opened - https://github.com/rust-lang/rust/pull/68716 ("Stabilize Span::mixed_site") and https://github.com/rust-lang/rust/pull/68717 ("Stabilize fn-like proc macros in expression, pattern and statement positions").

pksunkara commented 4 years ago

I thought https://github.com/rust-lang/rust/issues/55414#issuecomment-554005412 is a blocker for proc_macro_hygiene. Did I misunderstand?

petrochenkov commented 4 years ago

@pksunkara It's a blocker for the attribute part of proc_macro_hygiene, but not for fn-like macros.

(As I said repeatedly in this thread and id https://github.com/rust-lang/rust/pull/52121, proc_macro_hygiene as a single feature gate doesn't make much sense.)

corbinu commented 4 years ago

@SergioBenitez Just wanted to check is #68717 the last needed piece for stabilizing Rocket or is something more needed?

joshtriplett commented 2 years ago

There are a number of features tracked in this issue. It sounds like some have been stabilized, while others (like def_site) haven't. Would it be possible to get a summary of the remaining work needed for proc macro hygiene? (It may also make sense to break this issue up into multiple tracking issues, or alternatively, get an issue with a checklist of other issues.)

TheButlah commented 2 years ago

I came here from: https://github.com/danakj/dynpath/issues/1

But reading this issue, I have very little clue how "non-inline modules in proc macro input" relates to this issue

antimora commented 1 year ago

I'm looking forward to this feature. Do you have any updates on the progress or timeline for this feature to become stable?

Specifically I would like this subfeature to be released:


error[E0658]: custom attributes cannot be applied to statements

JohnScience commented 11 months ago

The users of my qualifier_attr library would benefit from this because then library authors can write

use qualifier_attr::qualifiers;

#[cfg_attr(semver_exempt, qualifiers(pub))]
mod private_mod1;
#[cfg_attr(semver_exempt, qualifiers(pub))]
mod private_mod2;
// ...

instead of

#[cfg(semver_exempt)]
pub mod private_mod1;
#[cfg(not(semver_exempt))]
mod private_mod1;

#[cfg(semver_exempt)]
pub mod private_mod2;
#[cfg(not(semver_exempt))]
mod private_mod2;
// ...

where private_mod1 and private_mod2 are non-inline modules.

rust-lang / rust

Tracking issue for procedural macros and "hygiene 2.0" #54727