Tracking issue for "macro naming and modularisation" (RFC #1561)

[nikomatsakis]

Tracking issue for https://github.com/rust-lang/rfcs/pull/1561.

Roadmap: https://github.com/rust-lang/rust/issues/35896#issuecomment-277870744.

cc @nrc @jseyfried

[mark-i-m]

PSA: if you, like me, are wondering why use will only work for extern macros and not crate-local macros, see https://github.com/rust-lang/rust/issues/35896#issuecomment-299575710.

[dtolnay]

Seems like I should have raised this last month but sorry I just found this thread: I would be strongly opposed to stabilizing use imports of extern macro_rules macros until we have a solution for private helper macros.

extern crate log;
use log::warn;

fn main() {
    // error: cannot find macro `log!` in this scope
    warn!("Warning!");
}

extern crate serde_json;
use serde_json::json;

fn main() {
    // error: cannot find macro `json_internal!` in this scope
    let j = json!({ "k": "v" });
}

Our whole system mostly works for now because of the way #[macro_use] brings in all the macros. Moving toward importing macros individually like items will make everything seem more broken and fragile and will be immensely frustrating for macro users and macro authors compared to the current way.

An additional constraint is that ideally we would solve this in a way that a crate could support macro_use and use at the same time. For example something like telling log::warn! to invoke $crate::log! would not work because that leaves them no way to support old compilers as well as the new use log::warn.

I just found the thread so here is an approach with barely more than zero thought behind it, but just to illustrate one possibility that fits my criteria:

#[macro_export]
#[bikeshed_also_export(__log)]
macro_rules! log {...} // forward to __log

// `use log::warn` brings both `warn!` and `__log!` in scope.
#[macro_export]
#[bikeshed_also_export(__log)]
macro_rules! warn {...} // call __log

#[doc(hidden)]
#[macro_export]
macro_rules! __log {...}

Here as long as bikeshed_also_export is somehow cfg'd away or ignored by old compilers then the same code works equally well both for #[macro_use] extern crate log on old compilers and use log::warn on new compilers.

[dtolnay]

Tagging @rust-lang/libs because this stabilization would seem to put macro libraries in a tough situation.

[withoutboats]

Here as long as bikeshed_also_export is somehow cfg'd away or ignored by old compilers then the same code works equally well both for #[macro_use] extern crate log on old compilers and use log::warn on new compilers.

Its not though, is it? Unknown attributes are a hard error. You could just as easily cfg the whole macro definition based on whether you intend to be imported with #[macro_use] or not as you could cfg the attribute you're suggesting, and $crate::log! is much cleaner.

[dtolnay]

You could just as easily cfg the whole macro definition

I would not say that keeping two parallel macro definitions (one that calls private_helper! and one that calls $crate::private_helper!) is just as easy as a cfg_attr on one line.

Unknown attributes are a hard error.

I am not concerned about this because it seems easy to find a way around. Just for example the following compiles back to rustc 1.0.0:

#[macro_export(also_export(private_helper))]
macro_rules! ...

[golddranks]

@dtolnay I share your concern about the new use imports breaking stuff. What do you think about this proposal: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252/18 ?

[petrochenkov]

@dtolnay

I would be strongly opposed to stabilizing use imports of extern macro_rules macros until we have a solution for private helper macros.

I don'see how permitting use for macros is blocked by the private helpers problem. We are not forcing use and not deprecating #[macro_use] or anything, it still can be used when needed. (Yes, some adventurous people work on breaking lints like unnecessary_extern_crates, but thankfully they are allow by default.)

At the same time use_extern_macros is a prerequisite for stabilizing proc_macros/proc_macro_attributes that can't be imported with #[macro_use].

[petrochenkov]

#[macro_use] (at least on extern crate) works like a glob in use, but glob targeting only one namespace - macro namespace.

Perhaps more general feature - allowing use items to import only in selected namespace will help here as well.

// The syntax is exposition only
use a::b in value; // imports `fn` b, but not `type` b
use a::b::* in macro; // imports all macros from `a::b`

[petrochenkov]

The private helper problem is rather a blocker for recommending use over #[macro_use] in documentation.

[dtolnay]

@petrochenkov

I don'see how permitting use for macros is blocked by the private helpers problem.

Without a provision for helper macros, I would actively discourage users from using use for macro_rules macros: PSA do not use this feature, do not get in the habit of using this feature, you will be confused, you will be sad, your code will break, authors of the macros you use will be sad, stay away, etc. The fact that I would discourage people from using this feature so strongly is the blocker.

On top of that I would make no effort to support use import of macros that I write because it would be an unreasonable maintenance burden. This is just a continuation of how I already make no attempt to support macro_use(...).

I hear you that we are not forcing use and not deprecating macro_use but if we are going to be saying do it the old way, please please never use the new way then we should not stabilize the new way until the recommendation is different.

[dtolnay]

@petrochenkov

At the same time use_extern_macros is a prerequisite for stabilizing proc_macros/proc_macro_attributes that can't be imported with #[macro_use].

My objection:

I would be strongly opposed to stabilizing use imports of extern macro_rules macros until we have a solution for private helper macros.

I am concerned only about use of macro_rules macros. Proc macros can generally be factored in a way that does not require further expansion of helper macros.

[dtolnay]

The private helper problem is rather a blocker for recommending use over #[macro_use] in documentation.

Is this saying we should stabilize use but hope nobody finds out about it...?

[dtolnay]

@golddranks if that can be implemented, and if we can isolate the behavior to macro_rules macros, then that would be terrific and solves the problem. If I understand correctly, your idea is that just before a "cannot find macro `m!` in this scope" error, and only if the token m originated within the definition of a macro_rules macro, then it should make a last ditch effort to resolve m! within the crate that originated the m token. :+1:

[petrochenkov]

@dtolnay

I am concerned only about use of macro_rules macros.

I think it's possible to stabilize/enable the general macro importing mechanism while keeping imports of macro_rules macros sort of "not working" without a feature gate. It's not entirely trivial though, e.g. you can't just gate an import if it points to macro_rules, then e.g. use std::panic; would stop working.

#[bikeshed_also_export(__log)] What do you think about this proposal: https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252/18 ?

To clarify my priorities, I'd like to avoid:

• Making import resolution more complex. It's already too complex. Implicitly adding use my_crate::__log; on use my_crate::log; makes import resolution more complex. I can't predict how exactly it will affect the fixed-point import resolution algorithm, but it will certainly become stuck more often because every import turns into a glob in some sense because we no longer have guarantee that use a::b; can import only items named b, if b is a macro with "linked" helpers.
• Making hygiene more complex. It's already too complex. @golddranks proposes some new bespoke hygiene scheme in which call-site hygiene for macro_rules falls back to def-site hygiene.
• (Note: Currently macro_use on crates behaves sorta kinda like an additional prelude, affects only "relative" names (scope-based resolution), i.e. X, but not a::X or ::X, so it doesn't participate neither in import resolution nor in hygiene.)

So, my recommendation would be to:

• For new compiler versions: do not introduce new features into import resolution or hygiene systems, fully stabilize use_extern_macros, solve the private helper problem with $crate::__log, recommend use over macro_use.
• For old compiler versions if/while they need to be supported: on the user side #[macro_use] is the only way, on the library author side cfgs are somehow employed to generate both __log and $crate::__log macro paths depending on the compiler (sorry :( ). Or use is discouraged for a few compiler versions, then the switch from __log to $crate::__log happens.

[dtolnay]

In my opinion your recommendation does not meet the bar for the cross-edition interoperability story that people envisioned. To make a library that works equally well on 2015 and 2018 (as Serde and many other libraries would want to do for some reasonable transition period) we would be telling people:

1. You add a build-dependency on version_check. The dependency adds 0.7 seconds to compile time.
2. You add a build script, build = "build.rs". The build script adds 0.6 seconds to compile time.
3. In the build script you use version_check to determine whether the compiler version is sufficiently new to use $crate::private_helper!. The third-party version_check crate implements this by using the std::process API to shell out to rustc --version, then parsing the numbers out of a version string that looks like rustc 1.27.0-beta.5 (84b5a46f8 2018-05-15).
4. If sufficiently new, print to stdout the magic string "cargo:rustc-cfg=crate-macros".
5. Duplicate the definition of your public macros and any private helpers transitively called by them. Change one copy to use $crate::private_helper! and tag with #[cfg(crate_macros)].
6. Attempt to keep the duplicated code in sync as your crate evolves.

From the library authors' perspective this seems like a tough sell. For comparison the other approaches in this thread, as tricky as they would be to implement, look like this. Some sort of bikeshed_also_export way:

1. You make a one-line change

   - #[macro_export]
   + #[macro_export(...)]

   and your macro works flawlessly through #[macro_use] on all Rust compilers back to 1.0.0 and flawlessly through use on all sufficiently new compilers.

And @golddranks' hygiene way:

0. (There is no step 1. Your existing macros already work flawlessly through #[macro_use] on all Rust compilers back to 1.0.0 and flawlessly through use on all sufficiently new compilers.)

[withoutboats]

To make a library that works equally well on 2015 and 2018 (as Serde and many other libraries would want to do for some reasonable transition period) we would be telling people:

AFAIK this is a stabilization in both editions; we're talking about people who don't upgrade to the compiler version 1.29.0, not people who don't upgrade to the 2018 language edition. While libraries often support people on old compiler editions, it is a very different thing to require people to upgrade their compiler than to require that they update their code.

[dtolnay]

The way I mean this is: for some time Serde will want to work equally well on rustc <1.29 (which only supports 2015-style macro_use) and rustc >=1.29 (on which people expect to be able to use 2018-style nice new features like use). I agree that upgrading your compiler and changing your code are different things but telling a library to follow unappealing steps 1/2/3/4/5/6 or else release a breaking change for no other reason than we couldn't figure out macro imports -- seems not great.

[withoutboats]

@dtolnay its important to be clear here: with the $crate:: solution, you can continue to use #[macro_use] as long as your compiler is a recent enough stable (1.29 or whatever). What you call the "2015-style" system will still work, because macros can be compatible with both "styles." When you say "a breaking change" I am surprised - I did not know that serde adopted the position that increasing the minimum Rust version it required was a breaking change. Is that the case?

[dtolnay]

The minimum required compiler version from Serde 1.0.0 through today has always been rustc 1.13. So far we have had no trouble catering to users of new compilers using only 1.13's feature set. This would be the first time in 15 rustc releases that it becomes complicated to support users of new compilers -- which I guess is driving my concern here.

I am nominating for the libs team to discuss how we imagine the library situation playing out over the rest of the year. Around 1.29 or whatever are we expecting most libraries to drop support for compilers older than 1.29, whether through a massive round of breaking changes or by patch versions that aggressively push people to upgrade compilers?

@rust-lang/core may be interested as well: this affects peoples' perception of the stability of the language which may already be a sensitive topic around publicity of the edition.

[petrochenkov]

@dtolnay If something like https://github.com/rust-lang-nursery/api-guidelines/issues/123#issuecomment-390456962 works, it would be great. But I wouldn't personally mind a documentation-only solution "my macro library doesn't support importing with non-glob use and macro_use(named)" + possibly internal future-proofing with dummy internal helpers, until minimal compiler version is bumped to 1.28-1.29 naturally in the next year or two.

Now, let's assume that the combination "supporting use + simultaneously supporting older compiler versions + using same code for both" is absolutely critical and we need a language solution for it, then:

• It's important to understand that the solution is a temporary hack needed only during some transition period, it will go into the deprecation land later. At the same time this hack needs to be urgently implemented as insta-stable (a bit risky).
• We are limited by syntax macro_export(a, b, c) because it's accidentally accepted by older compiler versions or no new syntax at all, so we can't solve the problem by some new more generally useful mechanism, like hygiene opt-in for macro_rules (the opposite of https://github.com/rust-lang/rust/pull/47992).
• I think, this means we need some absolutely minimal and isolated solution getting the job done, but not interacting with other features (import resolution in particular) and complicating the whole system.

I think the most simple and local solution would be:

• We have a single-segment fn-like macro call my_helper!(...) for which name resolution fails as determined.
• Then we know hygienic context of the my_helper identifier, from which we can figure out definition of the macro in which it was originally written, in particular crate and kind of that macro.
• Then, if the macro kind is macro_rules, we either 1) try to resolve my_helper as $crate::my_helper with that crate, i.e. automatically doing the library author's job from my previous recommendation (simpler) or 2) try to resolve my_helper at def-site of that macro (may be more complex for macro_rules (as opposed to macro), I'm not sure the necessary infrastructure is in place).
• The resulting resolution is optionally filtered by the name list from macro_export(a, b, c). Increases complexity, not strictly necessary.
• We'll be able to emit a warning for this resolution scenario later if necessary.

ping @jseyfried who implemented a couple of similar hacks to support legacy behavior of macro_rules (in case he's still reading messages from github).

[alexcrichton]

Thanks for bringing these points up @dtolnay, always good to know about them regardless of where we are on the stabilization timeline!

To make sure I understand the issue, the point you're bringing up @dtolnay is basically that this code doesn't work today?

#![feature(use_extern_macros)]

extern crate log;

use log::info;

fn main() {
    info!("test");
}

If that's the case, that does indeed seem worrisome! I'm not sure, though, that it necessarily implies we should delay this or add more features to macros 1.0. For example we know that all code will continue compiling as-is (as it uses #[macro_use]). The only question is how we actually signal this transition.

So far we've been saying that you should replace #[macro_use] use use krate::macro_name;, but what if we instead suggested the true replacement for #[macro_use], namely use krate::*;? That's actually (modulo namespaces) what literally #[macro_use] is doing today (whether it looks like that or not). While I agree that a glob import is indeed unsightly it's also why we're developing a new macro system!

This to me seems like it leaves us with two downsides:

• We're recommending a "subpar" experience still. It's better than #[macro_use] but use log::*; isn't great either.
• The "correct" experience effectively requires an opt-in from all macro authors. Every single crate exporting a macro will have to go back and rewrite the macro to do one of two things:
  1. If backwards-compatibility is important, everything goes into the same macro invocation. (this I don't think works with log::info!() though?)
  2. Otherwise invocations of log!() is replaced with $crate::log!()

It seems to me that the #[macro_export] solution you're thinking about still has the problem of "all current macro authors must go back and maybe edit their code", right? The main difference, I believe, is that the backwards-compatible solution is much nicer in that you can just list dependent macros.

In other words the delta over where we are today (if we stabilize), is that macro authors who both go back and take a look at their macro-exporting crates while also considering backwards-compatibility don't have to duplicate their definitions. I think though the cost of stabilizing this is still the same?

To me that seems like a wortwhile tradeoff to make. The primary use case for this features is macros 1.2 which is targeted to be stable at the same time as this feature. It's sort of secondary that we expect macro_rules! to transition to this as well, but I think it's totally fine to basically just delay our messaging here. I think we can "fix" this in a backwards-compatible way in the sense of making authoring macros a bit nicer, but I don't think we should halt or delay the stabilization of 1.2 because macro_rules! isn't so great (as it can't possibly be worse than today which we're already "happy with")

[alexcrichton]

@dtolnay and I had a chat about this on IRC, and I'll try to summarize here.

Let's say we stabilize this feature in 1.29. There exist popular crates which will maintain compatibility with pre-1.29 compilers, for example log and bitflags. We cannot as-is reasonably transition users of these two crates to using the module system instead of #[macro_use]. One of two possible options seems like a way to avoid this pain:

1. As a first option, we would not stabilize using the module system to import macro_rules macros. This means that you'd use this feature to import procedural macros, custom attributes, etc. You would not, however, use it to import macro_rules macros. This means that #[macro_use] extern crate foo; is still a thing, an obvious downside.
2. Instead we stabilize some sort of feature to allow the log and bitflags crates to get implemented in a "reasonable fashion". This would be along the lines of solutions like proposed by @dtolnay and @petrochenkov.

Alternatively we could stabilize this feature and simply not announce it. Instead we could wait until a sufficient mass of "popular crates" transition to requiring 1.29 or future compilers, in which case everything is right as rain and we can sound the trumpets at that point.

[golddranks]

@alexcrichton So the proposal for "last resort defsite hygiene" isn't considered feasible here? @petrochenkov already that he considers it making hygiene too complex (https://github.com/rust-lang/rust/issues/35896#issuecomment-390446792) but I'd like to hear your opinion too.

[petrochenkov]

@golddranks

So the proposal for "last resort defsite hygiene" isn't considered feasible here?

That's item 2. from @alexcrichton's list. "Last resort to $crate::my_helper" (https://github.com/rust-lang/rust/issues/35896#issuecomment-390477706) should be simpler than "last resort to defsite hygiene" due to subtle differences like this:

// Def-site resolution for `public_macro` would refer to this private non-exported macro.
// This kind of legacy interaction is not currently supported even for `macro` items.
macro_rules! my_helper { ... }

#[macro_export]
macro_rules! public_macro {  ... my_helper!()  ...  }

// Macros with same name can shadow each other in a module.
// `$crate::my_helper` refers to this exported macro.
#[macro_export]
macro_rules! my_helper { ... }

That said, I'd still prefer not doing this.

[golddranks]

Ah, I see. Pardon my confusion. Indeed, that is simpler.

[Nemo157]

Should this currently be working for macros exported by std, testing the below code gives error[E0432]: unresolved import `std::assert`:

use std::assert as std_assert;
std_assert!(2 == 3);

[petrochenkov]

@Nemo157 https://github.com/rust-lang/rust/pull/48813 made assert built into the language so it's no longer defined by libstd.

[Nemo157]

@petrochenkov interesting, follow up question then.

Should built-in macros somehow act as if they were exported from libstd/libcore for this feature?

It seems that stabilizing this feature would make it impossible to transition any other macros from being real macros into compiler builtins, otherwise any uses like above would stop compiling.

One way might be to keep the macros as real macros that expand into the compiler builtins, something like macro_rules! assert { ($($t:tt)*) => { __builtin_assert!($($t)*) } }. That way the macros themselves would still be able to participate in the normal naming/modularisation scheme.

[petrochenkov]

macro_rules! assert { ($($t:tt)*) => { __builtin_assert!($($t)*) } }

Yes, that's probably something we should do when moving a macro from the library to the language.

[nikomatsakis]

Is it correct (forgive me if I am repeating) that if we supported $crate::bar!() as a way to invoke macros, then you could write macros that use private helpers without requiring users to manually import them? (Users would of course have to be using a new enough compiler to support that.)

If so, I definitely feel that just supporting that syntax is a viable solution. If a crate X wants to retain compatibility with older compilers, it means that its consumers just have to use #[macro_use] extern crate the_crate. Suboptimal but not a total disaster.

(I think we should make some effort at promoting community wide "compatibility ranges" when it comes to rustc versions, as well, but that is perhaps a discussion best had elsewhere.)

[nikomatsakis]

That said, @nrc and I were talking on discord and I had an idea that I kind of like which I wanted to write down. It's a variation on a proposal that @nrc raised.

TL;DR: We try to make the ability to impor a macro via use a new feature which some macro-defining crates will have opted into and others will not have yet done. We do not attempt to "retrofit" existing crates into this model without some opt-in.

The first part is to make it "opt in" to have macros usable via use — existing macros that are not changed would still require #[macro_use] extern crate the_crate; to be used. This means then that crates can choose when to enable this feature based on whether they use helper macros and whether they are ok with requiring that their consumers have a newer compiler (e.g., one that can support the new edition).

Now, how does that opt in look? I'm actually going to spin two variants of this proposal, one more aggressive and one less aggressive, because I don't quite know how hard each one would be to do.

---

The more aggressive variant

We allow macro_rules! macros to be declared pub. This would be the only macro that is allowed to be declared as pub. Ideally, you would be able to declare it as pub anywhere. This would also mean the macro does not use the default macro_rules! mechanism but instead opts in to the more lexical mechanism -- also within the crate. In other words, it works just like pub macro was meant to work.

(Maybe, in the new edition, we can also make macro_rules! foo without pub work using the lexically scoped rules? We might not be able to write a migration lint for that, though, but the only real problem is in the case of shadowed macros, I guess, and that seems like a corner case that we could detect and require manual intervention? Maybe?)

Now, for compatibility, we can still permit #[macro_export] on public macros. This means that a crate can do something like this:

pub mod macros {
    #[macro_export]
    pub macro_rules! my_macro  { ... }
}

and now folks with a new compiler can do use the_crate::macros::foo or use the_crate::macros::*, but folks with an older compiler can still do #[macro_use] extern crate the_crate;.

This does not solve hygiene. If you have private "hidden" macros like _foo!, they are "observable" to your users, who must either import them manually or use a glob import. I think though that having the option to do use the_crate::macros::* is less .. unsightly than requiring use the_crate::*.

As a bonus, macro-rules within a crate work like all other items, just like we always wanted. Huzzah. (Naturally #[macro_use] on modules would have to be .. deprecated? The interactions here may just be too complex.)

---

The less aggressive variant

Instead of pub macro_rules!, we could do #[macro_export(pub_use)] to signal that you want this macro to be imported via use and not via #[macro_use]. This would be a breaking change. Perhaps there is some other way to say you want both: #[macro_export(macro_use, pub_use)]. Intra-crate, nothing changes (as today).

---

Observations:

These proposals basically add a new feature: macro imports using use. We consider this the "recommended" way to use macros, but the old way continues. Naturally this means one can support older compilers just by not adopting the new feature (or by adopting both).

There are some problems that are not "fully solved" in some sense -- in particular, if you want to have private helper macros and retain compatibility with older compiler versions, you still have to force your consumers to use #[macro_use] extern crate the_crate;. But that feels ok: it often happens that crates stick to old idioms for some time, and this is just an instance of that (as stated in the TL;DR above).

[petrochenkov]

@nikomatsakis

Is it correct (forgive me if I am repeating) that if we supported $crate::bar!() as a way to invoke macros, then you could write macros that use private helpers without requiring users to manually import them?

$crate::bar!() is already supported with #![feature(use_extern_macros)], you just can't use macros expanding into it in the same crate, but that shouldn't be a big problem for libraries that produce such macros rather than consume them.

EDIT: ... and I'm pretty sure that we can provide support for $crate::my_helper!(...) in the same crate with exactly same semantics as $crate::my_helper!(...) from other crates without any other changes, if its lack causes too much pain to library authors.

[petrochenkov]

I'm not quite sure about the motivation for proposal in https://github.com/rust-lang/rust/issues/35896#issuecomment-391856772. It adds some features, edition-breaks macro_rules!, but doesn't actually change the situation with @dtolnay's concern - #[macro_use] still has to be used for some time and use still can't be used everywhere for some time - this is already true without any changes.

[alexcrichton]

Ok I had a bit more discussion with @nikomatsakis on discord about his previous proposal and some things I wanted to write down...

---

The crux of the problem here is pre-1.29 compatibility. AFAIK all features of use_extern_macros work great, and the problem only arises when a crate wants to support using its macros via use krate::foo and also work with #[macro_use] extern crate krate; for pre-1.29 compilers.

Now this aspect of crates is actually more far reaching than just pre-1.29 compatibility. For example the whole feature here is using macros from other crates, ones that you're possibly not writing yourself. In that situation the upstream crate may have a different development policy than you, supporting different versions of the compiler. This can run the risk of having a network effect where popular crates like bitflags and log may be laggard in enabling usage of their macros via the module system and specific paths (use log::info; vs use log::*;).

The first question then is is it a goal of this issue that we want to support this pre-1.29 compatibility use case. It's not an easy one to achieve, but the cons of not supporting this are:

• Crates which do want to support pre-1.29 compatibility really do have to bend over backwards. One option is to rewrite macros to only use themselves recursively. This can be a maintenance burden as well as untenable in the long run. Alternatively they need to use $crate::foo!() but use compiler version detection at build time to select one of two definitions of the macro. This duplication can also be a maintenance burden. All in all, this con is that it's likely pretty few crates which want pre-1.29 compatibility are likely to be usable with macros and the module system.
• If a number of popular crates (e.g. log, bitflags, lazy_static, etc) do not support the module system then this feature may feel "incomplete". For example you'll have to remember which crates to use the module system with and which crates to use #[macro_use] with, and we'll risk being in a weird transition period for a few cycles (maybe longer?)

And I think I'm forgetting the last one! In any case though it's also worth pointing out that it's not clear what the impact is here. For example we don't know which popular macros don't work with the module system basically as-is or are easily modifiable. Additionally it's always possible to require use log::*; which will work very close to what #[macro_use] does today.

So ok, let's say we do want to consider pre-1.29 compatibility an option. So far it sounds like there's two plausible (if not-so-fun-to-implement) solutions:

• @dtolnay's idea to exploit the fact that older compilers ignore extra tokens in #[macro_export ( ... )] to add annotations necessary to get use of the macro working
• @petrochenkov's idea to exploit name resolution errors and fall back to attempting to resolve macro invocations at the macro's definition (pseudo-hygiene)

If we decide to go one of these two routes then we'll want to avoid stabilizing modules and the macro_rules! system for 1.29, but we'll probably want to continue to stabilize procedural macros and attributes.

And finally, one last thing worth mentioning. No matter what we do it's likely that when we stabilize macros and the module system it won't be usable with the large majority of macros already in existence in the ecosystem. In other words most macro definitions will need source level changes (in one way or another) to work with the new system (like using $crate::__helper!()). In that sense we may want to consider some possible tweaks (like @petrochenkov's idea) to reduce the impact here and make more macros usable-by-default

[alexreg]

Just to be clear: is the idea to fully deprecate macro_use attributes with crate-local macros too, eventually? Having two different syntaxes depending on where the macro is defined seems like a really bad state to be in for any appreciable time. Certainly not something I would think should be stabilised.

[alexcrichton]

@alexreg eventually macro-macros (macros 2.0) is the way of the future which fully integrates with the module system, there's currently no interim plans to fully deprecate #[macro_use] with macro_rules! for within a crate

[alexreg]

@alexcrichton Fair enough; thanks for clarifying.

[petrochenkov]

I made a rough implementation for the "macro_rules! helper fallback" as described in - https://github.com/rust-lang/rust/issues/35896#issuecomment-390477706.

As it turns out, it's not "just a fallback" it's "fallback in the middle of fixed-point resolution" again.

To truly rely on the fallback we need to expand everything before doing it, but expansion cannot progress without performing the fallback. The workaround is similar to other situations with fallback in resolution - optimistically perform the fallback even if the resolution is undetermined (i.e. there are macros that can potentially expand to #[macro_use]), then detect "time travel" post-factum and report an error if new #[macro_use] imports appear that would be preferred to the fallback.

I think the helper_fallbacks.contains(name) check on global_macros.insert(name) should catch all the cases of "time travel", but I'm not 100% sure. EDIT: Unfortunately it doesn't, see the second commit.

It's certainly a trade-off, but now my feeling that supporting this would harm the language long-term in favor of short-term version-migration benefits is stronger than before.

[petrochenkov]

Status update: https://github.com/rust-lang/rust/pull/51145 addresses the last known regression from enabling use_extern_macros.

I propose stabilizing use_extern_macros without providing a language solution (like https://github.com/rust-lang/rust/issues/35896#issuecomment-392393413) for the macro helper problem discussed above. I think we should do it now, so $crate::my_macro becomes available on stable as soon as possible. We could also do the backport of the stabilization PR (https://github.com/rust-lang/rust/pull/50911) and bugfix PRs (https://github.com/rust-lang/rust/pull/50355, https://github.com/rust-lang/rust/pull/50760, https://github.com/rust-lang/rust/pull/50908, https://github.com/rust-lang/rust/pull/51145) to beta, then it will be available on stable starting with 1.27.

[aturon]

cc @rust-lang/lang on this last comment; nominating for meeting as well.

[dtolnay]

Here is a hopefully clearer and less shouty writeup of my perspective in anticipation of the lang team discussion.

At issue

Enabling use of individual macro_rules macros from another crate.

use log::warn;

Background: new features

We are all used to new language features and have seen some fantastic ones recently. From the point of view of a particular library, new features generally break down into one or multiple of:

1. Features that make it possible to solve some problem that was impossible to solve before in Rust. For example union allows a sys crate to expose a signature ABI-compatible with some C function that passes unions. Before Rust 1.19 it just couldn't have provided a binding for such functions. This is great and expands the world of problems that Rust is suitable for.

2. Features that prompt users to redesign the API of a library that was previously designed and working. For example Rust 1.20 added associated constants, prompting a redesign and major version bump of the bitflags crate. Bitflags existed and worked decently well before 1.20, but the new feature provided a materially better way to solve the problem bitflags intended to solve. The authors took into account the benefit of the improved API as a tradeoff against the cost of rolling out a change to the API. This is healthy and it is great when language features are adopted because of the API design improvements they make possible.

3. Ergonomics improvements that make Rust code easier to write, read, and maintain without much affecting API design. An example of this is default binding modes in match. These are great and a quality of life improvement for beginners as well as experienced users.

Background: foundational crates

A defining characteristic of a foundational crate like Serde is that only type-2 features are relevant to us.

• The library addresses a particular problem domain, so unless we are expanding the problem domain, type-1 features geared toward things that couldn't be done before are not relevant. The library does a thing, so the thing it does is not one of the things that cannot be done prior to the new feature.

• We pay close attention to type-2 features. For example if some future version of generic associated types makes it possible to do what Serde does using a radically nicer API, we would redesign the API and release the improvement as a breaking change.

• We don't pick up type-3 features until such time as we bump the required compiler version for type-2 reasons. This has to do with how much weight is on one side of the tradeoff that exists between benefit of ergonomic improvements to development within the Serde codebase, versus cost of pushing a compiler upgrade to users. Regardless of how big an ergonomic improvement may be, there exists some threshold of number of downstream users beyond which their upgrade friction outweighs our internal ergonomic benefit.

Why the change is concerning

The thing that is unprecedented about use imports of macro_rules is that, while it does not break existing library APIs (a non-starter), it does break users' expectations of existing library APIs.

That is, compiler developers would categorize the change as type-2 in the sense that libraries today expose a working API that looks like #[macro_use] extern crate log, and the new feature gives them a way to expose a nicer API that behaves more like imports elsewhere in the language, use log::warn, if they choose. It is easy to see this as no different from a canonical type-2 new feature like associated constants. The authors of bitflags would have the duty of deciding whether the API improvements afforded by the new feature outweigh the cost of rolling out a change to the API.

But in this aspect the point of view of compiler developers diverges from that of library developers and library consumers. Unlike compiler developers, everybody else does not perceive the API of a crate as "it exposes such-and-such macro importable through #[macro_use]". Rather, they perceive the API as "it exposes such-and-such macro" and independently, "here is how macros in Rust are imported."

The distinction is important because it deprives library authors of the choice of following their ordinary type-2 decision process. As an author of a crate that exports macros I cannot weigh the two choices and decide whether to stick with #[macro_use] for now or jump to use. Outside of my control, and regardless of anything I may write in documentation, users will expect to be able to use my macros because that is how they understand macros are imported in Rust.

When they write a use and receive the following error:

cannot find macro `__some_internal_helper!` in this scope

then depending on their personal experience the user will either blame Rust ("macro imports sometimes work and sometimes don't work, what am I doing wrong?") or blame the library ("I tried to use your library and your code does not compile"). Either way the ecosystem feels flaky and perpetually broken in a way that it doesn't today.

2018 transition

The whole thing is only a problem during a brief 2018 transition period right?

This is true. The transition period only lasts until the macro-exporting crates people use have bumped their minimum supported compiler version and moved to invoking helpers through $crate::helper! syntax.

It is hard to say how long that would be because library authors have differing opinions about how to do this correctly and each library will have hard decisions to make: Do we force the ecosystem through a serde 2.0 upgrade? Do we prolong the transition period during which our API feels flaky and broken when people try to use it in the ways they expect? Do we aggressively force a compiler upgrade on users by breaking their builds? Regardless of which way you would decide, notice how all three of these result in an ecosystem that feels unstable.

It is possible that our attention to stability has over-indexed on one aspect of stability: the "lifespan" of code meaning how long before a compiler change breaks the code and it no longer compiles (which we promise is never, with some well-reasoned exceptions). This thread brings up "healthspan" as a different aspect of stability, how long before a compiler change breaks users' expectations around code by dropping it into a "transition period" that requires code changes to escape out of.

Large codebases

My experience in some large codebases leads me to value lifespan and healthspan as equally important. If it is expected that compiler upgrades are going to require ongoing maintenance investment in the form of periodic source-level changes to escape out of "transition periods", the value of never being actually broken by a compiler upgrade is greatly diminished.

In a large codebase we require the ability to write a library, finish it, and trust that it will age well until a type-2 redesign of its API.

Also the larger the codebase, the longer it takes to adopt compiler versions. Suppose that through perfectly legitimate inference breakage or soundness fixes we break X% of source lines of code every release (where X is a number much less than 1%). Those take increasingly long to work through. Also large codebases are increasingly likely to hit blocking perf regressions, again taking time to resolve. All of this means that for a large codebase there is value in foundational libraries supporting a generous range of old compilers.

Path forward

The notion of healthspan is why I was particularly excited about the approach in @petrochenkov's prototype https://github.com/rust-lang/rust/issues/35896#issuecomment-392393413 which entirely avoids breaking developer's expectations of existing APIs. That is, we would change their expectations, but we would not break them because their new expectations of being able to use log::warn would work seemlessly. I would love to see something like this adopted. I am grateful that you took the time to develop the implementation and I think it was an important thing to try. Thanks also to @golddranks for the idea and internals forum discussion in https://github.com/rust-lang/rust/issues/35896#issuecomment-390386189.

That said, if the compiler team believes that the solution there is not tenable then obviously we can't ship it.

Long-term I expect compiler versions understood by Cargo will tip the scales heavily in favor of rapid adoption of new language features including by foundational libraries. I know Josh has been working on this and I am very excited about progress there.

[nikomatsakis]

OK, we had a long discussion. I think there was general consensus that current state of affairs is indeed a cause for concern and that we would rather not stabilize the features "as implemented". We did not reach any proposal that had a clear consensus. So I'm going to present a few possible routes. You will find a DropBox paper with our notes at this link.

Two goals

One of the things that we realized is that there are two features at play here:

• Extern crate elision: Removing the need to write extern crate.
  • In particular, with the new module system proposal, there isn't really any other reason to write extern crate besides bringing in macros.
• Selective import: Basically, the ability to import individual macros, and generally to treat macros as "ordinary items". This also means that you get to treat procedural and macro-rules macros identically.
  • Note that there remain distinctions that are important, for example macro-rules crates cannot be imported within a crate via use today. (One question that was raised was: how hard would this be to do anyway?)

For some people, one of these goals may be more important than the other, which influences the shape of a satisfactory solution.

Concerns

There are various concerns to be balanced:

• Splitting the ecosystem. Clearly, if serde drops support for older compilers, that will cause a certain amount of churn (as @dtolnay elegantly summarized here).
• User confusion. If we tell users that they can import macros, but then doing use log::debug gives an error, that's suboptimal. Furthermore, we should ideally be able to guide users relatively clearly on how to import any given macro.
• Technical debt. We are particularly concerned with the idea of adding new modes of fallback into name resolution. That never seems to end well, and we don't really have a strong "model" for how name resolution works to convince ourselves that it will be ok. Plus most "fallback-like" proposals tend to sound like hygiene -- which we know is hard and has a bunch of subtle concerns to be addressed.

Some possible routes

extern crate 4eva. One point of view is that this whole conundrum is evidence that trying to "blur the line" between "macro-rules" and "macros 2.0" was a mistake. We should back off from supporting this feature at all and instead just continue to have people use #[macro_use] extern crate foo; as they ever did. Procedural macros would still use use. The main point here is that any attempt to bridge this gap will result in technical debt and language complexity that we can't get rid of. Obviously taking this route solves neither of the two goals.

Macro glob. We considered @petrochenkov's proposal of having some kind of macro glob form (some syntax proposals below). This could well address the "extern crate elision" goal but does not address selective import. This can be couple with pub macro_rules! as well to address select import. Some syntax proposals:

• #[macro_use] use foo; -- presumably this would imply the old "no scope" behavior though?
• use foo::macro::*;
• use foo::macro *;
• use foo::*!;

One thing to consider here is other namespaces. For example, we might (in the future) want to support use foo::type * or use foo::impl *.

Public macro-rules. The idea here was to have macro-rules macros opt-in to use import as a way to signal that this is how users should use them. This aims to address the selective import goal primarily while trying to avoid user confusion -- that is, it effectively defines a new set of macros, kind of "macros 1.5", which are macro-rules macros that can be brought in through use. Said macros should use $crate to invoke helpers and so forth. This allows us to give relatively clear errors: for example, trying to import an "old style" macro can result in a message like "older-style macro-rules macros can only be imported with an extern crate" (or perhaps a macro glob, if we offered that).

recurse-within-crate. We did have one fresh idea for how we might try to "have our cake and eat it too". We were thinking that we could potentially add an annotation like so (obviously the precise name is TBD):

#[macro_export(recurse_within_crate)]
macro_rules! foo { .. }

The effect of this annotation would be that any ! that appears within the definition of foo (to be determined by span information) is always resolved against the source crate (macros that appear in the arguments of foo would expand as normal, of course). This is different from @petrochenkov's proposal in a subtle, but important way: it is not a fallback mechanism. Rather, we would resolve only against the source crate, as this is usually what crates need anyway (the only exception would be macros that are using higher-order macros or something).

This allows crates like log to upgrade while still working with older versions of rustc, but new users can import crates without pain. It doesn't seem like that much technical debt to bear (my rule of thumb, at least, for name resolution is that fallback is bad, but hard choices like "always resolve from here" are fine).

Conclusion

I don't really have a conclusion, but it seemed like the tendency in the meeting was to want to push towards one extreme or the other:

• Either we can make it so that you can use macros across crates and existing crates can be upgraded to work with that (e.g., via the "recurse within crate" proposal);
• or, we try to back off and draw clearer lines -- but exactly what to back off to is not clear.

[nikomatsakis]

@petrochenkov

I'd very much appreciate your feedback on this "recurse-within-crate" idea. I'll "quote" it from my previous comment here for ease of reading. =)

---

recurse-within-crate. We did have one fresh idea for how we might try to "have our cake and eat it too". We were thinking that we could potentially add an annotation like so (obviously the precise name is TBD):

#[macro_export(recurse_within_crate)]
macro_rules! foo { .. }

The effect of this annotation would be that any ! that appears within the definition of foo (to be determined by span information) is always resolved against the source crate (macros that appear in the arguments of foo would expand as normal, of course). This is different from @petrochenkov's proposal in a subtle, but important way: it is not a fallback mechanism. Rather, we would resolve only against the source crate, as this is usually what crates need anyway (the only exception would be macros that are using higher-order macros or something).

This allows crates like log to upgrade while still working with older versions of rustc, but new users can import crates without pain. It doesn't seem like that much technical debt to bear (my rule of thumb, at least, for name resolution is that fallback is bad, but hard choices like "always resolve from here" are fine).

---

PS, if this idea was raised before, then I missed it, and I apologize.

[nikomatsakis]

I was pondering @dtolnay's breakdown of features and I wanted to relate it to the various things contained in this summary comment. @dtolnay classified features as "type 2 or 3" (I'm ignoring type 1):

• [Type 2:] Features that prompt users to redesign the API of a library that was previously designed and working.
• [Type 3:] Ergonomics improvements that make Rust code easier to write, read, and maintain without much affecting API design.

I think this is an insightful way of breaking things down, and I think it is useful to look at the proposals in those terms.

(As @dtolnay said, the current implementation of this feature doesn't really fit this breakdown. It's not a feature that crates choose to use or not to use. All exported macros are opted into it, for the convenience of their consumers, but many crates are not made to be used that way.)

The "pub macro-rules" proposal, which aims to make the new import style "opt-in", I think is an attempt to repackage this feature as a "type 2" feature: something that may be worth overhauling your crate in order to support, because users expect it (but not something you must support).

The "recurse_within_crate" proposal aims to exempt the feature from this breakdown. That is, this remains a feature that crates must use, but we provide a way for the vast majority of them to do so seamlessly (presuming they update their source if it is necessary).

It's not clear to me whether we should be concerned about crates that are never updated, which will have their macros "exposed" but which may not work — it depends how many such crates there are and whether they use helper macros. I suspect this will be a minor problem in practice. If we were truly paranoid, though, we could say that plain #[macro_extern] means that the macro can only be used the old way, but that one can opt-in to import, e.g., by choosing between #[macro_export(recurse_at_invocation)] (today's default behavior) or #[macro_export(recurse_at_definition)] (the new behavior, renamed).

(Actually, I think I like this way of declaring that a macro is "use"-able better than writing pub, since it does not suggest that it will work intracrate.)

[eternaleye]

@nmatsakis:

The effect of this annotation would be that any ! that appears within the definition of foo (to be determined by span information) is always resolved against the source crate (macros that appear in the arguments of foo would expand as normal, of course).

How would that play with 'dynamic' macro calls, like nom uses heavily?

/// Wraps a parser in a closure
#[macro_export]
macro_rules! closure (
    ($ty:ty, $submac:ident!( $($args:tt)* )) => (
        |i: $ty| { $submac!(i, $($args)*) }
    );
    ($submac:ident!( $($args:tt)* )) => (
        |i| { $submac!(i, $($args)*) }
    );
);

It's parsed/broken up in argument position, not expanded - then expansion happens in the body, from an ident given by the caller, but with different arguments.

[nikomatsakis]

@eternaleye

How would that play with 'dynamic' macro calls, like nom uses heavily?

Indeed, we discussed nom as an example in the meeting. My assumption would be that nom would not opt into that "recursive-call" feature. The question is whether nom also uses "hidden" helper macros that users aren't supposed to know about. Or -- at least -- I thought that was the important question. Thinking about it now, I think maybe it doesn't matter so much if the helpers are hidden or not. If we're going to support select import, you'd still like to be able to only import the things you directly reference.

So at the end of the day, the question is just whether the crate has both macros that invoke one another and the need invoke macros from the user's crate (likely via indirection). If so, the mechanism is going to be approximating hygiene to some degree, and that seems (to me) to be approaching a line of "too much complexity".

That argues I think against permitting selective import of macros, at least without some explicit opt-in.

[nikomatsakis]

That argues I think against permitting selective import of macros, at least without some explicit opt-in.

That is, to clarify: if there were some opt-in mechanism, then nom could either elect to do nothing, or rewrite to use $crate:: and opt-in to permitting selective import.

[jimmycuadra]

If the "extern crate 4eva" approach is taken, that means that only macro_rules! macros are forever bound to being imported by #[macro_use], correct? When we have Macros 2.0 declarative macros (i.e. the macro keyword), that would share normal use-based imports with Macros 2.0 procedural macros, correct? If I'm understanding all that right, then I'm in favor of sticking with extern crate/macro_use. It'd be nice to have a clean split between the old system and the new system. Shipping use-based imports only with Macros 2.0 (in its forms) would be a nice carrot on a stick to encourage people to port their macros to the new system. And of course it avoids the confusion of having multiple ways to import macros from the old system.

[nikomatsakis]

@jimmycuadra

If the "extern crate 4eva" approach is taken, that means that only macro_rules! macros are forever bound to being imported by #[macro_use], correct?

Correct.

When we have Macros 2.0 declarative macros (i.e. the macro keyword), that would share normal use-based imports with Macros 2.0 procedural macros, correct?

Correct.

If I'm understanding all that right, then I'm in favor of sticking with extern crate/macro_use. It'd be nice to have a clean split between the old system and the new system.

I'm starting to lean that way myself, after having thought it over since yesterday. It's a tough call, though, but I feel like saying #[macro_use] extern crate foo; is the (somewhat verbose) syntax for using macros from an external crate is reasonable (and analogous to #[macro_use] mod bar being the syntax for getting macros from a module). The extern crate form also has this side-effect that foo becomes a member of that module, which isn't really needed anymore, but that's ok.

[withoutboats]

I see removing extern crate from idiomatic Rust as a key goal of the 2018 namespacing changes, and I don't see introducing a new syntax which is semantically equivalent as "blurring the lines" at all (i.e. you still can't import 1.0 macros individually). I don't have strong opinions about what the replacement syntax should be except that it should be connected to use statements.

While from the perspective of someone who already understands extern crate, this may seem like just two ways to do the same thing, from the perspective of someone who comes into a Rust where extern crate is not normal, the current syntax will be extremely weird. It seems much more natural to tell them that you can import 1.0 style macros using use log::macro*; or whatever that's connected syntactically to other imports.