search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
98.77k stars 12.77k forks source link

Tracking issue: declarative macros 2.0 #39412

Open nrc opened 7 years ago

nrc commented 7 years ago

Tracking issue for declarative macros 2.0 (aka macro aka decl_macro aka macros-by-example).

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

RFC PR: https://github.com/rust-lang/rfcs/pull/1584

cc @rust-lang/compiler

About tracking issues

Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the relevant feature gate label. Discussion comments will get marked as off-topic or deleted. Repeated discussions on the tracking issue may lead to the tracking issue getting locked.

Tasks

Potentially blocking issues:

CryZe commented 7 years ago

@nrc Typo in Issue Name: "issuse"

jseyfried commented 7 years ago

Tasks

(dtolnay edit: moved the checklist up to the OP)

nrc commented 7 years ago

We need to RFC a whole bunch of stuff here. In particular, I would like to propose some new syntax for declaring macros and we should RFC the changes to matchers.

tikue commented 7 years ago

Can the hygiene RFC mention pattern hygiene? This in particular scares me:

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

macro_rules! ignorant_macro {
    () => {
        let i = 0;
        println!("{}", i);
    };
}

fn main() {
    // oh no!
    ignorant_macro!();
}
jseyfried commented 7 years ago

@tikue I'm not sure patterns need special treatment with respect to hygiene.

For example, on the hygiene prototype,

#[allow(non_camel_case_types)]
struct i(i64);

macro ignorant_macro() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

fn main() {
    ignorant_macro!(); // NOTE: in this macro invocation
}

This makes sense to me since let i = 0; is shadowing a tuple struct. In particular, if let i = 0; were removed then the following use of i would resolve to the tuple struct, no matter where ignorant_macro is used.

Note the symmetry to this example:

#[allow(non_camel_case_types)]
struct i(i64);

fn ignorant_fn() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

If the tuple struct i isn't in scope at macro ignorant_macro() { ... }, then the let i = 0; does not shadow it and there is no error. For example, the following compiles on the hygiene prototype:

mod foo {
    pub macro ignorant_macro() {
        let i = 0;
        println!("{}", i); // Without `let i = 0;`, there would be no `i` in scope here.
    }
}

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

fn main() {
    foo::ignorant_macro!();
}
alexreg commented 7 years ago

Has there been any progress on Macros 2.0 lately?

mark-i-m commented 6 years ago

Note for those who haven't seen yet: macros 2.0 is apparently slated to be stable later this year, according to the proposed roadmap (rust-lang/rfcs#2314)...

On the one hand, that's pretty exciting :tada:!

On the other hand, I was really surprised that the feature is so close to being done and so little is known about it by the broader community... The RFC is really vague. The unstable book only has a link to this issue. This issue(s) in the issue tracker mostly have detailed technical discussions. And I can't find that much info anywhere about what's changed or how stuff works.

I don't mean to complain, and I really truly appreciate all the hard work by those implementing, but I would also appreciate more transparency on this.

alexreg commented 6 years ago

@marcbowes Yeah, I'm kind of worried too. It seems like there's quite a lot of work left for stabilisation this year. I offered to work on opt-out hygiene for identifiers myself, but have received no response yet... Some greater transparency would be nice, as you say.

petrochenkov commented 6 years ago

Macros 2.0 are not even in the RFC stage yet - the @jseyfried's RFC linked in the issue was never submitted, there's only very high level RFC 1584. The implementation is purely experimental and mostly unspecified, and large part of it (not related to hygiene) is reused from macro_rules! without fixing problems that macros 2.0 were supposed to fix.

Some greater transparency would be nice

The problem is that no work happen right now, so there's nothing to reveal :( @nrc is busy, @jseyfried is busy, I worked on macros 2.0 a bit and would really like to dig into them more, but I'm busy too, unfortunately. We need someone to adopt the feature, support and extend it, and gain expert-level knowledge of it, otherwise we will end up breaking hygiene and syntax details left and right after stabilization.

alexreg commented 6 years ago

@petrochenkov Ah, fair enough. I mean, that's a shame, but it makes sense at least. I guess this is an open call to anyone who might be able to take ownership of this feature, since no one comes to mind? I could still have a go at a little sub-feature, but I'm certainly in no position to take ownership of this.

petrochenkov commented 6 years ago

@alexreg

I could still have a go at a little sub-feature

Yes, please do! Hygiene opt-out is one of the primary missing parts and implementing it would be useful in any case.

IIRC, two questions will need to be decided on during implementation:

alexreg commented 6 years ago

Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in #40847)

Yeah, this was the plan. :-)

What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

What's your inclination? I'm leaning towards the context of m's invocation, but curious to hear your thoughts...

petrochenkov commented 6 years ago

@alexreg

I'm leaning towards the context of m's invocation

I think this is what should be implemented first, just because this is a more conservative alternative.

On the other hand, it makes writing internal helper macros harder, e.g.

macro m_helper() {
    struct #S;
}

macro m() {
    m_helper!(); // `S` has this context
    let s = S; // OK
}

fn main() {
    m!(); // `S` is not accessible here
    let s = S; // ERROR
}
alexreg commented 6 years ago

@petrochenkov Yeah, good point. I wonder if adding syntax like m_helper!#() (reuse the current context for the invocation) would help with that, or if there's a more elegant way that covers all use cases without too much pain?

pierzchalski commented 6 years ago

At that point you might start providing utility macros like lift!(m_helper!(...)) (move all tokens generated by m_helper!(...) up one context, essentially pretending foo! was called by main in the example above):

macro m_helper() {
    struct #S;
    struct T;
}

macro m() {
    lift!(m_helper!()); // Sets caller context of `m_helper!` to caller context of `m!`.
    let s = S; // Not OK: `S` is in callers context.
    let t = T; // Not OK: `T` is in `m_helper!` context.
}

fn main() {
    m!();
    let s = S; // OK: `S` is in `main` context.
}

As a bonus, I think this would be possible to implement with some minor extensions to the proc macro API (mostly getting and setting the parent of an arbitrary scope/span, rather than only having access to the def and call site scopes).

alexreg commented 6 years ago

@pierzchalski Yeah, that's not a bad idea at all. Thoughts, @jseyfried / @petrochenkov?

pierzchalski commented 6 years ago

@alexreg Actually, I just realised this is the use-case I was looking for for call_from in the proc macro RFC I put up.

alexreg commented 6 years ago

@pierzchalski I'll give that RFC a read tomorrow. Anyway, I certainly won't be including this lift macro or similar into my RFC; only the basic hygiene opt-out syntax. If the lift macro could go in another crate eventually, that would be ideal.

Incidentally, you seem to have a good knowledge of macro expansion and hygiene. Could I tempt you to contribute to https://github.com/rust-lang-nursery/rustc-guide/issues/15? :-)

pierzchalski commented 6 years ago

@alexreg Unfortunately my understanding of expansion and hygiene is a bit abstract - you'll notice the reference-level explanation in the RFC was rather thin! But if the RFC is accepted and I end up implementing it then I'd definitely like to document what I discover along the way.

dtolnay commented 6 years ago

I filed #49629 to consider dropping support for #[$m:meta] in favor of #[$($meta:tt)*] now that attributes are allowed to contain an arbitrary token stream.

dead-claudia commented 6 years ago

Stylistic question: why is it macro foo() and not macro foo!(), like the way it's called?

mark-i-m commented 6 years ago

I believe the question mark isn't considered part of the name.

dead-claudia commented 6 years ago

@mark-i-m You mean exclamation point, not question mark, right?

Also, in the docs, it usually refers to macros including the exclamation point as if it were part of the name, such as println! or vec!. As a concrete example, here's one page in the book that uses it exclusively.

mark-i-m commented 6 years ago

Oh, yes, I meant exclamation mark. I might be wrong, but I believe the compiler itself doesn't count the exclamation mark as party of the ident.

dead-claudia commented 6 years ago

@mark-i-m I would expect the compiler not to, but I'm speaking of the language itself, not the implementation. I'm suggesting matching what people think, not what computers process.

mark-i-m commented 6 years ago

That said, I don't see any reason why we cannot do something different from the internal representation.

If we did want to make ! part of the name, we would also want to do it in macro imports in the 2018 edition.

dead-claudia commented 6 years ago

@mark-i-m If you change how you do it in imports, you could even integrate them into use, to avoid the need to deal with separate syntax altogether:

use mod::foo::some_macro!;
use mod::foo::{some_fn, some_macro!};

Even as recently as a couple days ago, I had to google how to import a macro from a peer module, and this would make it a million times simpler. It also wouldn't require any special attribute or whatever - it'd just work.

Similarly, exporting macros could be simply pub macro foo!() or similar. Alternatively, because ! could serve as a delimiter, you could remove macro altogether and just do pub fn foo!($a: ident) { ... }, or you could reserve macro foo! for macros with multiple syntax rules (like vec!), and let fn foo! be for ones with only a single value (like how try! could be written if this RFC gets implemented.

mark-i-m commented 6 years ago

@isiahmeadows you can do use foo::bar::some_macro in the 2018 edition. It is part of the module/macro system changes being stabilized.

Long term IIUC, the plan is to get rid of the attributes altogether (currently, you still need them to export macros).

I strongly prefer not to conflate functions and macros. They are very different.

dhardy commented 6 years ago

use mod::foo::{some_fn, some_macro!};

The ! suffix is not technically necessary here, but since this may be the only thing pointing out that it is indeed a macro, seems like a good idea (if feasible and not too late — probably is too late).

But of course if foo and foo! are two distinct identifiers, they should not both be allowed to exist in the same scope.

mark-i-m commented 6 years ago

Macros live in another namespace. You can have a macro foo and a function foo and a type foo all coexisting.

Ekleog commented 6 years ago

This makes me wonder. If I have a crate mycrate defining function foo and procedural macro foo, then if I do use mycrate::foo, which one is imported into scope? both? (I'm starting from the idea that some day in the hopefully near future the -derive crates and the not--derive crates will be merge-able)

Adding the ! as something to remove the ambiguity would likely help (in addition to being more explicit about what happens, which is always a good thing).

Assuming it's still time to do it, though… otherwise, add the ! suffix as a possibility and a warning to push using it?

Nemo157 commented 6 years ago

@Ekleog yes, both (along with potentially a module foo and type foo as well).

SimonSapin commented 6 years ago

If I have a crate mycrate defining function foo and procedural macro foo

You cannot do that today:

error: `proc-macro` crate types cannot export any items other than functions tagged with `#[proc_macro_derive]` currently
 --> src/lib.rs:1:1
  |
1 | pub fn some_public_function() {}
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

However in an hypothetical future Rust where that restriction is lifted somehow, or if a non-proc-macro crate re-exports a proc macro and also a function of the same name, then yes use will import both.

Nokel81 commented 6 years ago

Should small proposals for Macros2.0 be brought up here or in a separate issue?

GrayJack commented 5 years ago

What the status of this?

petrochenkov commented 5 years ago

What the status of this?

Bugs in macros 2.0 are occasionally fixed, but the feature is not on the 2019 roadmap.

johnw42 commented 4 years ago

I'd like to clarify the rules about visibility and propose a mechanism for breaking hygiene in a controlled way.

I believe visibility should be resolved relative to where the macro is defined. For example, imagine a module like this:

fn foo<T>(arg: T) {...}
pub macro call_foo($arg:expr) => {
    foo($arg)
}

With the current rules, expanding call_foo! in a different module doesn't work because foo isn't public. It should be should be fine because foo is visible where call_foo! is defined.

As for breaking hygiene, I think some additional flexibility would be very useful, and I think some solution needs to be introduced at the same time as the new hygiene rules, because otherwise it becomes impossible to break hygiene in ways that macros can depend on now (e.g. when defining a new type). I propose a built-in macro whose argument is an identifier that will appear verbatim in the expansion of a macro where it's used. Let's call it verbatim! for now (although I don't think that's a great name for it). It could, for example, be used to define an "anaphoric" map whose argument is an implied closure with a fixed argument name:

pub macro map_it($iter:expr, $body:expr) {
    $iter.map(|verbatim!(it)| $body)
}

It could also be used to translate an old macro that defines a type in the current module:

// before:
macro_rules! define_foo {
    () => {
        struct Foo {...}
    }
}

// after:
pub macro define_foo() => {
    struct verbatim!(Foo) {...}
}

Special names like self and Self, which are treated as identifiers for the purpose of macro expansion, should be implicitly verbatim. With the current rules, this doesn't work because $body can't refer to self:

macro_rules! paranoid_method {
    ($name:ident, $body:expr) => {
        pub fn $name(&mut self) {
            self.verify_preconditions();
            $body;
            self.verify_postconditions();
        }
    }
}

Currently the only workaround is to pass self as an additional argument to the macro, which is pretty silly since passing any other identifier would produce a syntactically invalid expansion.

One last thing on my wishlist is to extend verbatim! to support multiple arguments, which are concatenated together to produce a new verbatim identifier. This could be used to do something like define a pair of related methods:

/// Defines a pair of conversion methods, `as_$name` and
/// `into_$name`, where `into_$name` consumes `self` and 
/// `as_$name` clones `self`.
macro conversions($name:ident, $ty:ty, $body:expr) => {
    pub fn verbatim!(into_, $name)(self) -> $ty { $body }
    pub fn verbatim!(as_, $name)(&self) -> $ty {
        self.clone().verbatim!(into_, $name)()
    }
}
spearman commented 4 years ago

I'd like to clarify the rules about visibility and propose a mechanism for breaking hygiene in a controlled way.

I believe visibility should be resolved relative to where the macro is defined. For example, imagine a module like this:

fn foo<T>(arg: T) {...}
pub macro call_foo($arg:expr) => {
    foo($arg)
}

With the current rules, expanding call_foo! in a different module doesn't work because foo isn't public. It should be should be fine because foo is visible where call_foo! is defined.

Could you give an example of this? This seems to work: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=a3e26640f0fb170b9a5a620250155f1c

You might need an absolute path $crate::path::to::foo($arg), but I'm not sure if this applies to decl macros or only macro_rules.

jjpe commented 4 years ago

It could, for example, be used to define an "anaphoric" map

I know it's just meant as an example, but anaphoric macros were a bad idea in Common Lisp, and I believe they're a bad idea here. The issue with them is that they don't scale: imagine that you end up in a situation where you have a nested usage of map_it. What does it refer to? Of course here it's easy enough to figure out in this contrived example, but at scale it causes ambiguity in the mind of the programmer.

verbatim!(it)

I would like to note that that requires macros to be expandable to identifiers. I don't believe they currently have that capacity, so that would need to be added as well. As for the name, I believe this is one instance where another cue from Lisps might help. While they use ~ (Closure) or , (CL) for interpolation, we could similarly use a sigil for identifier interpolation (leaving the default case to be verbatim instead). I'm not sure how that would interact with identifier concatenation though.

mayabyte commented 4 years ago

Weather or not anaphoric macros are allowed, the ability to create new identifiers from ones passed into the macro seems highly desirable. Consider the following case with macro_rules!:

macro_rules! example {
    ($($a:ident),*) => {
        $(
            // assume each $a is an identifier for a HashMap<K,V>, where each 
            // K and V can have a different type
            let side_effect = $a.remove("some_key");
            ... // do some other stuff
        ),*
        ...
        $( 
            // do some more things with each side effect created earlier, 
            // like putting them in a tuple
        ),*
    };
}

This doesn't work because side_effect has to have a fixed name, so it'd be shadowed each 'iteration'. (Achieving something like this is possible, but it's hacky and involves a lot of boilerplate).

Being able to do something like one of these:

let side_effect_${a} = ...
let verbatim!(side_effect_, $a) = ...

would be really nice.

eddyb commented 4 years ago

Doesn't side_effect_$ already successfully tokenize as side_effect_ $?

petrochenkov commented 4 years ago

Some status update (copypasted from Zulip https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/decl.20macro.20syntax/near/194354354):

macro items have multiple components to figure out before stabilizing, syntactic and semantic:

petrochenkov commented 4 years ago

One more issue is expanding fragments like $e:expr as token streams rather than AST pieces (this is also mentioned in the top comment and links to https://github.com/rust-lang/rust/issues/26361), but this is equally applicable to macro_rules where it should be doable in a (almost) backward-compatible way.

jgarvin commented 4 years ago

Do macros 2.0 address the self both is and isn't an identifier inconsistencies described here?: https://danielkeep.github.io/tlborm/book/mbe-min-non-identifier-identifiers.html

Or the confusing behavior of macro_rules invoking other macro_rules macros not behaving the same as calling directly?: https://danielkeep.github.io/tlborm/book/mbe-min-captures-and-expansion-redux.html

petrochenkov commented 4 years ago

Do macros 2.0 address the self both is and isn't an identifier inconsistencies described here?

self is always an identifier, keywords are a (reserved) subset of identifiers. I don't think macros 2.0 change anything here.

Or the confusing behavior of macro_rules invoking other macro_rules macros not behaving the same as calling directly?

This is not decided and needs design.

jgarvin commented 4 years ago

@petrochenkov are you saying the the author is mistaken, or that rust has changed since it was written? A number of confusing examples are provided at the link.

petrochenkov commented 4 years ago

@jgarvin Neither. The author is right in the sense that all the examples are correct etc., but he uses a different definition of "identifier" than the language and that's probably the source of the confusion. (Anyway, this is pretty off-topic for this issue.)

A1-Triard commented 4 years ago

Does macros 2.0 allow to parse generic parameters definition? With current macro_rules! the best approximation is < $( $lt:tt $( : $clt:tt $(+ $dlt:tt )* )? ),+ $(,)?>, but it does not cover all possible cases.

mexus commented 4 years ago

Does anybody happen to know if there are any plans to put "macros 2.0" into the 2021 roadmap?

mark-i-m commented 4 years ago

Given the ruat 2021 posts I've read so far and my vague knowledge of the state of things, it seems unlikely. There is still some design work needed and there doesn't seem to be anyone interested in pushing it over the finish line atm.