[DESIGN BUG] declarative macros lack of neat way to simulate lookahead within rust grammer syntax `const X: Y`

[loynoir]

I tried this code:

macro match and echo non const X: Y pattern is fine.

    macro_rules! echo1 {
        (pub type $ident:ident<$($gi:ident),*> = $($tt:tt)*) => {
            pub type $ident<$($gi),*> = $($tt)*;
        };
    }

    echo1!(pub type Foo1<T, N> = (T, N));

But within current rust desgin, to let declarative macro match lookahead within rust grammer syntax const X: Y

• or input rust code, and do lookahead simulation

• or input lookahead free dsl, and output lookahead code

    macro_rules! echo2 {
        (@derive_foo pub type $ident:ident<$($gi:ident $(lookahead_qualifier=$gq:tt)? $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($($gq)? $gi $(: $gt)?),*> = $($tt)*;
        };

        (@derive_bar pub type $ident:ident<$($gi:ident $(lookahead_qualifier=$gq:tt)? $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($($gq)? $gi $(: $gt)?),*> = $($tt)*;
        };

        (@lookahead_workaround pub type $ident:ident<$($gi:ident $(lookahead_qualifier=$gq:tt)? $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($($gq)? $gi $(: $gt)?),*> = $($tt)*;
        };

        (pub type $ident:ident<$($(const)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(const)? $gi $(: $gt)?),*> = $($tt)*;
        };
    }

    // // TODO: https://github.com/rust-lang/rust/issues/130928
    // echo2!(pub type Foo2<T, const N: usize> = [T; N]);

    echo2!(@lookahead_workaround pub type Foo2<T, N lookahead_qualifier=const: usize> = [T; N]);

I expected to see this happen:

• Neat way to match lookahead within rust grammer syntax const X: Y

Instead, this happened:

• Need to simulate lookahead within rust grammer syntax const X: Y

• Leads to ugly simulation code.

• Thus I think it's a design bug.

Meta

rustc --version --verbose:

rustc 1.83.0-nightly (9b72238eb 2024-09-14)
binary: rustc
commit-hash: 9b72238eb813e9d06e9e9d270168512fbffd7ee7
commit-date: 2024-09-14
host: x86_64-unknown-linux-gnu
release: 1.83.0-nightly
LLVM version: 19.1.0

Backtrace

``` ```

related

Also found similar issue back to 2021 in pin-project-lite.

So, I guess there is no way yet.

https://github.com/taiki-e/pin-project-lite/issues/62

[pacak]

This part is not going to work: $(const)?

According to https://doc.rust-lang.org/reference/macros-by-example.html

Each repetition in the transcriber must contain at least one metavariable to decide how many times to expand it.

But you can totally match it using something like incremental TT muncher.

[loynoir]

But problem is none of MacroFragSpec is working.

I try with EVERY macro frag spec, end up with open an issue here.

  block | expr | ident | item | lifetime | literal | meta | pat | pat_param | path | stmt | tt | ty | vis

    macro_rules! echo_block {
        (pub type $ident:ident<$($(gm:block)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_expr {
        (pub type $ident:ident<$($(gm:expr)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_ident {
        (pub type $ident:ident<$($(gm:ident)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_item {
        (pub type $ident:ident<$($(gm:item)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_lifetime {
        (pub type $ident:ident<$($(gm:lifetime)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_literal {
        (pub type $ident:ident<$($(gm:literal)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_meta {
        (pub type $ident:ident<$($(gm:meta)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_pat {
        (pub type $ident:ident<$($(gm:pat)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_pat_param {
        (pub type $ident:ident<$($(gm:pat_param)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_path {
        (pub type $ident:ident<$($(gm:path)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_stmt {
        (pub type $ident:ident<$($(gm:stmt)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_tt {
        (pub type $ident:ident<$($(gm:tt)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    macro_rules! echo_ty {
        (pub type $ident:ident<$($(gm:ty)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(gm)? $gi $(: $gt)?, )*> = $($tt)*;
        };
    }

    echo_block!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_expr!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_ident!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_item!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_lifetime!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_literal!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_meta!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_pat!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_pat_param!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_path!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_stmt!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_tt!(pub type Foo2<T, const N: usize> = [T; N]);
    echo_ty!(pub type Foo2<T, const N: usize> = [T; N]); 

[pacak]

I try with EVERY macro frag spec, end up with open an issue here.

You should be able to match const with tt, but echo_tt! won't work because macro_rules performs no lookahead and you are trying to match both T and const N: usize with the same pattern.

Try searching for incremental tt muncher and implementing one.

[loynoir]

because macro_rules performs no lookahead

Awesome, inspire me to workaround with lookbehind.

    macro_rules! echo2 {
        (@foo pub type $ident:ident<$($gi:ident $(lookahead_qualifier=$gq:tt)? $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($($gq)? $gi $(: $gt)?),*> = $($tt)*;
        };

        (@bar pub type $ident:ident<$($gi:ident $(lookahead_qualifier=$gq:tt)? $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($($gq)? $gi $(: $gt)?),*> = $($tt)*;
        };

        (@lookahead_workaround pub type $ident:ident<$($gi:ident $(lookahead_qualifier=$gq:tt)? $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($($gq)? $gi $(: $gt)?),*> = $($tt)*;
        };

        (pub type $ident:ident<$($(const)? $gi:ident $(: $gt:ty)?),*> = $($tt:tt)*) => {
            pub type $ident<$($(const)? $gi $(: $gt)?),*> = $($tt)*;
        };
    }

    // // TODO: https://github.com/rust-lang/rust/issues/130928
    // echo2!(pub type Foo2<T, const N: usize> = [T; N]);

    echo2!(@foo pub type Foo2<T, N lookahead_qualifier=const: usize> = [T; N]);

[loynoir]

But still think rust should provide basic look ahead like $(const)?.

Because look ahead DOES exists in rust language syntax.

Otherwise, declarative macro need to simulate look ahead leads to unreadable unmaintainable look ahead simulation code.