Allow proc macros to expose their grammar to rust analyzer for syntax error understanding

[lahwran]

My understanding of rust-analyzer - correct me if I'm wrong - is that it can partially analyze code with syntax errors as long as it doesn't use proc macros, and that this is core functionality as part of providing completions for partially written lines such as let x =. If that is correct, what would it take for proc macros to expose their grammar to get the same level of support as the rest of rust when code is incomplete? I'm most interested in getting tab completion for identifiers in proc macros, because I have several different proc macros I'd like to use very heavily and don't want to lose completion functionality in them.

[flodiebold]

The proc macros need to be written in a way that passes through even incomplete syntax. Most proc macros (maybe all) will probably fail to expand if their input is not syntactically correct. They would instead need to try to expand as best as possible. Ideally there should be a way for proc macros to return an expansion, but also signal an error (this might be an interesting thing to prototype / think about, but would require support in the compiler); but for proc macros where parts of the input are just normal Rust syntax that's passed through to a certain place in the output, that's not really necessary (they can just pass through the invalid syntax and rely on the compiler to complain afterwards).

I think it'd be interesting to look at concrete cases that aren't working right now and what it would take to make them work.

(I think "exposing their grammar" is probably not necessary or the right approach, except maybe for syntax highlighting completely custom DSLs...)

[jonas-schievink]

I don't think this is much of a rust-analyzer issue, the proc macro ecosystem just needs to become more IDE-friendly

[edwin0cheng]

Totally agree on @jonas-schievink and @flodiebold .

And one thing we could improve here is : https://github.com/rust-analyzer/rust-analyzer/blob/607b9ea160149bacca41c0638f16d372c3b235cd/crates/proc_macro_srv/src/rustc_server.rs#L415

We do not implement all span information for proc-macro (which is hard) such that error recovery is impossible in some cases.

[matklad]

@ogoffart of https://sixtyfps.io came up with ingeniously simple scheme for this:

I have some ideas on how rust-analyzer could use some kind of "in-band-signaling" to help with auto-complete within proc-macro. I don't know if you ever considered it. For example, consider some_macro!(hello i am a macro) The cursor is located just after "am", and the user press ctrl+space to auto-complete. rust-analyzer would send this token stream to the macro: "hello i __rust_analyzer_completion a macro" if the macro expand to some rust code, rust_analyzer would find the rust_analyzer_completion token in the expansion, and run normal rust completion code to auto-complete the rust code. But the macro would also understand the rust_analyzer_completion within the token stream, and instead of expanding to its normal expansion, expand to something like __rust_analyzer_completion![ "am", "have", "foo", "bar" ]

Notably, we already do the __rust_analyzer_completion thing here:

https://github.com/rust-analyzer/rust-analyzer/blob/557c1e36ddbb19cc76f7a1f04d1b327942aafcb9/crates/completion/src/context.rs#L107-L114

[matklad]

cc @vlad20012

[ModProg]

I could imagine one case where exposing something like a gramar would make sense, and that is proc macros that use a relatively simple syntax to just accept some attributes like e.g.

• #[attr(1, 4, 5)]
• #[attr(prop="<syn::Path>")]
• #[attr(prop=syn::Expr)]
• #[attr(prop(propa=<usize>, probb=<str>))]

[jssblck]

i am writing a custom dsl macro and would love to add intellisense support for the dsl, so i found this issue. has there been any movement on this elsewhere, or is this issue still the current state?

i'd really like to make the experience as good as i can get it, the dsl is actually effectively rust syntax but the things being named in the dsl aren't actually in the program.

right now i'm vaguely thinking i can generate the things being named into actual syntax, then produce partially formed token streams in an effort to contort Rust and R-A into producing completions and useful errors... i'm not sure how feasible that is but it's the closest i've got

[Veykril]

The best info on this we have currently is https://github.com/rust-lang/rust-analyzer/discussions/15452.

[jssblck]

oh awesome, thanks! sounds like the rough idea should work then!

[chipnertkj]

I have opened a discussion thread in the Rust Internals Forum related to this issue. If there is something constructive you could add to the discussion (criticism, possible solutions, concerns, use cases), please head through the link below. https://internals.rust-lang.org/t/discussion-adding-grammar-information-to-procedural-macros-for-proper-custom-syntax-support-in-the-toolchain/21496