Off-by-one error when performing autocomplete on an expression inside a proc macro

[shadaj]

In recent versions of Rust Analyzer, inlay hints and completions seem to be broken for expressions passed to proc macros which emit the input expression as-is. See #16004 for hints, but completions also seem to be having trouble with the wrong expression being used to compute completions. For example, consider the following case:

Rust Analyzer correctly infers the type of v to be a Result<BytesMut, ...>, but the completions are for the BytesMut rather than the Result. This used to behave fine in previous versions of Rust Analyzer, so something seems to have changed.

rust-analyzer version: 0.3.1766-standalone

rustc version: rustc 1.76.0-nightly (3340d49d2 2023-12-12)

[Veykril]

Would be nice if you could come up with a concrete code sample where this happens and what proc-macro is being affected with this.

[shadaj]

@Veykril trying to put together a reproducer for the off-by-one issue, but for inlay hints https://github.com/shadaj/rust-analyzer-proc-macro-types should reproduce the issue (the outer variable in the example shows inline hints but the inner one does not).

[shadaj]

@Veykril managed to put together a reproducer for the off-by-one bug (in the same repo)! This one is a big more funky, it shows up when the input tokens are dumped into the output twice, once with all spans removed (by stringifying and parsing back the tokens) and once with the original spans. From my understanding, this should still run correctly though because the span-removed output should be completely disconnected from what Rust Analyzer uses for completions.

[shadaj]

Further simplified! It seems that .to_token_stream() is what messes up the completions. Here's the minimized reproducer for both inlay hints and completions.

Macro:

use proc_macro::TokenStream;
use quote::quote;

#[proc_macro]
pub fn passthrough(input: TokenStream) -> TokenStream {
    input
}

#[proc_macro]
pub fn passthrough_wrapped(input: TokenStream) -> TokenStream {
    let input_string: String = input.to_string();
    let input_2 = proc_macro2::TokenStream::from(input);
    quote! {
        {
            let _ = #input_string;

            {
                #input_2
            }
        }
    }.into()
}

Example:

use passthrough::{passthrough, passthrough_wrapped};

fn main() {
    let _test1 = 123;
    passthrough!({
        let _test2 = 123; // no inlay hints here
    });

    let res = Result::<i32, ()>::Ok(123);
    let _: i32 = passthrough_wrapped!(res.unwrap().abs()); // completions after `res.` show `abs` and similar APIs for i32
}

[Veykril]

Inlay hints not working in passthrough is expected, we don't show inlay hints inside function-like macro calls at all (and never did). The other one I'll have to investigate tomorrow.

[shadaj]

Hm, what about for closures though? That used to work, if passthrough!(...) is being passed to a function like map, the arguments to the closure used to have type hints displayed (as I remember). But now they don't seem to be showing up.

[shadaj]

Ah, hm maybe I am confused I'm looking back at some old screenshots and looks like the hints never displayed. Is there a way to get them to be displayed though? We're building a staged-programming API where we use macros to capture ASTs but also type-check expressions in place, so would be great to display inlay hints for those.

[Veykril]

No we never rendered them inside function like macro calls because its tricky to get them right in those cases. We do want to support the hints there when it makes sense for macro_rules! macros at least (if something got captured as a pattern, expr, etc). For procedural function-like ones I don't see how we can figure out that it makes sense to render the hints or not.

[xxshady]

Further simplified! It seems that .to_token_stream() is what messes up the completions. Here's the minimized reproducer for both inlay hints and completions.

Thanks for the finding! I probably never would have found it myself

[xxshady]

Nevermind, seems like in my case the cause is fs::write(...).unwrap();

I can reproduce it with

#[proc_macro_attribute]
pub fn inline_client_code(_: TokenStream, input: TokenStream) -> TokenStream {
    fs::write("out_client_wasm/something", "").unwrap();
    input
}

See issue 17234

[shadaj]

It seems that completions are still behaving strangely in the latest Rust Analyzer release, I'm getting the same issue as earlier where Rust Analyzer sometimes provides completions but they seem to be for the wrong place in the code.