rust: Uses wrong target architecture for proc-macro targets when cross-compiling

[sdroege]

Describe the bug

meson currently always uses the host architecture for building proc-macro targets, which is wrong. proc-macro targets are loaded and run by the compiler during compilation so must be compiled for the build architecture.

Also in extension this means that all dependencies of the proc-macro target also must be compiled for the build architecture, and with this it can happen that dependencies have to be compiled for both the build and host architecture (if they're used by targets for both).

To Reproduce

Try cross-compiling test cases/rust/18 proc-macro and see how it fails:

[1/2] rustc --target aarch64-unknown-linux-gnu -C linker=aarch64-linux-gnu-gcc --color=always --crate-type proc-macro -g -C relocation-model=pic --crate-name proc_macro_examples --emit dep-info=proc_macro_examples.d --emit link -o libproc_macro_examples.so -C prefer-dynamic ../proc.rs
[2/2] rustc --target aarch64-unknown-linux-gnu -C linker=aarch64-linux-gnu-gcc --color=always --crate-type bin -g --crate-name main --emit dep-info=main.d --emit link -o main --extern proc_macro_examples=libproc_macro_examples.so -L . -C prefer-dynamic -C 'link-arg=-Wl,-rpath,$ORIGIN/:/home/slomo/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' -C 'link-arg=-Wl,-rpath-link,/home/slomo/Projects/meson/test cases/rust/18 proc-macro/build-cross/:/home/slomo/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' ../use.rs
FAILED: main 
rustc --target aarch64-unknown-linux-gnu -C linker=aarch64-linux-gnu-gcc --color=always --crate-type bin -g --crate-name main --emit dep-info=main.d --emit link -o main --extern proc_macro_examples=libproc_macro_examples.so -L . -C prefer-dynamic -C 'link-arg=-Wl,-rpath,$ORIGIN/:/home/slomo/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' -C 'link-arg=-Wl,-rpath-link,/home/slomo/Projects/meson/test cases/rust/18 proc-macro/build-cross/:/home/slomo/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' ../use.rs
error[E0463]: can't find crate for `proc_macro_examples`
 --> ../use.rs:1:1
  |
1 | extern crate proc_macro_examples;
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: cannot determine resolution for the macro `make_answer`
 --> ../use.rs:4:1
  |
4 | make_answer!();
  | ^^^^^^^^^^^
  |
  = note: import resolution is stuck, try simplifying macro imports

error[E0425]: cannot find function `answer` in this scope
 --> ../use.rs:7:20
  |
7 |     assert_eq!(42, answer());
  |                    ^^^^^^ not found in this scope

error: aborting due to 3 previous errors

When running the commands manually and selecting the correct target for the proc-macro it builds fine:

rustc --color=always --crate-type proc-macro -g -C relocation-model=pic --crate-name proc_macro_examples --emit dep-info=proc_macro_examples.d --emit link -o libproc_macro_examples.so -C prefer-dynamic ../proc.rs
rustc --target aarch64-unknown-linux-gnu -C linker=aarch64-linux-gnu-gcc --color=always --crate-type bin -g --crate-name main --emit dep-info=main.d --emit link -o main --extern proc_macro_examples=libproc_macro_examples.so -L . -C prefer-dynamic -C 'link-arg=-Wl,-rpath,$ORIGIN/:/home/slomo/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' -C 'link-arg=-Wl,-rpath-link,/home/slomo/Projects/meson/test cases/rust/18 proc-macro/build-cross/:/home/slomo/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' ../use.rs

CC @dcbaker

[dcbaker]

I'm wondering if the approach of using a regular target for proc_macro was the wrong one, maybe it would make more sense to add proc_macro to the rust module, where we can just enforce that everything is for the build target and not the host target

[sdroege]

That would make sense, yes. Does meson have any kind of concept of a "build dependency" for something, or a "build tool" that has to be built itself during the build to produce some other target?

[eli-schwartz]

Yes? You just create e.g. an executable with type native: true. You don't need anything fancier. A custom_target can use any executable, even a native one, as its command: kwarg.

[sdroege]

Sure but that seems extremely generic and doesn't seem specific enough for exposing proc-macros. So maybe a special custom target as part of the Rust module makes most sense after all, which then could enforce that it's only used in the right places and is always compiled natively.

[eli-schwartz]

It's not entirely clear to me how this all works, but I think basically you need this:

pm = shared_library(
  'proc_macro_examples',
  'proc.rs',
  rust_crate_type : 'proc-macro',
)

to be this:

pm = shared_library(
  'proc_macro_examples',
  'proc.rs',
  rust_crate_type : 'proc-macro',
  native: true,
)

except that because it is a proc-macro crate it automatically forces native: true and also despite that usually native and cross targets cannot be linked into the same final executable or shared library, a proc-macro can because it's actually

loaded and run by the compiler during compilation

and not in fact linked into the executable?

[dcbaker]

yeah, exactly. It's more like a compiler extension

[dcbaker]

One thing that this makes me realize we're probably going to have to solve is that currently a subproject can only be built for one machine (ie, either host or build) but not both

[sdroege]

I guess this could be made to work with the current mechanism but would require adding a couple of special cases all over the place. Maybe it's not too bad, worth a try.

One thing that this makes me realize we're probably going to have to solve is that currently a subproject can only be built for one machine (ie, either host or build) but not both

That's a limitation of cargo too AFAIU. Either your crate is a proc-macro or it's not, but you can't make it both at once.

What situation do you imagine where this could be useful?

[eli-schwartz]

It's not really rust-specific, but. Imagine you have, say, a program that acts as a codegen tool. myproject-generate-sources.c.

This program makes heavy use of glib for the general reasons that people make general use of glib. So you need to depend on a dependency('glib-2.0', native: true). But the produced sources are used to create "myproject", an executable that also makes heavy use of glib, and thus requires dependency('glib-2.0', native: false).

So this works fine in a native build, because both native: false and native: true are the same machine and the same *.so files. You can build glib as a subproject and this works. But it fails in cross compilation: you can get both native and cross glib from the system and from a sysroot, but what do you do about building one (cross) glib subproject for myproject.exe, and one (native) glib subproject for myproject-generate-sources.exe?

[sdroege]

I see, I didn't get that connection. That's problematic indeed :) There will be many cases where a proc-macro crate will have a dependency that is also used by normal build targets, so that dependency would have to be built for both.

[sdroege]

So making the main part of this issue work was actually less bad than expected. From what I could find, only 2 places needed a bit of special casing. See https://github.com/mesonbuild/meson/pull/11743

I'll create another issue for the bigger problem of requiring multiple builds of a dependency.

[sdroege]

See https://github.com/mesonbuild/meson/issues/11744

[vanc]

The PR doesn't seem to work. I pulled the patch into the meson master, and changed the proc-macro crate with native: true, but got an error in setup time. I was trying to port the foreign-types crate which depends on foreign-types-macros crate.

../subprojects/foreign-types-macros-0.2.3/meson.build:16:23: ERROR: Tried to mix libraries for machines 0 and 1 in target 'foreign_types_macros' This is not possible in a cross build.

[sdroege]

Why don't you comment on the PR? :)

[sdroege]

Also can you make a testcase available? For me the PR still works with test cases/rust/18 proc-macro and cross-compiling from Linux/x86-64 to Linux/aarch64.

[vanc]

Why don't you comment on the PR? :)

Sorry. I should've done that.

[vanc]

Also can you make a testcase available? For me the PR still works with test cases/rust/18 proc-macro and cross-compiling from Linux/x86-64 to Linux/aarch64.

If the proc-macro crate has no dependencies, it's fine. Perhaps I hit this issue #11744.

[sdroege]

Yeah if it has dependencies you need to make sure that all dependencies are native: true too (and that they're not used for anything native: false, i.e. that they're only used for proc macros).

[xclaesse]

I think this is fixed by https://github.com/mesonbuild/meson/pull/11714. We now have rust.proc_macro() which is always native build.