Closed Ilia-Kosenkov closed 3 years ago
Hmmm last time i checked this was not needed on x64. See also https://github.com/r-rust/hellorust which passes CI using rtools40. Where are you getting this error ?
Op zo 17 jan. 2021 20:12 schreef Ilia notifications@github.com:
We attempt to bridge R and rust and use rtools for building rust code. However, we encountered a linker error reporting there is no libgcc_eh when targeting x86_64.
Preliminary googling suggested that if gcc is built with --disabled-shared, libgcc_eh is not emitted.
We found that mingw32 version of gcc contains libgcc_eh, which is an exact copy of libgcc, and rust libraries targeting i686 can be successfully built with the help of rtools.
PKGBUILD of gcc contains a "hack" implemented for mingw32 (comment suggests it is needed for rust compiler), but no such hack is present for mingw64.
We confirm that our linking issues are resolved by duplicating libgcc as libgcc_eh in mingw64 gcc folder. Therefore, we would like to have this "hack" implemented for mingw64 as well.
A possible implementation based on existing mingw32 code:
if [ "${MINGW_CHOST}" = "x86_64-w64-mingw32" ]; then cp ${pkgdir}${MINGW_PREFIX}/lib/gcc/${MINGW_CHOST}/${pkgver}/libgcc.a ${pkgdir}${MINGW_PREFIX}/lib/gcc/${MINGW_CHOST}/${pkgver}/libgcc_eh.a fi }
Alternatively, if condition can be removed so the cp command is executed for both architectures.
Relevant discussions can be found in extendr/rextendr#19 https://github.com/extendr/rextendr/issues/19.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/r-windows/rtools-packages/issues/185, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABUZ7YTLQEC2HLT7FZDPU3S2MZBHANCNFSM4WGKSARA .
@jeroen,
In all cases where we build targeting x86_64-pc-windows-gnu
.
Take for instance https://github.com/extendr/libR-sys
With default rust toolchain (stable-msvc)
$env:PATH = "$env:RTOOLS40_home\bin\mingw64;$env:PATH"
(cargo test --target x86-64-pc-windows-gnu -- --nocapture) *> log_64.dat
Same thing, but for i686
:
$env:PATH = "$env:RTOOLS40_home\bin\mingw32;$env:PATH"
(cargo test --target i686-pc-windows-gnu -- --nocapture) *> log_32.dat
Same x86_64
setup, but using `libgcc_eh" hack:
cp $env:RTOOLS40_HOME\mingw64\lib\gcc\x86_64-w64-mingw32\8.3.0\libgcc.a $env:RTOOLS40_HOME\mingw64\lib\gcc\x86_64-w64-mingw32\8.3.0\libgcc_eh.a
$env:PATH = "$env:RTOOLS40_home\bin\mingw64;$env:PATH"
(cargo test --target x86-64-pc-windows-gnu -- --nocapture) *> log_64_hack.dat
What I observed is the need for libgcc_eh
is not universal, perhaps in your r-rust/hellorust
it is not invoked.
Also, you are using stable-gnu
as default toolchain, we rely on stable-msvc
for a number of reasons.
We also provide options to use different toolchains if a user so desires, so it is important for us that it works in such configuration.
There is also a relevant discussion about rust gnu toolchains here https://github.com/rust-lang/cargo/issues/8990.
EDIT01: to test libR-sys
, just install rustup
, set default toolchain to stable-msvc
and add x86_64-pc-windows-gnu
and i686-pc-windows-gnu
targets to said toolchain. You also need to have $R_HOME\bin\x64
or $R_HOME\bin\i386
on your path for tests to grab R.dll
.
Sorry i'm completely lost in all your topics what you're trying to do.
Rtools is a gcc/mingw-w64 toolchain for building R packages, I don't understand why/how you are using the MSVC version of rust with rtools.
Am I correct to understand the problem does not appear when you're using the the native gnu version of rust, but only when using MSVC to cross-compile for mingw-w64? And if you're already cross-compiling, why do you need rtools in the first place?
@jeroen, I am sorry for the messy explanation, let me try one more time. The rust code that goes into an R package makes use of a number of rust crates that provide convenient instruments for rust developers, like macros for automatic exporting and auto-generated bindings to R C API.
These crates are built targeting either x86_64-pc-windows-gnu
or i686-pc-windows-gnu
depending on the architecture.
So, to target *-gnu
we need mingw
toolchains. rust stable-x86_64-pc-windows-gnu
toolchain provides tools for building x86_64
target, however, we are unable to cross-compile to i686
using only rust toolchain (which, apparently, is possible in https://github.com/r-rust/hellorust). We can, in principle, use another toolchain, stable-i686-pc-windows-gnu
, to compile for i686
, but this means additional tweaking of cargo and asking users to install and configure two toolchains.
As an alternative, we can use mingw toolchain provided elsewhere for cross-compilation. One possibility is rtools, as it should be already available and configured on a Windows machine that attempts to build non-R code for packages. When using mingw toolchain from rtools, we found that the "hack" implemented for mingw32 is also required for mingw64 to enable simple cross-compilation. Therefore, we suggest to extend this "hack" to include both mingw64 and mingw32.
To illustrate the issue with the gnu
cross-compilation, here is an example output of building our core rust package:
(cargo +stable-x86_64-pc-windows-gnu test --target i686-pc-windows-gnu -- --nocapture) *> log.dat
Because this package is referenced in rust-R packages, any build errors occuring at this step will propagate and ruin builds for any derived product.
What I observed is the need for
libgcc_eh
is not universal, perhaps in yourr-rust/hellorust
it is not invoked.
I found this actually happen even on r-rust/hellorust if you run cargo test
instead of cargo build
. Comparing the actual rustc
commands executed, it seems cargo test
doesn't pass --crate-type lib
to rustc
. I don't know why this matters and this is beyond my knowledge, but you can confirm the same error happens here:
commit: https://github.com/yutannihilation/hellorust/commit/ce49dc598b519a57b363f18de43c72979797262d job run: https://github.com/yutannihilation/hellorust/runs/1721221185?check_suite_focus=true#step:8:212
diff (this one is of extendr/libR-sys
, but it's probably mostly the same in the hellorust's case):
--- ".\\cargo_build.txt" 2021-01-18 21:22:57.629898200 +0900
+++ ".\\cargo_test.txt" 2021-01-18 21:21:02.806275900 +0900
@@ -3,12 +3,12 @@
--edition=2018 src\lib.rs
--error-format=json
--json=diagnostic-rendered-ansi
- --crate-type lib
- --emit=dep-info,metadata,link
+ --emit=dep-info,link
-C embed-bitcode=no
-C debuginfo=2
- -C metadata=6227798a48c8b7f3
- -C extra-filename=-6227798a48c8b7f3
+ --test
+ -C metadata=1f3043e575d47cae
+ -C extra-filename=-1f3043e575d47cae
--out-dir C:\path\to\libR-sys\target\i686-pc-windows-gnu\debug\deps
--target i686-pc-windows-gnu
-C incremental=C:\path\to\libR-sys\target\i686-pc-windows-gnu\debug\incremental
So, this probably means,
Am I correct to understand the problem does not appear when you're using the the native gnu version of rust,
you are correct as long as we only build staticlib, but we might run tests or dynamic library (as an intermediate artifact?) (Sorry, this seems my misunderstanding).
Just as an aside, the hellorust example also uses the msvc toolchain: https://github.com/r-rust/hellorust/runs/1482980650?check_suite_focus=true#step:4:15
To be clear, the extendr project has three distinct use cases with somewhat differing requirements:
rust_function()
In cases 2 and 3, linking is done by cargo
, and it insists on injecting -lgcc_eh
. Not entirely sure why, but I'll also note that we're just working around a gcc bug here, gcc should figure out that the symbols are already available in libgcc but it doesn't when cross-compiling.
Here's a post from 2003 where they discuss the gcc workaround for static builds and that it doesn't work for cross-compiles: https://sourceware.org/legacy-ml/libc-alpha/2003-09/msg00100.html
But it sounds like you don't need rtools for 2 and 3 really? I mean I don't mind, but I'm afraid you're going to rely on a "hack" that won't work on other systems. Does this same problem appear when you use the upstream msys2 compilers (of which rtools40 is simply a variation)?
@jeroen,
The latest version of msys2 has gcc 10.2.0 and it includes ligcc_eh.a
both in mingw32 and mingw64.
Up until now we were using msys2 successfully, but we want to switch to rtools to ditch msys2 dependency for endusers.
The reason Rtools needs this hack is because gcc is built with --disable-shared
.
@Ilia-Kosenkov Just to be clear: It's not an issue of gcc 10.2 vs 8.3. It's --disable-shared
vs --enable-shared
. When you compile gcc with --disable-shared
then the static compile chain breaks for certain use cases, ironically.
@clauswilke,
Yes I know, I just replied to @jeroen stating that msys2 is configured in such a manner that gcc emits libgcc_eh
and we experience no issues when using msys2, but not rtools.
@Ilia-Kosenkov Since @jeroen said he is fine in principle with the proposed solution, would you mind preparing a PR? I think a simple unconditional copy would be best. This will continue to work even if in the future there's a different Windows architecture that needs to be supported (e.g. ARM).
It would also be helpful to expand a bit on what's happening in the comment, so in the future people can understand why this was added and what exactly it does.
@clauswilke, @jeroen, I actually have a fork where I tested this thing (by building gcc and later installing it for rtools using pacman as per guidelines). https://github.com/Ilia-Kosenkov/rtools-packages/commit/e1fd3bece3abf218efe2b370fd666e99674d40f7
I will open a PR and the discussion can be moved there.
We attempt to bridge R and rust and use rtools for building rust code. However, we encountered a linker error reporting there is no
libgcc_eh
when targetingx86_64
.Preliminary googling suggested that if gcc is built with
--disabled-shared
,libgcc_eh
is not emitted. https://github.com/r-windows/rtools-packages/blob/df6b4363b0422db30abf7dd78378022666a24ab4/mingw-w64-gcc/PKGBUILD#L167We found that mingw32 version of gcc contains
libgcc_eh
, which is an exact copy oflibgcc
, and rust libraries targetingi686
can be successfully built with the help of rtools.PKGBUILD
of gcc contains a "hack" implemented for mingw32 (comment suggests it is needed for rust compiler), but no such hack is present for mingw64. https://github.com/r-windows/rtools-packages/blob/df6b4363b0422db30abf7dd78378022666a24ab4/mingw-w64-gcc/PKGBUILD#L305-L309We confirm that our linking issues are resolved by duplicating
libgcc
aslibgcc_eh
in mingw64 gcc folder. Therefore, we would like to have this "hack" implemented for mingw64 as well.A possible implementation based on existing
mingw32
code:Alternatively,
if
condition can be removed so thecp
command is executed for both architectures.Relevant discussions can be found in https://github.com/extendr/rextendr/issues/19.