search

firasuke / mussel

The shortest and fastest script to build working cross compilers targeting musl libc
ISC License
88 stars 12 forks source link

Patches for musl explicitly produce a broken libc #2

Closed richfelker closed 4 years ago

richfelker commented 4 years ago

All three patches (-ffast-math, powerpc, and powerpc64) are removing checks for conditions that produce a broken libc. By removing the checks, you're getting yourself (and your users) a broken libc. It will produce seriously wrong results, especially for the strtod and printf families, and might even be able to produce exploitable buffer overflows in them. It will also produce powerpc toolchains with silently broken ABI.

All of these patches should be removed, and the conditions that caused you to add them investigated and fixed. (They're indicative of your cross toolchain procedure not actually working.)

firasuke commented 4 years ago

I get your point and I respect it.

I just wish that musl-cross-make had a better documentation of how the build process is done.

I initially had planned that mussel uses a musl-cross-make style for the build steps, but failed to replicate what musl-cross-make does with POSIX Dash.

Here's the original script: https://gist.github.com/firasuke/82ee9ed032aaf61af26826e07a3db14d

I've also extracted the steps that mcm does for a cross compiled toolchain:

all
obj_binutils/.lc_configured
obj_binutils/.lc_built
obj_gcc/.lc_configured
obj_sysroot
obj_sysroot/usr
obj_sysroot/lib32
obj_sysroot/lib64
obj_sysroot/include
obj_gcc/gcc/.lc_built
obj_musl/.lc_configured
obj_sysroot/.lc_headers
obj_gcc/x86_64-linux-musl/libgcc/libgcc.a
obj_musl/.lc_built
musl
obj_sysroot/.lc_libs
obj_gcc/.lc_built
gcc
binutils
all
install
install-kernel-headers
install-musl
install-gcc
install-binutils
install

It shouldn't be this hard to understand musl-cross-make and I don't mind modifying mussel to an mcm styled script written in POSIX Dash as long as it remains a single pass for GCC.

richfelker commented 4 years ago

POSIX Dash

Small aside, but there's no such thing as "POSIX Dash". Dash is an implementaton of the standard POSIX shell, but putting #!/usr/bin/dash in your scripts is not portable and won't run on any system I've used recently. The way you write a portable script is just by only using portable constructs, not by hard-coding a specific implementation's name/pathname.

I've also extracted the steps that mcm does for a cross compiled toolchain: ...

This is good. Basically, for each of those rules, you want to look at what commands it invokes. Binutils is easy and completely independent of libc, so I'll focus on the gcc part. The keys are:

firasuke commented 4 years ago

Small aside, but there's no such thing as "POSIX Dash". Dash is an implementaton of the standard POSIX shell, but putting #!/usr/bin/dash in your scripts is not portable and won't run on any system I've used recently. The way you write a portable script is just by only using portable constructs, not by hard-coding a specific implementation's name/pathname.

Well it's to indicate that it was written in DASH, and is POSIX compliant nothing more. What's the portable version of a shebang, is it #!/bin/sh? I don't think such a thing as a portable shebang exists...

This is good. Basically, for each of those rules, you want to look at what commands it invokes. Binutils is easy and completely independent of libc, so I'll focus on the gcc part. The keys are:

  • obj_gcc/.lc_configured - running gcc's configure
  • obj_gcc/gcc/.lc_built - make all-gcc to compile gcc but not any target libs
  • obj_musl/.lc_configured - running musl'c configure using the xgcc from the above. Note that at this point there is no libgcc.a; musl doesn't care.
  • obj_sysroot/.lc_headers - running musl's make install-headers into build sysroot
  • obj_gcc/x86_64-linux-musl/libgcc/libgcc.a - running gcc's make all-target-libgcc with enable_shared=no forced, to build just libgcc.a.
  • obj_musl/.lc_built - building musl (now using libgcc.a that was just built)
  • obj_sysroot/.lc_libs - installing musl into sysroot
  • obj_gcc/.lc_built - running the remainder of the gcc build process to get all target libs (including shared libgcc, libstdc++, etc.)

I'll modify mussel back to an mcm style script, and report breakages here.

Also what's with:

obj_sysroot/usr: | obj_sysroot
    ln -sf . $@

Wouldn't passing --with-native-system-header-dir=/include to gcc remove the need for such a symlink? Another solution would be to modify lib64 to ../../lib and not ../lib in the t-linux64 file (in GCC's sources) since $XTARGET/bin/ld is called before bin/$XTARGET-ld (but that's the uglier one of the solutions).

Also, can you elaborate more on why we're using a build sysroot (obj_sysroot) in addition to a regular one $SYSROOT?

richfelker commented 4 years ago

Well it's to indicate that it was written in DASH, and is POSIX compliant nothing more. What's the portable version of a shebang, is it #!/bin/sh? I don't think such a thing as a portable shebang exists...

Indeed, POSIX explicitly/intentionally does not specify one, which is annoying. But in practice, #!/bin/sh is how you get a POSIX shell in any relevant real-world environment.

Wouldn't passing --with-native-system-header-dir=/include to gcc remove the need for such a symlink?

Yes, I think so. There's an open issue about this on the mailing list.

Another solution would be to modify lib64...

Yes, you're probably right about that too. It might not even be needed now that we're disabling multilib properly. However generally I prefer working around things gcc's build system does in a black-box manner over applying patchs to GCC, since the latter is less maintainable.

Also, can you elaborate more on why we're using a build sysroot (obj_sysroot) in addition to a regular one $SYSROOT?

This is to make it so that nothing in the build depends on files outside of the build tree. The final install location is supposed to be write-only so that the build isn't affected by files in there from previous builds, manually installed libraries, etc. However it may (probably does) make more sense to "install" binutils and gcc to a temp staging dir under the build dir, and only perform the final make install via cp -a or similar. This would also allow removing some unwanted files that the gcc/binutils install processes create before they get installed. However, it would also require more temp/working space.

firasuke commented 4 years ago

Indeed, POSIX explicitly/intentionally does not specify one, which is annoying. But in practice, #!/bin/sh is how you get a POSIX shell in any relevant real-world environment.

I see.

Yes, I think so. There's an open issue about this on the mailing list.

Are you referring to this?

Yes, you're probably right about that too. It might not even be needed now that we're disabling multilib properly. However generally I prefer working around things gcc's build system does in a black-box manner over applying patchs to GCC, since the latter is less maintainable.

Ah yes, you're setting this MULTILIB_OSDIRNAMES= to empty which removes the need for that.

This is to make it so that nothing in the build depends on files outside of the build tree. The final install location is supposed to be write-only so that the build isn't affected by files in there from previous builds, manually installed libraries, etc. However it may (probably does) make more sense to "install" binutils and gcc to a temp staging dir under the build dir, and only perform the final make install via cp -a or similar. This would also allow removing some unwanted files that the gcc/binutils install processes create before they get installed. However, it would also require more temp/working space.

This is probably where my confusion stems from.

I managed to get the mcm-like version of mussel to work up to the point when libgomp is being built within gcc, then I received the following error

 configure:3907: checking whether the C compiler works                                                                                                                
  3 configure:3929: /home/firasuke/Projects/test2/builds/gcc/./gcc/xgcc -B/home/firasuke/Projects/test2/builds/gcc/./gcc/ -B/x86_64-linux-musl/bin/ -B/x86_64-linux-musl/    lib/ -isystem /x86_64-linux-musl/include -isystem /x86_64-linux-musl/sys-include --sysroot=/home/firasuke/Projects/test2/bsysroot   -g -O2   conftest.c  >&5         
  2 /home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find -lgcc_s                                                                                         
  1 /home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find -lgcc_s

Also, in the README of musl-cross-make you state that:

One configured invocation them configures all the GNU toolchain components together in a manner 
that does not require any of them to be installed in order for the others to use them.

This is confusing because each package is configured in its own step, and they're not all configured with a single ./configure invocation as one might infer from the statement above.

firasuke commented 4 years ago

Gave it another go, and it seems to fail with the same error as above. It gets pretty far with the build. If we numbered the steps of mcm as follows:

1- all
2- obj_binutils/.lc_configured
3- obj_binutils/.lc_built
4- obj_gcc/.lc_configured
5- obj_sysroot
6- obj_sysroot/usr
7- obj_sysroot/lib32
8- obj_sysroot/lib64
9- obj_sysroot/include
10- obj_gcc/gcc/.lc_built
11- obj_musl/.lc_configured
12- obj_sysroot/.lc_headers
13- obj_gcc/x86_64-linux-musl/libgcc/libgcc.a
14- obj_musl/.lc_built
15- musl
16- obj_sysroot/.lc_libs
17- obj_gcc/.lc_built
18- gcc
19- binutils
20- all
21- install
22- install-kernel-headers
23- install-musl
24- install-gcc
25- install-binutils
26- install

Then it's failing at step no. 17, which is quite far (the last step that requires building), the step is obj_gcc/.lc_built and it's failing particularly after successfully building the shared libgcc, when building libgomp with the error that the C compiler doesn't work, and upon inspecting the config.log it seems that the ld-new inside binutils build directory isn't finding the currently built shared libgcc?

configure:3907: checking whether the C compiler works                                                                                                                
configure:3929: /home/firasuke/Projects/test2/builds/gcc/./gcc/xgcc -B/home/firasuke/Projects/test2/builds/gcc/./gcc/ -B/x86_64-linux-musl/bin/ -B/x86_64-linux-musl/    lib/ -isystem /x86_64-linux-musl/include -isystem /x86_64-linux-musl/sys-include --sysroot=/home/firasuke/Projects/test2/bsysroot   -g -O2   conftest.c  >&5         
/home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find -lgcc_s                                                                                         
/home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find -lgcc_s

Could it be because the shared libgcc wasn't installed before attempting to build the other libs (libstdc++-v3, libgomp...)?

richfelker commented 4 years ago

This is confusing because each package is configured in its own step, and they're not all configured with a single ./configure invocation as one might infer from the statement above.

Thanks for catching that. It's outdated documentation because mcm used to work that way. Eventually it was determined to be fragile and incompatible with any nontrivial amount of version skew between binutils and gcc, and contributors worked out a way to get it to use the newly built binutils without configuring binutils and gcc together in a combined source tree.

richfelker commented 4 years ago

The gcc target libs build process should work fine without anything being installed. It's very odd/surprising to me, though, that the command that fails is attempting to use -lgcc_s internally since it's not present on the command line. Was gcc configured to use shared libgcc by default?

firasuke commented 4 years ago

Not really, enable_shared=no was passed when building libgcc.a, and that's about it. The configuration is identical to that of mcm, and I'm unsure why it's trying to find the shared version of libgcc.

I think removing enable_shared=no allow for both the shared and static versions of libgcc to exist, and would resolve the error above. That, or adding enable_shared=no when building the final gcc at step 17, which I think is a bad idea.

firasuke commented 4 years ago

Ok, removing enable_shared=no prevented libgcc from being built at step 13, because it obviously requires a C library to build a shared version:

gcc_s.so.1.tmp ./libgcc_s.so.1 && (echo "/* GNU ld script"; echo "   Use the shared library, but some functions are only in"; echo "   the static library.  */"; echo "GROUP ( libgcc_s.so.1 -lgcc )" ) > ./libgcc_s.so
/home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find crti.o: No such file or directory
/home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find -lc
/home/firasuke/Projects/test2/builds/binutils/ld/ld-new: cannot find crtn.o: No such file or directory

And adding enable_shared=no to the final GCC built at step 17 also fails with the same error above cannot find -lgcc_s.

richfelker commented 4 years ago

enable_shared=no should be there only for the early libgcc.a part. However what I'm confused about is how the command:

/home/firasuke/Projects/test2/builds/gcc/./gcc/xgcc -B/home/firasuke/Projects/test2/builds/gcc/./gcc/ -B/x86_64-linux-musl/bin/ -B/x86_64-linux-musl/    lib/ -isystem /x86_64-linux-musl/include -isystem /x86_64-linux-musl/sys-include --sysroot=/home/firasuke/Projects/test2/bsysroot   -g -O2   conftest.c  >&5

is causing -lgcc_s to appear on the link command line (as shown in the error). It should not be linked by default, only as a dependency of other libs or by explicit request.

firasuke commented 4 years ago

Regarding this issue, I've:

The build works for all archs, and it's still a single pass to build GCC.

I have one question though, and I can be confident that the new modifications will fix the broken ABI.

Since musl-headers is the first package we have, how should it be configured? Is it ok not to pass ARCH, CC and CROSS_COMPILE if all we want are the headers? We're currently passing ARCH=$XTARGET, CC=gcc (which is the host's gcc) and we're leaving the value of CROSS_COMPILE empty, and regardless of the value we provide to ARCH, the generated config.mak will still use the ARCH of the host system (so if we were on x86-64 and we wanted to target aarch64, we would pass ARCH=aarch64 and CC=gcc and CROSS_COMPILE= to configure musl-headers initially just to get the headers from musl, we would still get ARCH=x86-64 in the generated config.mak, is passing these variables required if all we want is the headers? Can't we just do ./configure (without any variable) then make install-headers if all we want is the headers?

richfelker commented 4 years ago

No, the headers differ per arch. You can skip running configure at all, and do make install-headers ARCH=aarch64 or whatever, and it should work.

firasuke commented 4 years ago

Alright, I'm now skipping the configuration of musl-headers and relying on make ARCH=$XARCH prefix=/usr DESTDIR=$MSYSROOT install-headers to install musl's headers per targetted arch.

I'm closing this issue as all the changes you proposed have been made.

Thanks for your time and effort.