Bootstraping GCC 13.2 with clang 17?

ilg-ul commented 7 months ago

I'm trying to build GCC 13.2 with the latest clang available in Homebrew (17.0.6) and the build fails with multiple errors, but apparently the cause is a C++ header file:

In file included from /Users/ilg/Work/xpack-dev-tools-build/gcc-13.2.0-2/darwin-x64/sources/gcc-13.2.0/gcc/cp/mapper-client.cc:31:
In file included from /Users/ilg/Work/xpack-dev-tools-build/gcc-13.2.0-2/darwin-x64/sources/gcc-13.2.0/gcc/system.h:227:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/map:2529:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/functional:526:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/__functional/boyer_moore_searcher.h:27:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/vector:321:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/__format/formatter_bool.h:20:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/__format/formatter_integral.h:32:
In file included from /Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/locale:202:
/Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/__locale:657:37: error: expected ';' at end of declaration list
  657 |     char_type toupper(char_type __c) const
      |                                     ^
/Users/ilg/.local/homebrew/hb/Cellar/llvm/17.0.6_1/bin/../include/c++/v1/__locale:663:48: error: too many arguments provided to function-like macro invocation
...

rm gcc.pod
make[2]: *** [all-stage1-gcc] Error 2
make[1]: *** [stage1-bubble] Error 2
make: *** [all] Error 2

I tried the same script with llvm@16 and the build passed.

I'm using an Intel macOS 14.3.1 and CLT 15.1. The GCC configuration has an explicit --enable-bootstrap and the same patches as used by Homebrew.

I asked a friend to try a similar build (GCC 13.2 with clang 17) on an up to date Linux Arch, and it passed, so the problem might be in the GCC platform dependent code.

If this issue is confirmed, perhaps there is still time to fix it before 13.3 is out.

iains commented 7 months ago

I think CLT 15.1 has multiple issues, I would recommend that you update to 15.3.

We have some patches to help support macOS 14 better - they are already on the upstream releases/gcc-13 branch which will become gcc-13.3 probably some time in May.

ilg-ul commented 7 months ago

Can you confirm that you managed to build releases/gcc-13 with Homebrew clang 17 and CLT 15.3?

iains commented 7 months ago

Can you confirm that you managed to build releases/gcc-13 with Homebrew clang 17 and CLT 15.3?

My builds are all done with upstream (or specific patched branches) usually with an earlier GCC as the bootstrap compiler (since I test Ada and D as well as the other languages). I do occasionally bootstrap with Apple clang (the same version that is in the CLT in use).

So , to be clear, I have neither home-brew nor macports in my PATH. I am not in a position to answer home-brew questions, those are better on the home-brew site.

ilg-ul commented 7 months ago

The question is not Homebrew specific, but clang 17 specific. I first encountered it with my own clang 17 distribution, but re-tested with Homebrew, to have a better reference.

The issue occurs during bootstrap, so if you use an older GCC, not clang 17, you cannot detect it. Even clang 16 is ok.

And you do not need to have Homebrew in your PATH, you can very well install Homebrew in a custom location and explicitly add it to your PATH only when needed. (I have multiple instances below /Users/ilg/.local/homebrew).

@fxcoudert, did you manage to build the current Homebrew GCC 13.2 with clang 17 and CLT 15.3?

iains commented 7 months ago

Perhaps this will be a disappointment - but this will not count as a bug in either upstream GCC or the development branches here.

the supported bootstrap toolchains are whatever Apple releases + the versions of GCC supported as bootstrap by upstream.
The combinatorial explosion in testing matrix would make it impossible for us to test every permutation of potential bootstrap compiler.
It might be that homebrew wishes to test this specific combination you mention (since it is a home-brew release compiler) - and I will of course be happy to fix actual bugs.

ilg-ul commented 7 months ago

I created a new discussion in Homebrew: https://github.com/orgs/Homebrew/discussions/5273.

iains commented 7 months ago

I can confirm that the current releases/gcc-13 branch bootstraps on macOS 14 using CLT 15.3 with Apple clang 15.0.0 (clang-1500.3.9.4) as the bootstrap compiler - I realise this will not surprise you - but it does confirm that the supported case is currently OK on macOS 14.

ilg-ul commented 7 months ago

I'm not surprised, given that I was able to bootstrap it even with clang 16.

The problem was introduced in the recent libc++, starting with llvm 17. I am not able to tell if it's llvm's fault (the new libc++ does not conform to the standard) and must be fixed, or GCC's fault (the way GCC uses the headers is problematic) and must be fixed, but I expect the solution is to add some #ifdef to GCC to accommodate the new definitions in clang libc++.

iains commented 7 months ago

I am not able to tell if it's llvm's fault (the new libc++ does not conform to the standard) and must be fixed, or GCC's fault (the way GCC uses the headers is problematic) and must be fixed, but I expect the solution is to add some #ifdef to GCC to accommodate the new definitions in clang libc++.

I wish it was only one of those two cases - but, in general, it's more complex than that - and can depend on how the clang was built, where it's installed, how it finds the SDK headers etc. and how that interacts with the --with-sysroot= you are using [if you mismatch these things, then you can expect trouble at least subtle and more likely terminal].

iains commented 7 months ago

For the record, I just built upstream LLVM-17.0.6 on macOS 14 (using CLT 15.3) installed that and then used it to bootstrap releases/gcc-13 (taking care to use a consistent sysroot throughout). It worked fine:

$ ../gcc-13/prev-gcc/cc1 -v -version
GNU C17 (GCC) version 13.2.1 20240405 [remotes/repos/releases/gcc-13 revision r13-8591-g045de0ab5864] (x86_64-apple-darwin23)
    compiled by GNU C version Clang 17.0.6 (/repos/llvm-git.git 6009708b4367171ccdbf4b5905cb6a803753fe18), GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.24-GMP

fxcoudert commented 7 months ago

Hi @ilg-ul, as Iain says you're pretty much on your own here. Neither Iain (as GCC maintainer) nor me (as Homebrew maintainer who handles most of GCC stuff) can test all compiler combinations.

GCC in Homebrew is built with Apple's clang
GCC generally supports bootstrap with earlier versions of GCC, and Apple clang for macOS

That being said, detecting incompatibility with newer clang versions is interesting, because they will likely become more used in the future (including by Apple). Could you check if the issue exists with latest trunk?

I believe it might be fixed by https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=68057560ff1fc0fb2df38c2f9627a20c9a8da5c5 and https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=e95ab9e60ce1d9aa7751d79291133fd5af9209d7

iains commented 7 months ago

I believe it might be fixed by https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=68057560ff1fc0fb2df38c2f9627a20c9a8da5c5 and https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=e95ab9e60ce1d9aa7751d79291133fd5af9209d7

It is possible - those patches are already applied to releases/gcc-13 which is what I tested with clang-17.0.6 ...

I will start making pre-release branches for GCC-13, 12, and 11 soon (May is not far away) - hopefully, interested parties can test them before we actually push the release button to save band-aid afterwards.

ilg-ul commented 7 months ago

I confirm too, when using the upstream releases/gcc-12 (I got it via https://github.com/gcc-mirror/gcc/archive/refs/heads/releases/gcc-13.zip), bootstrapping with Homebrew llvm 17 was fine:

$ .../prev-gcc/cc1 -v -version
GNU C17 (xPack GCC x86_64) version 13.2.1 20240410 (x86_64-apple-darwin23.3.0)
    compiled by GNU C version Homebrew Clang 17.0.6, GMP version 6.3.0, MPFR version 4.2.1, MPC version 1.2.1, isl version isl-0.26-GMP

In exactly the same environment (same build scripts, same HB clang 17), but using the official GCC 13.2.0 archive plus the Homebrew patches from https://raw.githubusercontent.com/Homebrew/formula-patches/3c5cbc8e9cf444a1967786af48e430588e1eb481/gcc/gcc-13.2.0.diff, the build failed with the /include/c++/v1/__locale error, as initially reported.

So the problem was in the outdated sources, and was fixed in recent upstream commits. That's nice! :-)

Is upstreaming now complete?

Are there functional issues that justify the extra effort to redo the GCC 13.2.0 as 13.2.1 using these new upstream sources, or are there several patches still required, and it is better to wait for 13.3?

iains commented 7 months ago

Is upstreaming now complete?

That depends on what you regard as "upstreaming" - I have back ported and applied the fixes that fall within the "regressions and documentation" constraint for release branches.

The Darwin gcc-13.3 branch will still contain a number of important improvements that are on trunk, but cannot be back ported under those rules.

As far as upstreaming Arm64 is concerned, we made some good progress during gcc-14 (current trunk) cycle, but there is still quite some distance to travel.

Are there functional issues that justify the extra effort to redo the GCC 13.2.0 as 13.2.1 using these new upstream sources, or are there several patches still required, and it is better to wait for 13.3?

That is a matter for the distributions to decide - I somehow doubt it will be worth the churn when the usual release schedule (May) is quite close - IMO the distributions' time would be better spent testing pre-release branches so that we get a good .0 ... (but that's just my 0.02GBP)

ilg-ul commented 7 months ago

detecting incompatibility with newer clang versions is interesting, because they will likely become more used in the future (including by Apple)

That's exactly why I raised this issue.

I generally try to keep my clang and GCC cross platform distributions up to date, and after each new update I retest all existing builds with the new binaries.

This time, after doing the clang 17 binaries, I noticed that I can no longer build the old GCC 13.2.0. I retried with Homebrew llvm 17, and the result was the same, thus I assumed that the problem might be in GCC.

I'm glad the problem was already solved in upstream GCC.

Thank you for your help in diagnosing this.

ilg-ul commented 7 months ago

The Darwin gcc-13.3 branch will still contain a number of important improvements that are on trunk, but cannot be back ported under those rules.

As far as upstreaming Arm64 is concerned, we made some good progress during gcc-14 (current trunk) cycle, but there is still quite some distance to travel.

Ok, so my strategy will remain to use the same patches as Homebrew.

iains / gcc-13-branch

Bootstraping GCC 13.2 with clang 17? #17