Problem building GCC/gfortran/Ada with XCode 15

[Romendakil]

First of all, the testsuites of gmp, mpfr, mpc segfault, secondly, gcc (14.0.0 current master) does not compiler, but fails: libtool: link: /usr/local/packages/gcc_master/_build/./gcc/xgcc -shared-libgcc -B/usr/local/packages/gcc_master/_build/./gcc -nostdinc++ -L/usr/local/packages/gcc_master/_build/x86_64-apple-darwin22.6.0/libstdc++-v3/src -L/usr/local/packages/gcc_master/_build/x86_64-apple-darwin22.6.0/libstdc++-v3/src/.libs -L/usr/local/packages/gcc_master/_build/x86_64-apple-darwin22.6.0/libstdc++-v3/libsupc++/.libs -B/usr/local/x86_64-apple-darwin22.6.0/bin/ -B/usr/local/x86_64-apple-darwin22.6.0/lib/ -isystem /usr/local/x86_64-apple-darwin22.6.0/include -isystem /usr/local/x86_64-apple-darwin22.6.0/sys-include -fno-checking -dynamiclib -o .libs/libstdc++.6.dylib .libs/libstdc++.6.dylib-master.o -L/usr/local/packages/gcc_master/_build/x86_64-apple-darwin22.6.0/libstdc++-v3/libsupc++/.libs -L/usr/local/packages/gcc_master/_build/x86_64-apple-darwin22.6.0/libstdc++-v3/src -L/usr/local/packages/gcc_master/_build/x86_64-apple-darwin22.6.0/libstdc++-v3/src/.libs -liconv -lm -Wl,-exported_symbols_list -Wl,libstdc++-symbols.explist -install_name /usr/local/lib/libstdc++.6.dylib -compatibility_version 7 -current_version 7.33 0 0x10e99ef43 __assert_rtn + 64 1 0x10e8a0f43 ld::AtomPlacement::findAtom(unsigned char, unsigned long long, ld::AtomPlacement::AtomLoc const&, long long&) const + 1411 2 0x10e8bd431 ld::InputFiles::SliceParser::parseObjectFile(mach_o::Header const) const + 19745 3 0x10e8cdb71 ld::InputFiles::parseAllFiles(void (ld::AtomFile const*) block_pointer)::$_7::operator()(unsigned long, ld::FileInfo const&) const + 657 4 0x7ff80b531066 _dispatch_client_callout2 + 8 5 0x7ff80b54418f _dispatch_apply_invoke_and_wait + 213 6 0x7ff80b543692 _dispatch_apply_with_attr_f + 1207 7 0x7ff80b543847 dispatch_apply + 45 8 0x10e966972 ld::AtomFileConsolidator::parseFiles(bool) + 370 9 0x10e8edd67 main + 12263 ld: Assertion failed: (resultIndex < sectData.atoms.size()), function findAtom, file Relocations.cpp, line 1336. collect2: error: ld returned 1 exit status make[6]: [libstdc++.la] Error 1 make[5]: [all-recursive] Error 1 make[4]: [all-recursive] Error 1 make[3]: [all] Error 2 make[2]: [all-stage1-target-libstdc++-v3] Error 2 make[1]: [stage1-bubble] Error 2 make: *** [all] Error 2

[iains]

this is a known problem with Xcode 15, it is expected to be fixed in some future release, but for the present time the best solution is to re-install 14.3

[rhdtownsend]

A warning for those of you who've upgraded to MacOS 14 (Sonoma) -- Xcode 14.3 won't install on Sonoma, and so after the upgrade you'll be stuck without being able to build the compiler.

[iains]

A warning for those of you who've upgraded to MacOS 14 (Sonoma) -- Xcode 14.3 won't install on Sonoma, and so after the upgrade you'll be stuck without being able to build the compiler.

There is a workaround for the broken XC15; assuming that you have the command line tools installed in the default place: add --with-ld=/Library/Developer/CommandLineTools/usr/bin/ld-classic to your configure line (assuming you are bootstrapping the compiler with clang).

[simonjwright]

Just built 4c1229e (Mon Oct 9 14:06:57 2023 +0100) for C,C++,Ada, bootstrap, with CLT 15.1 beta; no trouble with the linker, but

Comparing stages 2 and 3
Bootstrap comparison failure!
aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o differs
make[2]: *** [compare] Error 1

The last time I built the compiler (28 September) I'd disabled bootstrap, unfortunately.

[iains]

Just built 4c1229e (Mon Oct 9 14:06:57 2023 +0100) for C,C++,Ada, bootstrap, with CLT 15.1 beta; no trouble with the linker, but

Comparing stages 2 and 3
Bootstrap comparison failure!
aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o differs
make[2]: *** [compare] Error 1

The last time I built the compiler (28 September) I'd disabled bootstrap, unfortunately.

Hm.. OK - I bootstrapped that on macOS 12.7 with XC14.3 .. my Sonoma box is (a) x86_64 and (b) tied up with build/test of a release1 patch stack for gcc-13.2 .. so will have wait (I do not have Arm64 Sonoma, not enough h.w).

[iains]

Just built 4c1229e (Mon Oct 9 14:06:57 2023 +0100) for C,C++,Ada, bootstrap, with CLT 15.1 beta; no trouble with the linker, but

Comparing stages 2 and 3
Bootstrap comparison failure!
aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o differs
make[2]: *** [compare] Error 1

The last time I built the compiler (28 September) I'd disabled bootstrap, unfortunately.

Hm.. OK - I bootstrapped that on macOS 12.7 with XC14.3 .. my Sonoma box is (a) x86_64 and (b) tied up with build/test of a release1 patch stack for gcc-13.2 .. so will have wait (I do not have Arm64 Sonoma, not enough h.w).

Can you confirm that using the --with-ld=/path/to/ld-classic still works here (it should, since the version of ld-classic was not changed AFAIU).

[rhdtownsend]

On Oct 9, 2023, at 2:55 PM, Iain Sandoe @.***> wrote:

Just built 4c1229e (Mon Oct 9 14:06:57 2023 +0100) for C,C++,Ada, bootstrap, with CLT 15.1 beta; no trouble with the linker, but Comparing stages 2 and 3 Bootstrap comparison failure! aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o differs make[2]: *** [compare] Error 1

The last time I built the compiler (28 September) I'd disabled bootstrap, unfortunately. Hm.. OK - I bootstrapped that on macOS 12.7 with XC14.3 .. my Sonoma box is (a) x86_64 and (b) tied up with build/test of a release1 patch stack for gcc-13.2 .. so will have wait (I do not have Arm64 Sonoma, not enough h.w). Can you confirm that using the --with-ld=/path/to/ld-classic still works here (it should, since the version of ld-classic was not changed AFAIU).

Yes, I can confirm this — just completed a build of 13.1 on Sonoma with your suggested workaround, and it went fine.

cheers,

Rich

[simonjwright]

Just built 4c1229e (Mon Oct 9 14:06:57 2023 +0100) for C,C++,Ada, bootstrap, with CLT 15.1 beta; no trouble with the linker, but

Comparing stages 2 and 3
Bootstrap comparison failure!
aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o differs
make[2]: *** [compare] Error 1

The last time I built the compiler (28 September) I'd disabled bootstrap, unfortunately.

I didn't say in the above comment, but this is on an M1 mini running Sonoma 14.0.

Repeating the build but with --with-ld=/Library/Developer/CommandLineTools/usr/bin/ld-classic, the bootstrap build succeeded, as did make check-acats (same results as last time).

[iains]

Just built 4c1229e (Mon Oct 9 14:06:57 2023 +0100) for C,C++,Ada, bootstrap, with CLT 15.1 beta; no trouble with the linker, but

Comparing stages 2 and 3
Bootstrap comparison failure!
aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o differs
make[2]: *** [compare] Error 1

The last time I built the compiler (28 September) I'd disabled bootstrap, unfortunately.

I didn't say in the above comment, but this is on an M1 mini running Sonoma 14.0.

Repeating the build but with --with-ld=/Library/Developer/CommandLineTools/usr/bin/ld-classic, the bootstrap build succeeded, as did make check-acts (same results as last time).

Hum, that's probably going to be very very tricky to debug - since the implication is that the problem is in different code-gen between stage2 and stage3 compilers, but based on how the compiler binaries were linked (since, with ld-classic/ld64 the compiler apparently works and nothing else was changed); ... and by a linker that we cannot debug.

[simonjwright]

I need to re-run both these builds, to check for repeatability.

[simonjwright]

The link with /usr/bin/ld failed, as before.

Changing Makefile as below:

--- Makefile~   2023-10-10 11:02:09
+++ Makefile    2023-10-10 12:47:05
@@ -27718,6 +27718,7 @@
          gcc/cc*-checksum$(objext) | gcc/ada/*tools/* | gcc/m2/gm2-compiler-boot/M2Version* | gcc/m2/gm2-compiler-boot/SYSTEM* | gcc/m2/gm2version*) \
            echo warning: $$file differs ;; \
          *) \
+           echo error: $$file differs; \
            echo $$file differs >> .bad_compare ;; \
        esac; \
      fi; \

the only file that fails is aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o.

I've attached a zip of this file in the stage?-aarch64-apple-darwin21/libstdc++-v3/src/.libs/ directories.

objdump -D shows no differences, aside from the file names.

cmp shows they differ at char 2891121.

gcc-14-linker-ld.zip

[rhdtownsend]

On Oct 10, 2023, at 8:53 AM, Simon Wright @.***> wrote:

The link with /usr/bin/ld failed, as before. Changing Makefile as below: --- Makefile~ 2023-10-10 11:02:09 +++ Makefile 2023-10-10 12:47:05 @@ -27718,6 +27718,7 @@ gcc/cc-checksum$(objext) | gcc/ada/tools/ | gcc/m2/gm2-compiler-boot/M2Version | gcc/m2/gm2-compiler-boot/SYSTEM | gcc/m2/gm2version) \ echo warning: $$file differs ;; \ *) \

• echo error: $$file differs; \ echo $$file differs >> .bad_compare ;; \ esac; \ fi; \

the only file that fails is aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o. I've attached a zip of this file in the stage?-aarch64-apple-darwin21/libstdc++-v3/src/.libs/ directories. objdump -D shows no differences, aside from the file names. cmp shows they differ at char 2891121. gcc-14-linker-ld.zip

I got my build to succeed with aarch64-apple-darwin20 — but I doubt that’s responsible for the issue.

cheers,

Rich

[iains]

The link with /usr/bin/ld failed, as before.

Changing Makefile as below:

--- Makefile~ 2023-10-10 11:02:09
+++ Makefile  2023-10-10 12:47:05
@@ -27718,6 +27718,7 @@
        gcc/cc*-checksum$(objext) | gcc/ada/*tools/* | gcc/m2/gm2-compiler-boot/M2Version* | gcc/m2/gm2-compiler-boot/SYSTEM* | gcc/m2/gm2version*) \
          echo warning: $$file differs ;; \
        *) \
+         echo error: $$file differs; \
          echo $$file differs >> .bad_compare ;; \
      esac; \
    fi; \

the only file that fails is aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o.

I've attached a zip of this file in the stage?-aarch64-apple-darwin21/libstdc++-v3/src/.libs/ directories.

objdump -D shows no differences, aside from the file names.

cmp shows they differ at char 2891121.

gcc-14-linker-ld.zip

OK so the procedure for examining compare debug fails is.

1. cat .bad_compare (tells you which failed)
2. /path/to/src/contrib/compare-debug -p <stage2 file> <stage3 file>

so now you will have stage2-xxxx/file.o.stripped and stage3-xxxx/file.o.stripped

then you can do things like

otool -tV stage2-xxxx/file.o.stripped >s2.txt otool -tV stage3-xxxx/file.o.stripped >s3.txt

and then compare the disassembled text (s2.txt and s3.txt) which can show if you really have differen tcode .

or otool -lv (to see if the objects have different section layouts)

[iains]

actually, I do not get a compare-debug fail with those files using XC14.3 - please could you give me the output of /path/to/src/contrib/compare-debug <stage2 file> <stage3 file> using your linker and the output of ld -v

note that we have long used ld to strip (remove all non-critical sections like debug and LTO) files because stock cctools strip does not understand some of the GCC special sections. I guess we could make it so that we have to build our own strip - but I've been trying to avoid things like this.

[simonjwright]

compare-debug: doesn't fail.

Having stripped with compare-debug -p (which again doesn't fail), cmp doesn't fail.

otool -tV, otool -lv: the only difference is the file name in the first line.

$ /usr/bin/ld -v
@(#)PROGRAM:ld  PROJECT:dyld-1021
BUILD 00:34:28 Sep 26 2023
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h
will use ld-classic for: armv6 armv7 armv7s arm64_32 i386 armv6m armv7k armv7m armv7em
LTO support using: LLVM version 15.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 15.0.0 (tapi-1500.0.12.6)
Library search paths:
Framework search paths:

The reason for patching the Makefile was that (it looked to me as though) only the last difference was reported, and I wanted to get an idea of how many files were affected.

[iains]

I believe the .bad_compare scheme works :) [I've seen many files in there on ocassion].

So, in summary, the bad compare is being reported (and stops the build) but is not reproducible with compare-debug?

compare-debug does the cmp too - so if it exits with OK status, that means the compare worked [if invoked without the -p]

is your configuration using --with-ld=/path/to/ld-classic?

If so .. then perhaps the problem is that the compare-debug script has no idea about --with-ld= and is presumably using /usr/bin/ld... hmm

[simonjwright]

@rhdtownsend said

I got my build to succeed with aarch64-apple-darwin20 — but I doubt that’s responsible for the issue. cheers, Rich

so, since I'm running on Darwin 23.0.0, I configured with aarch64-apple-darwin23 instead of -darwin21 and unset MACOSX_DEPLOYMENT_TARGET (which I had set to 12, Monterey).

Same build failure.

[iains]

@rhdtownsend said

I got my build to succeed with aarch64-apple-darwin20 — but I doubt that’s responsible for the issue. cheers, Rich

so, since I'm running on Darwin 23.0.0, I configured with aarch64-apple-darwin23 instead of -darwin21 and unset MACOSX_DEPLOYMENT_TARGET (which I had set to 12, Monterey).

Same build failure.

With XC CLT 15.1beta? or XC15b/.0

[iains]

I guess I will have to see if this reproduces with x86_64 on darwin23 (I do not have arm64-darwin23).

[iains]

I built and tested current trunk and https://github.com/iains/gcc-darwin-arm64/commit/4c1229eb649dfd56c7e19f8c0871970b8673b381 on x86-64-darwin23 with the current Xcode 15.1beta CLT (without the ld-classic workaround).

$ clang --version
Apple clang version 15.0.0 (clang-1500.1.0.1.1)

I am building --enable-languages=all, with a bootstrap GCC-11.4Dr0, for D and Ada (I rebuilt that [11.4] compiler --with-ld=/path/to/ld-classic... so that I do not need to do the horrendously complex dance with GNATMAKE to pass that through).

It built and tested without any (new) issues, so nothing there to help debug what you are seeing on Arm64 - unfortunately, as noted, I do not have enough hardware to run macOS14/Sonoma on Arm64.

Probably, the next thing to try to debug is why the compare-debug script is producing a fail when run inside the build, but a pass when we try to replicate the issue outside.

It's just a shell script, so should be possible to do some logging/printf debugging. AFAIR, you can just try to run the (failed compare) build again - as long as the compare fails, it will keep stopping.

[simonjwright]

The .bad_compare scheme almost certainly works - I didn't notice the >> which appends failure reports.

So, there's only one failure: aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o. This file is a partial link generated by libtool; note, it's not /usr/bin/libtool, it's the GNU script(version 2.2.7a); trying to replace it by /usr/bin/libtool fails. The latest version is 2.4.6, but since the comparison failure happens with ld and not with ld-classic it seems unlikely that updating this would help (also, looks hairy).

Using clt15.1's ld (not ld-classic) the identical failure happens with x86_64 GCC 13.1.0 (built on my M1 mini under Rosetta, start by arch -x86_64 /bin/bash).

The comparison is done using cmp --ignore-initial=16 rather than contrib/compare-debug.

[iains]

The .bad_compare scheme almost certainly works - I didn't notice the >> which appends failure reports.

So, there's only one failure: aarch64-apple-darwin21/libstdc++-v3/src/.libs/libstdc++.6.dylib-master.o. This file is a partial link generated by libtool; note, it's not /usr/bin/libtool, it's the GNU script(version 2.2.7a); trying to replace it by /usr/bin/libtool fails. The latest version is 2.4.6, but since the comparison failure happens with ld and not with ld-classic it seems unlikely that updating this would help (also, looks hairy).

They are completely different things and not interchangeable (unfortunate that they have the same name and are both related to building libraries).

the GNU script is the correct thing to be used there; but interesting that it's a partial link (so ld ... -r ... ?)

Using clt15.1's ld (not ld-classic) the identical failure happens with x86_64 GCC 13.1.0 (built on my M1 mini under Rosetta, start by arch -x86_64 /bin/bash).

Out of curiosity - why gcc13.1? .. the current release is 13.2 (and I could not repeat the failure on native x86_64 Sonoma with it + 15.1b ld)

EDIT: but note that a gcc-13-2-darwin-release1 is pending (in testing) so probably worth delaying an update for a few more days.

The comparison is done using cmp --ignore-initial=16 rather than contrib/compare-debug.

that seems just wrong - so maybe the issue lies with detecting the availability of the compare-debug script I will try to take a look at the weekend.

[iains]

The comparison is done using cmp --ignore-initial=16 rather than contrib/compare-debug.

that seems just wrong - so maybe the issue lies with detecting the availability of the compare-debug script I will try to take a look at the weekend.

This seems to be the output of ACX_PROG_CMP_IGNORE_INITIAL

but the config.log here says that the BUILD_CONFIG=bootstrap-debug, which should mean that contrib/compare-debug is used. Will have to poke further.

[simonjwright]

Stop poking :-(

For reasons I don’t have logged, I added --with-build-config=no to my build scripts in September 2021.

Sorry for the noise.

[iains]

For reasons I don’t have logged, I added --with-build-config=no to my build scripts in September 2021.

Actually, I am still slightly confused about how that gave a fail - since "no" ought to be suppressing the debug-compare checks (and if it did not, I would expect every compare to fail since cmp --ignore-initial=16 is no use for comparing objects with/without debug [at least on Darwin]).

Anyway, more pressing things to attend to - if you get a working build with the defaults, and we are all agreed that XC15.1b produces a working build, then I think we can close this issue.

[simonjwright]

I think the objects being compared consistently have/don’t have debug. I’ve certainly been running like this since 11.2.0, this is the first time it’s gone wrong. Maybe the new ld has an uninitialized data problem?

XC15.1b produces a working build (for c,c++,ada, anyway).

[iains]

I think the objects being compared consistently have/don’t have debug. I’ve certainly been running like this since 11.2.0, this is the first time it’s gone wrong. Maybe the new ld has an uninitialized data problem?

To clarify (for the benefit of archeology) the purpose of the compare-debug script.

The intent is to demonstrate that a source compiled with and without -g generates the same code (exactly). So the compare-debug process strips debug sections (and other non-code sections like LTO byte code) and then checks that the stripped binaries agree. The stripping of debug is non-trivial in general (and we delegate it to ld, which has proved to be more capable than the cctools strip).

This test is repeated for every significant object between stage 2 (no debug) and stage 3 (with debug). If it fails, sometimes there's a real code-gen issue, sometimes it's peripheral (i.e. some difference in object layout) - but in either case we have to address it.

Since we delegate the strip job to ld i was not surprised to get a report that a new ld version had a problem ;) good that it is not....

XC15.1b produces a working build (for c,c++,ada, anyway).

So I think we can say this Issue is done.