search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.43k stars 479 forks source link

Meta-ticket: Support Xcode 12 #30494

Closed mkoeppe closed 1 year ago

mkoeppe commented 4 years ago

Xcode 12 was publicly released on 2020-09-16.

Sage does not build from source using Xcode 12.2 beta, released 2020.09.17, clang version 12.0.0 (clang-1200.0.32.4)

Failing packages (blockers):

Failing packages with replacements available via homebrew:

Failing optional packages:

Failing experimental packages:

Tickets:

Follow-up:

CC: @jhpalmieri @dimpase @slel @vbraun

Component: porting

Keywords: macOS, Xcode

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30494

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,8 +1,15 @@
 Sage does not build from source using Xcode 12 beta.

 - https://groups.google.com/d/msg/sage-support/ZLgu-1pi6nc/W7Hqe7AgAgAJ
-- 
+- https://github.com/mkoeppe/sage/runs/1059961873

+Failing packages:
+- gf2x
+- ecm
+- symmetrica
+- rubiks
+- ecl
+- scipy

 Tickets: 
 - #30487 - GH Actions: Test also with Xcode 12 beta
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -13,4 +13,4 @@

 Tickets: 
 - #30487 - GH Actions: Test also with Xcode 12 beta
-
+- #28022 - Downgrade `rubiks` to optional
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -13,4 +13,5 @@

 Tickets: 
 - #30487 - GH Actions: Test also with Xcode 12 beta
+- #27754 - Upgrade: Python 3.8.5
 - #28022 - Downgrade `rubiks` to optional
jhpalmieri commented 4 years ago
comment:4

For what it's worth, I was trying to build without setting MACOSX_DEPLOYMENT_TARGET at all: I removed those lines from src/bin/sage-env.

mkoeppe commented 4 years ago
comment:5

Note that these lines are only executed when using python3 installed from spkg, not system python

dimpase commented 4 years ago
comment:6

looks like this should wait till 9.3

jhpalmieri commented 4 years ago
comment:7

Sure, Big Sur is still in beta.

jhpalmieri commented 4 years ago
comment:9

Xcode 12 has been released, and it causes breakage with Catalina, so perhaps this should be a blocker for 9.2. (For example I now cannot build Sage: I upgraded a laptop to a beta release of Big Sur, and on my main machine Xcode updated itself to version 12 overnight.)

mkoeppe commented 4 years ago
comment:10

I agree. It would be good to figure out if the upstream maintainers of these packages are already aware of these issues.

dimpase commented 4 years ago
comment:11

Can we score an https://en.wikipedia.org/wiki/Developer_Transition_Kit_(2020) - and Apple ARM prototype for developers somehow? It might be very useful for testing. Probably a Raspberry Pi with a 64-bit OS comes close, but not as much.

mkoeppe commented 4 years ago
comment:12

This is unrelated to this ticket.

mkoeppe commented 4 years ago
comment:13

I have opened #30592 (Meta-ticket: Port to Apple silicon) for that. Let's keep the present ticket focused please. No new hardware is required.

jhpalmieri commented 4 years ago
comment:14

It seems that when configuring gf2x and ecl, at least, something like -Wall is being used. I see this kind of thing in the config.log files for those:

conftest.c:4:3: error: implicitly declaring library function 'exit' with type 'void (int) __attribute__((noreturn))' [-Werror,-Wimplicit-function-declaration]
  exit(0);
  ^
conftest.c:4:3: note: include the header <stdlib.h> or explicitly provide a declaration for 'exit'
1 error generated.

Is there some setting in Xcode's command-line tools that has changed, causing these errors?

jhpalmieri commented 4 years ago
comment:15

Similar with ecm:

configure:13286: clang -O2 -pedantic -fomit-frame-pointer -m64 -mtune=nehalem -march=nehalem  -c conftes1.c >&5 && clang -O2 -pedantic -fomit-frame-pointer -m64 -mtune=nehalem -march=nehalem -c conftes2.s >&5 && clang -O2 -pedantic -fomit-frame-pointer -m64 -mtune=nehalem -march=nehalem  conftes1.o conftes2.o >&5
conftes1.c:4:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
main () { underscore_test(); }
^
conftes1.c:4:11: error: implicit declaration of function 'underscore_test' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
main () { underscore_test(); }
          ^
1 warning and 1 error generated.
emmanuelthome commented 4 years ago
comment:16

Hi,

I just learned about this ticket. (me = upstream gf2x)

I don't have access to a box of this kind, so I can't test. (all I have is macOS 10.15.5 + XCode 11.6).

It looks like latest xcode doesn't like the tests that are generated by autotools, right ? If yes, does a autoreconf with fresh autotools fix the problem ?

E.

jhpalmieri commented 4 years ago
comment:17

The Sage build log for symmetrica contains:

hiccup.c:478:3: error: implicit declaration of function 'set_multiplier' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                set_multiplier(S_MO_K(S_L_S(a)));
                ^
jhpalmieri commented 4 years ago
comment:18

Replying to @emmanuelthome:

Hi,

I just learned about this ticket. (me = upstream gf2x)

I don't have access to a box of this kind, so I can't test. (all I have is macOS 10.15.5 + XCode 11.6).

I'm seeing this on OS X 10.15.6 after an upgrade to Xcode 12. You could try upgrading to Xcode 12, if you're willing to take that risk.

It looks like latest xcode doesn't like the tests that are generated by autotools, right ? If yes, does a autoreconf with fresh autotools fix the problem ?

I'm not an expert in this sort of thing, but I don't think this will work. autoreconf would help if Sage's top-level configure script were running into problems, but the problems are coming from Sage packages. Those are just tarballs, and running autoreconf once won't help with those. For what it's worth, I unpacked the gf2x tarball and ran autoreconf on it, but I got the same error.

(This is with autoreconf 2.69 installed via homebrew. I did brew upgrade first. I'm not sure how to get a "fresh autotools".)

jhpalmieri commented 4 years ago

Attachment: config.log

gf2x config.log

jhpalmieri commented 4 years ago
comment:19

Here is the config.log for gf2x, in case it helps.

jhpalmieri commented 4 years ago
comment:20

I'm happy to run some experiments, if you have suggestions.

mkoeppe commented 4 years ago
comment:21

Replying to @jhpalmieri:

It seems that when configuring gf2x and ecl, at least, something like -Wall is being used. I see this kind of thing in the config.log files for those:

conftest.c:4:3: error: implicitly declaring library function 'exit' with type 'void (int) __attribute__((noreturn))' [-Werror,-Wimplicit-function-declaration]
  exit(0);
  ^
conftest.c:4:3: note: include the header <stdlib.h> or explicitly provide a declaration for 'exit'
1 error generated.

Is there some setting in Xcode's command-line tools that has changed, causing these errors?

I think errors like this are usually from code incorrectly assuming that some header pulls in some other header. Adding the correct includes should fix this.

emmanuelthome commented 4 years ago
comment:22

Replying to @jhpalmieri:

I don't have access to a box of this kind, so I can't test. (all I have is macOS 10.15.5 + XCode 11.6).

I'm seeing this on OS X 10.15.6 after an upgrade to Xcode 12. You could try upgrading to Xcode 12, if you're willing to take that risk.

Not light-heartedly for sure. It's a mac mini that has been living in an office of (very) mac-averse researchers for years and years, crawling under the ram demands of the latest macOS (and actually last it can support).

It looks like latest xcode doesn't like the tests that are generated by autotools, right ? If yes, does a autoreconf with fresh autotools fix the problem ?

I'm not an expert in this sort of thing, but I don't think this will work. autoreconf would help if Sage's top-level configure script were running into problems, but the problems are coming from Sage packages.

I didn't make myself clear. Of course I mean autoreconf for gf2x.

Those are just tarballs, and running autoreconf once won't help with those. For what it's worth, I unpacked the gf2x tarball and ran autoreconf on it, but I got the same error.

ok.

(This is with autoreconf 2.69 installed via homebrew. I did brew upgrade first. I'm not sure how to get a "fresh autotools".)

doesn't say much. 2.69 is the banner under which all autoconf version have been presenting themselves for 8 years. IDK about the others, but I suspect autoconf is at fault here anyway.

jhpalmieri commented 4 years ago
comment:23

The gf2x configure script says

cat >conftest.c <<EOF
int
main ()
{
  exit(0);
}
EOF

and it looks like this is the file causing the problems. How is this generated? autotools? Should autotools be adding some includes?

Edit: this is in config/acinclude.m4.

emmanuelthome commented 4 years ago

Attachment: gf2x-1.3.0-cc_for_build_stdlib.patch.gz

emmanuelthome commented 4 years ago
comment:24

grr. trac deleted my comment. Try the patch above. This will fix both the autoconf input (just in case) and output (so that you don't have to rerun autoconf, a plain ./configure would do). These bits of the config checks are created by m4 code that sits in gf2x, so it's really a gf2x bug.

jhpalmieri commented 4 years ago
comment:25

That looks good: ./configure; make; make check now works. Thanks!

emmanuelthome commented 4 years ago
comment:26

Replying to @jhpalmieri:

It seems that when configuring gf2x and ecl, at least, something like -Wall is being used.

I think that -Wall is not the problem. We're witnessing a -Werror behaviour here. That is odd. -Werror in config checks is a bit extreme, and easily counter-productive. config checks that battle the -Werror reports might well defeat the grand portability goal of autoconf

The catch with my patch above is that because of this -Werror behaviour, I think that more failures are likely to show up down the line. (well, you seem to say that it works fine. good!)

jhpalmieri commented 4 years ago
comment:27

I can do the same kind of fix with ecl: add #include <stdlib.h> in the configure script. I'm not sure what to do with ecm or symmetrica. I haven't looked at scipy yet at all.

jhpalmieri commented 4 years ago

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 - https://github.com/mkoeppe/sage/runs/1059961873

 Failing packages:
-- gf2x
+- gf2x #30593
 - ecm
 - symmetrica
 - rubiks
jhpalmieri commented 4 years ago
comment:28

I've put the gf2x fix on #30593.

jhpalmieri commented 4 years ago

Description changed:

--- 
+++ 
@@ -7,8 +7,8 @@
 - gf2x #30593
 - ecm
 - symmetrica
-- rubiks
-- ecl
+- rubiks (see also #28022)
+- ecl #30594
 - scipy

 Tickets:
jhpalmieri commented 4 years ago
comment:29

Change for ecl is at #30594. I don't know if it's the best approach, but it appears to work.

slel commented 4 years ago
comment:30

Found only one somewhat related discussion on the SciPy issue tracker

Might be worth opening a ticket there.

jhpalmieri commented 4 years ago
comment:31

I opened https://github.com/scipy/scipy/issues/12860

jhpalmieri commented 4 years ago
comment:32

Thank you for finding the older scipy issue, by the way.

zimmermann6 commented 4 years ago
comment:33

for the gmp-ecm issue, please can you report it on the ecm-discuss@lists.gforge.inria.fr list, configuring it outside of Sage, and specifying which release you used?

dimpase commented 4 years ago
comment:34

I cannot reproduce the breakade you are talking about. It all works for me with Xcode 12.0 on macOS 10.15.6. The latest Xcode 12 beta is in fact 12.2.beta. I'll check with it shortly.

dimpase commented 4 years ago
comment:35

Downgraded to critical, as this is about a beta, which should not stop the release.

Volker, what is your take on this? This is all about betaOS/beta tools. Would be nice to have in 9.2 fixed, but not strictly necessary.

dimpase commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-Sage does not build from source using Xcode 12 beta.
+Sage does not build from source using Xcode 12.2 beta, released 2020.09.17, clang version 12.0.0 (clang-1200.0.32.4)

 - https://groups.google.com/d/msg/sage-support/ZLgu-1pi6nc/W7Hqe7AgAgAJ
 - https://github.com/mkoeppe/sage/runs/1059961873
dimpase commented 4 years ago
comment:36

sorry for noise - it turns out that to properly reinstall command line tools, one needs to sudo rm -rf /Library/Developer/CommandLineTools Once I did the latter, and reinstalled, I got clang 12 there, too.

That is, XCode 12 (stable) also suffers from these ailments.

dimpase commented 4 years ago
comment:37

Replying to @zimmermann6:

for the gmp-ecm issue, please can you report it on the ecm-discuss@lists.gforge.inria.fr list, configuring it outside of Sage, and specifying which release you used?

I can report directly on gmp-ecm bug tracker - but it's still on gforge. Are you planning to move to https://gitlab.inria.fr/ ?

dimpase commented 4 years ago
comment:38

Replying to @dimpase:

Replying to @zimmermann6:

for the gmp-ecm issue, please can you report it on the ecm-discuss@lists.gforge.inria.fr list, configuring it outside of Sage, and specifying which release you used?

I can report directly on gmp-ecm bug tracker - but it's still on gforge. Are you planning to move to https://gitlab.inria.fr/ ?

Reported on https://gforge.inria.fr/tracker/index.php?func=detail&aid=21856&group_id=135&atid=623

zimmermann6 commented 4 years ago
comment:39

thanks. Yes it is planned to move to gitlab, when I find some time to do so...

dimpase commented 4 years ago

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@

 Failing packages:
 - gf2x #30593
-- ecm
+- ecm #30600
 - symmetrica
 - rubiks (see also #28022)
 - ecl #30594
zimmermann6 commented 4 years ago
comment:41

the gmp-ecm issue is fixed upstream, and waiting for a new release, the patch at the link in comment [comment:38] can be used in Sage.

dimpase commented 4 years ago
comment:42

Replying to @zimmermann6:

the gmp-ecm issue is fixed upstream, and waiting for a new release, the patch at the link in comment [comment:38] can be used in Sage.

it is already in #30600 - with a configure patch too, naturally (as we don't require autotools).

dimpase commented 4 years ago
comment:43

as to symmetrica - I don't grok its code. It seems that set_root_multiplier is a no-op?

% cd upstream/symmetrica-2.0
% grep -R set_root_mu .
./hiccup.c: set_root_multiplier (OP extra);
./hiccup.c:     set_root_multiplier(S_MO_K(S_L_S(a)));
./hiccup.c:set_root_multiplier (extra) OP extra;
./hiccup.c:     set_root_multiplier(S_MO_K(S_L_S(a)));
jhpalmieri commented 4 years ago
comment:44

I don't really know what I'm doing, but I feel like I've made some progress with symmetrica with the attached patch and with this change to Sage:

diff --git a/build/pkgs/symmetrica/spkg-install.in b/build/pkgs/symmetrica/spkg-install.in
index bc5d8e5f26..a071b6fece 100644
--- a/build/pkgs/symmetrica/spkg-install.in
+++ b/build/pkgs/symmetrica/spkg-install.in
@@ -4,7 +4,7 @@ cd src
 # as it has (syntactically) nothing in common with ours.
 cp -f ../patches/makefile makefile || exit $?

-export CFLAGS="-O2 -g $CFLAGS -fPIC -DFAST -DALLTRUE"
+export CFLAGS="-O2 -g $CFLAGS -fPIC -DFAST -DALLTRUE -DUNDEF"

 sdh_make
 sdh_install libsymmetrica.a "$SAGE_LOCAL/lib"

I don't know if this is a good approach, though.

jhpalmieri commented 4 years ago

Attachment: symmetrica-xcode12.patch.gz

Attempting to patch symmetrica to build with Xcode 12

jhpalmieri commented 4 years ago

Description changed:

--- 
+++ 
@@ -10,6 +10,7 @@
 - rubiks (see also #28022)
 - ecl #30594
 - scipy
+- openblas

 Tickets: 
 - #30487 - GH Actions: Test also with Xcode 12 beta