search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.37k stars 466 forks source link

Accept /usr/bin/python3 from XCode 12.3 on macOS 10.15 (Catalina) and 11 (Big Sur) #31227

Closed mkoeppe closed 3 years ago

mkoeppe commented 3 years ago

(from #31132)

27122 determines whether CC accepts -march=native and then adds it globally to CFLAGS in build/bin/sage-build-env.

30725 fixes this for macOS, where some packages are explicitly built with clang.

However, there is another mechanism how clang can come in: Python extension modules are compiled with the compiler listed in sysconfig. On macOS with homebrew, using /usr/bin/python3, this could again be clang, which does not accept -march=native in combination with -arch arm64 (which is provided even on x86_64 if ARCHFLAGS is unset).

In this ticket, we fix this by checking whether the compiler used by distutils accepts -march=native (or more generally, the flags determined earlier by configure). If it does not, then we disable the use of these flags.

In order to test this with /usr/bin/python3 from XCode 12.3 (on macOS 10.15 (Catalina)), we fix another issue: This python3 (3.8.2) is not able to build extension modules because it emits -arch arm64 -arch x86_64 unless ARCHFLAGS is set explicitly

$ /usr/bin/python3 -m sysconfig |grep '\bCFLAGS ='
    CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64"

$ ARCHFLAGS="" /usr/bin/python3 -m sysconfig |grep '\bCFLAGS ='
    CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers  -arch x86_64 "

In this ticket, if the distutils test fails, we try if setting ARCHFLAGS="" fixes it. Also, after accepting a system python and ARCHFLAGS so far has not been set, we test whether the system python wants to do the multi-arch build.

In either of these two cases, we store a configuration variable that causes sage-env to set ARCHFLAGS as well.

This is somewhat complicated logic - we are tiptoeing around previous breakage that had been caused by setting ARCHFLAGS unconditionally (#29408).

Depends on #31132 Depends on #30725

CC: @jhpalmieri @zlscherr @dimpase @kliem @vbraun

Component: build: configure

Author: Matthias Koeppe

Branch/Commit: fc8b676

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/31227

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from df17ad4 to 914ea58

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

914ea58build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 914ea58 to 986739b

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

986739bm4/sage_check_python_for_venv.m4: Fix reason
mkoeppe commented 3 years ago
comment:41

Thanks for spotting these mistakes!

kliem commented 3 years ago
comment:42

Otherwise, this looks very plausible.

I updated https://github.com/kliem/sage/pull/36/ (not the last two commits, but they are cosmetic).

jhpalmieri commented 3 years ago
comment:43

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before:

sage -t --long --random-seed=0 src/sage_setup/find.py
**********************************************************************
File "src/sage_setup/find.py", line 278, in sage_setup.find.installed_files_by_module
Failed example:
    f1
Expected:
    'sage/structure/__init__.py'
Got:
    '/Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc'
**********************************************************************
File "src/sage_setup/find.py", line 280, in sage_setup.find.installed_files_by_module
Failed example:
    f2
Expected:
    'sage/structure/....pyc'
Got:
    'sage/structure/__init__.py'

and

sage -t --long --random-seed=0 src/sage_setup/clean.py
**********************************************************************
File "src/sage_setup/clean.py", line 95, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        if '/ext_data/' in f: continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: /Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc
    Found stale file: /Users/jpalmier/Library/Caches/com.apple.python/Users/jpalmier/Desktop/Sage/git/sage/local/lib/python3.8/site-packages/sage/version.cpython-38.pyc

  [ and many more lines of such messages ]

Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

jhpalmieri commented 3 years ago
comment:44

This does not work for me with OS X Big Sur, though: many packages (cython, pynac, gmpy2, psutil, pillow, kiwisolver, cffi) fail to build with the error "clang: error: the clang compiler does not support '-march=native'".

Hold on, I have to run ./bootstrap and try again.

Confirmed, these packages all fail after

make distclean && ./bootstrap && ./configure --with-python=/usr/bin/python3 && make -k
mkoeppe commented 3 years ago
comment:45

Replying to @jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

mkoeppe commented 3 years ago
comment:46

config.log from the big sur system please

jhpalmieri commented 3 years ago
comment:47

Replying to @mkoeppe:

Replying to @jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

No, not normally and not after running ./sage --sh.

jhpalmieri commented 3 years ago

Attachment: config.2.log

config.log for failed Big Sur build

jhpalmieri commented 3 years ago
comment:48

Replying to @jhpalmieri:

Replying to @mkoeppe:

Replying to @jhpalmieri:

This works for me on OS X Catalina. If I also include #31204, it passes most doctests. I see a few failures I don't remember seeing before [...] Why is it looking in $HOME/Library? Also, the files there look from their dates as if they were created during this build of Sage.

Is PYTHONPYCACHEPREFIX set on your system? https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX (#31314)

No, not normally and not after running ./sage --sh.

However:

% /usr/bin/python3   
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/Users/jpalmier/Library/Caches/com.apple.python'

This is not how /usr/local/bin/python3 behaves: sys.pycache_prefix is None there. The Python documentation suggests that the default setting should be None since I am not passing a -X argument nor am I setting PYTHONPYCACHEPREFIX.

mkoeppe commented 3 years ago
comment:49

hm... config.2.log does not look like it's coming from this branch...

mkoeppe commented 3 years ago
comment:50

Replying to @jhpalmieri:

% /usr/bin/python3   
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.pycache_prefix
'/Users/jpalmier/Library/Caches/com.apple.python'

Same here. Looks like Apple has preconfigured their build to do this. I'll update the description of #31314.

jhpalmieri commented 3 years ago
comment:51

Replying to @mkoeppe:

hm... config.2.log does not look like it's coming from this branch...

Sorry, you're right. Typo on my part: I checked out #31127 instead of #31227. Trying again now.

jhpalmieri commented 3 years ago
comment:52

Now with Big Sur, cysignals and scipy fail. For scipy, the error looks like this:

    /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/lib/python3.8/site-packages/numpy/distutils/system_info.py:838: UserWarning: Specified path /usr/local/include/python3.8 is invalid.

Somehow it's picking up /usr/local. I'm not sure what's going on with cysignals. Logs attached.

jhpalmieri commented 3 years ago

config.log for failed Big Sur build

jhpalmieri commented 3 years ago

Attachment: config.3.log

jhpalmieri commented 3 years ago

Attachment: cysignals-1.10.2.log

mkoeppe commented 3 years ago
comment:53

Attachment: scipy-1.5.4.log

The config.log looks good.

mkoeppe commented 3 years ago
comment:54

Replying to @jhpalmieri:

Now with Big Sur, cysignals and scipy fail. For scipy, the error looks like this:

    /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/lib/python3.8/site-packages/numpy/distutils/system_info.py:838: UserWarning: Specified path /usr/local/include/python3.8 is invalid.

I think the actual error is this one:

  ar: adding 50 object files to build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a
  warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a will be fat and ar(1) will not be able to operate on it
  ar: adding 50 object files to build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a
  ar: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it)
  ar: build/temp.macosx-10.14.6-x86_64-3.8/libsuperlu_src.a: Inappropriate file type or format
mkoeppe commented 3 years ago
comment:55

Similar: https://github.com/scipy/scipy/issues/10902

mkoeppe commented 3 years ago
comment:56

Is MACOSX_DEPLOYMENT_TARGET set to something on your big sur system?

jhpalmieri commented 3 years ago
comment:57

Replying to @mkoeppe:

Is MACOSX_DEPLOYMENT_TARGET set to something on your big sur system?

No, but who knows what Xcode and/or homebrew are doing these days.

mkoeppe commented 3 years ago
comment:58

from cysignals.log: 

  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -Wp,-U_FORTIFY_SOURCE -DCYTHON_CLINE_IN_TRACEBACK=0 -U_FORTIFY_SOURCE -Isrc/cysignals -Isrc -I/Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c build/src/cysignals/signals.c -o build/temp.macosx-10.14.6-x86_64-3.8/build/src/cysignals/signals.o -pthread
  In file included from build/src/cysignals/signals.c:1570:
  In file included from build/src/cysignals/implementation.c:58:
  In file included from /Users/palmieri/Desktop/Sage/sage_builds/TESTING/clean/sage-9.3.beta6/local/include/pari/pari.h:49:
  ../src/kernel/none/divll_pre.h:30:10: error: invalid output constraint '=a' in asm
    return divll(~0UL, n);
           ^

This is trying to build for -arch arm64 -arch x86_64 and is tripping over the installed pari header ... which is arch-specific!

jhpalmieri commented 3 years ago
comment:59

Before I source .homebrew-build-env, I only set these environment variables:

BIBINPUTS=/Users/palmieri/iCloud/Documents/tex/bibtex:.:
EDITOR=emacs
LESS=-R
LSCOLORS=Gxfxcxdxbxegedabagacad
MAKE='make -j6'
MAKEFLAGS=-j6
PAGER=less
PATH=/Users/palmieri/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands

(If I run export at the start of my .zshrc file and also comment out the line that runs .homebrew-build-env, and then run export again with an actual running terminal, these are the differences.) Nothing strange there, nothing that I think should cause any problems. This is on the Big Sur system, but the Catalina system has a very similar configuration.

mkoeppe commented 3 years ago
comment:60

It seems that we need to set ARCHFLAGS="" in more situations. The default set by /usr/bin/python3, trying to build for both architectures, is unrealistic to support -- this would require major changes to how we install libraries.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 986739b to 0b3e70d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0b3e70dbuild/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it
mkoeppe commented 3 years ago
comment:62

Here's a new version to try on big sur

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

372adcb.github/workflows/tox.yml: Update xcode versions
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 0b3e70d to 372adcb

mkoeppe commented 3 years ago
comment:64

Started tests at https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -14,9 +14,9 @@

$ /usr/bin/python3 -m sysconfig |grep '\bCFLAGS =' CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64" -ARCHFLAGS="" /usr/bin/python3 -m sysconfig |grep '\bCFLAGS =' + +$ ARCHFLAGS="" /usr/bin/python3 -m sysconfig |grep '\bCFLAGS =' CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch x86_64 " -egret:~/s/sage/sage-rebasing/worktree-algebraic-2018-spring (t/31132/homebrewunusedpackagessingularpari____in__usr_local_leak_into_build_when_using_homebrew_s_python3 *


 In this ticket, if the distutils test fails, we try if setting `ARCHFLAGS=""` fixes it. In that case, we store a configuration variable that causes `sage-env` to set `ARCHFLAGS` as well.
mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -19,7 +19,10 @@
    CFLAGS = "-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers  -arch x86_64 "

-In this ticket, if the distutils test fails, we try if setting ARCHFLAGS="" fixes it. In that case, we store a configuration variable that causes sage-env to set ARCHFLAGS as well. +In this ticket, if the distutils test fails, we try if setting ARCHFLAGS="" fixes it. Also, after accepting a system python and ARCHFLAGS so far has not been set, we test whether the system python wants to do the multi-arch build.

+This is somewhat complicated logic - we are tiptoeing around previous breakage that had been caused by setting ARCHFLAGS unconditionally (#29408). + +

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

fc8b676SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 372adcb to fc8b676

zlscherr commented 3 years ago
comment:69

For the record, I was able to successfully build sage and documentation with this ticket on Big Sur using

--with-python=/usr/bin/python3

mkoeppe commented 3 years ago
comment:70

Thanks for testing!

jhpalmieri commented 3 years ago
comment:71

Works for me with the system Python 3 on both Big Sur and Catalina.

mkoeppe commented 3 years ago
comment:72

Thanks for testing!

Let's deal with issue #31314 (messages from find_stale_files) in a follow up.

Ready for review?

mkoeppe commented 3 years ago
comment:73

Replying to @mkoeppe:

Started tests at https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

Some results have been coming in:

Still waiting for the tests with other xcode versions (12.2, 12.3, 12.4). No failures so far.

Note all runs are on x86_64 - we do not have test infrastructor for macos-arm64 yet.

kliem commented 3 years ago

Changed reviewer from https://github.com/kliem/sage/pull/36/checks, Jonathan Kliem to Jonathan Kliem

kliem commented 3 years ago
comment:74

I'm happy with this ticket, but I would like another opinion on this.

kliem commented 3 years ago
comment:75

Ok, I give it another 24 hours. If nobody speaks up, I'll give a positive review (after taking a final look of course).

zlscherr commented 3 years ago
comment:76

I don't feel like I have enough experience to review this, but it certainly built on Big Sur and Catalina. I'm going to upgrade Big Sur to 11.2 later and see if this builds on 11.2 as well.

mkoeppe commented 3 years ago
comment:77

Replying to @mkoeppe:

Started tests at https://github.com/mkoeppe/sage/actions/runs/526769308 but it looks like there is a long wait time to get macos instances

Some more results:

mkoeppe commented 3 years ago
comment:78

Let's take care of the scipy build failures in #31326.

I think this is sufficient improvement for one ticket.

zlscherr commented 3 years ago
comment:80

Seems to work well on Big Sur 11.2 with standard homebrew packages. When running make ptest I get the annoying warnings:

ld: warning: dylib (/usr/local/lib/libgmp.dylib) was built for newer macOS version (10.15) than being linked (10.14.6)

which is probably because the MACOSX_DEPLOYMENT_TARGET for /usr/bin/python3 is set to 10.14.6 which doesn't match what Homebrew sets. It's easy enough to eliminate the warnings by setting MACOSX_DEPLOYMENT_TARGET=11.0 before doing all the testing.

Otherwise, everything seems to build/test correctly.