search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.45k stars 482 forks source link

OS X Mojave without /usr/include: a range of issues #26899

Closed jhpalmieri closed 5 years ago

jhpalmieri commented 5 years ago

On OS X Mojave, it is possible to have Xcode installed but to have no directory /usr/include.

Upstream: Reported upstream. No feedback yet.

CC: @embray

Component: packages: standard

Author: Dima Pasechnik, Volker Braun

Branch: db50d09

Reviewer: Dima Pasechnik, Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/26899

dimpase commented 5 years ago
comment:1

See #26713. I'd close this one, but yes, what's offered on #26713 is only a temporary solution. We need to think what to do with OSX (Homebrew? (Ana)conda?), it's getting too hard to jump through new and new hoops built up by Apple.

jhpalmieri commented 5 years ago
comment:2

On this machine, Sage 8.5.beta4 builds. I am now testing. So it's probably not the same issue as #26713 which was opened before 8.5.rc0 was released. Or it's not necessarily the same issue. In particular, I believe that the problem on this machine can be traced directly to #26286.

I believe that on this machine, I once ran

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

as advised at #26713, and then I believe a subsequent Xcode upgrade removed /usr/include. That upgrade apparently left the files necessary to build Sage.

jhpalmieri commented 5 years ago
comment:3

Confirmed: 8.5.beta6 builds but 8.5.rc0 does not. It's the zlib problem. Is there a way to configure Python to detect zlib in the proper location?

jhpalmieri commented 5 years ago
comment:4

The easiest fix is to delete the file build/pkgs/zlib/spkg-configure.m4, right?

dimpase commented 5 years ago
comment:5

Replying to @jhpalmieri:

Confirmed: 8.5.beta6 builds but 8.5.rc0 does not. It's the zlib problem. Is there a way to configure Python to detect zlib in the proper location?

Proper, i.e. only known to Xcode? I don't know. It's not really a zlib problem, it's an Apple bug, if you ask me. They really, really want you to do everything via Xcode.

dimpase commented 5 years ago
comment:6

Replying to @jhpalmieri:

The easiest fix is to delete the file build/pkgs/zlib/spkg-configure.m4, right?

./sage -i zlib

will build zlib from scratch and install it in SAGE_LOCAL, as always.

dimpase commented 5 years ago
comment:7

Once again, on the machine with this problem, do you have /usr/include at all?

I guess it might be that you need to run that open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg etc. after you upgraded XCode, as the latter could have left everything in a slightly out of sync state.

If this does not help then we need to find out whether it's that you actually have non-functioning zlib installation before you start building python, or it's a python issue (I suppose you can look at the logs of the python build and see why zlib extension is not built).

jhpalmieri commented 5 years ago
comment:8

Once again, no I do not have /usr/include on this machine, and once again, Sage 8.5.beta6 builds on it just fine. The problem is only zlib.

I can't tell whether it's something out of sync with this machine, as you suggest, or whether it could routinely happen to other people. If we think it could affect other machines, which is the more conservative point of view, then we should roll back the zlib part of #26286 to fix it. Assuming it's just my machine is riskier and could mean that others will not be able to build Sage.

dimpase commented 5 years ago
comment:9

Rather than blindly rolling back, we'd like to know the cause of the problem, no? This is why I asked you for the log...

vbraun commented 5 years ago
comment:10

I don't think we are going to be able to do anything in the short run, for now you just have to install macOS_SDK_headers_for_macOS_10.14

jhpalmieri commented 5 years ago
comment:11

From the main config.log:

configure:8396: === checking whether to install the zlib SPKG ===
configure:8410: checking for inflateValidate in -lz
configure:8435: g++ -o conftest -g -O2 -std=gnu++11    conftest.cpp -lz  -lm  >&5
configure:8435: $? = 0
configure:8444: result: yes
configure:8451: checking if zlib is wanted
configure:8473: result: yes
configure:8503: checking for inflateEnd in -lz
configure:8528: gcc -o conftest -g -O2    conftest.c -lz  -lm  >&5
configure:8528: $? = 0
configure:8537: result: yes
configure:8545: checking zlib.h usability
configure:8545: gcc -c -g -O2   conftest.c >&5
configure:8545: $? = 0
configure:8545: result: yes
configure:8545: checking zlib.h presence
configure:8545: gcc -E   conftest.c
configure:8545: $? = 0
configure:8545: result: yes
configure:8545: checking for zlib.h
configure:8545: result: yes

I'm rebuilding now (I deleted the old build to test whether removing build/pkgs/zlib/spkg-configure.m4 would work, and it did). I'll post the Python log when it's available, although I didn't see anything helpful there before, just a comment that the zlib component was not built. I think that Python doesn't know all of the places to look for zlib.h.

jhpalmieri commented 5 years ago
comment:12

Volker: a short term fix that doesn't hurt anything is just to remove build/pkgs/zlib/spkg-configure.m4. That is completely safe, so I think we could easily do it for rc1 (if it's decided that it's the right approach).

jhpalmieri commented 5 years ago
comment:13

Attachment: python2-2.7.15.p0.log

Here is the Python log. I don't see anything helpful in there. By the way, why is Python built twice? Is this a recent thing?

dimpase commented 5 years ago
comment:14

Thanks. In the log there not even an attempt to build Python's zlib extension, probably meaning that a pre-build check is failing (such as something that checks for headers somewhere...)

By the way, could you check whether zlib extension is built for the Python3? (and if not, how is it failing) There is a sea of difference between Python2 and 3 in regard of how configuration is done, so it could be a bug of sorts in Python2.

Another question is where that zlib.h is found by spkg-configure.m4 Could you search for zlib.h files on the system (I guess, in XCode tree in /Develop*)?

dimpase commented 5 years ago
comment:15

In Python2/3 configure.ac, one sees that OSX is treated in a special way:

dnl Check if system zlib has *Copy() functions
dnl
dnl On MacOSX the linker will search for dylibs on the entire linker path
dnl before searching for static libraries. setup.py adds -Wl,-search_paths_firs
t
dnl to revert to a more traditional unix behaviour and make it possible to
dnl override the system libz with a local static library of libz. Temporarily
dnl add that flag to our CFLAGS as well to ensure that we check the version
dnl of libz that will be used by setup.py.
dnl The -L/usr/local/lib is needed as wel to get the same compilation
dnl environment as setup.py (and leaving it out can cause configure to use the
dnl wrong version of the library)
case $ac_sys_system/$ac_sys_release in
Darwin/*)
        _CUR_CFLAGS="${CFLAGS}"
        _CUR_LDFLAGS="${LDFLAGS}"
        CFLAGS="${CFLAGS} -Wl,-search_paths_first"
        LDFLAGS="${LDFLAGS} -Wl,-search_paths_first -L/usr/local/lib"
        ;;
esac

AC_CHECK_LIB(z, inflateCopy, AC_DEFINE(HAVE_ZLIB_COPY, 1, [Define if the zlib library has inflateCopy]))

case $ac_sys_system/$ac_sys_release in
Darwin/*)
        CFLAGS="${_CUR_CFLAGS}"
        LDFLAGS="${_CUR_LDFLAGS}"
        ;;
esac

One may wonder whether it could pick up wrong old zlib from /usr/local, say, and then fail on testing it being recent enough).

By right, Python should have a configure option to tell it where zlib is, and do not fool around the system looking for one...

vbraun commented 5 years ago
comment:16

Python should be built only once. Our logging concatenates output from different builds.

vbraun commented 5 years ago
comment:17

IMHO the build/pkgs/zlib/spkg-configure.m4 check should be less leninent, and probably only search zlib in /usr. Right now there is 

zlib_places="/usr/local /usr /opt/local /sw"

and lots of our packages are not going to look into /sw for zlib...

jhpalmieri commented 5 years ago
comment:18

Replying to @dimpase:

Thanks. In the log there not even an attempt to build Python's zlib extension, probably meaning that a pre-build check is failing (such as something that checks for headers somewhere...)

By the way, could you check whether zlib extension is built for the Python3? (and if not, how is it failing)

Same for Python 3. As you note in your later comment, zlib is treated differently on OS X in configure.ac and in Modules/Setup.dist.

Another question is where that zlib.h is found by spkg-configure.m4 Could you search for zlib.h files on the system (I guess, in XCode tree in /Develop*)?

$ locate zlib.h prints the following (along with some matches for files containing "zlib.h", like "nazlib.htf", which I have omitted):

/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/zlib.h

Perhaps the relevant ones are

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h

In any case, there is nothing relevant in /usr/local. I also don't have /sw or /opt/local/.

jhpalmieri commented 5 years ago
comment:19

Replying to @vbraun:

Python should be built only once. Our logging concatenates output from different builds.

I did actually know that second part, which is what led me to ask the question in the first place.

If I run make distclean; make, I see this in the Python 2 and 3 log files:

Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
crypt module imported OK
readline module imported OK
socket module imported OK
_scproxy module imported OK

real    2m28.119s
user    1m17.674s
sys 0m21.064s
Installing python2-2.7.15.p0
make[4]: warning: -jN forced in submake: disabling jobserver mode.

The "real[TAB][time]" string is typically only printed when the installation is complete. Oh, I see, it's running spkg-build and spkg-install and reporting timings for each. That's a little confusing. The first one should say "Successfully ran spkg-build" at the end.

dimpase commented 5 years ago
comment:20

Replying to @vbraun:

IMHO the build/pkgs/zlib/spkg-configure.m4 check should be less leninent,

The requirements are needed to make libpg happy, the version we use needs these checks to pass.

and probably only search zlib in /usr.

Homebrew and freebsd have most things in /usr/local.

What is needed for conda?

Right now there is 

zlib_places="/usr/local /usr /opt/local /sw"

and lots of our packages are not going to look into /sw for zlib...

dimpase commented 5 years ago
comment:21

My current theory is that our macro to recognise/test zlib resorts to calling C complier/linker without extra flags, and everything works, whereas Python's one does something less clever, and fails.

That is we can patch Python's configure.ac and get everything working.

vbraun commented 5 years ago
comment:22

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

PS: Conda is not installed at any fixed path, users can place it whereever they want in their home directory.

dimpase commented 5 years ago
comment:23

Replying to @vbraun:

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

Assuming that mixing system's zlib and Sage's zlib is not a good idea, note that zlib is a dependence (1st or 2nd order) of several other libraries: boost, cbc, glpk, libpng, brial, freetype, giac, libgd, matplotlib, python2/3, etc

that means not using system's zlib forces building of a lot of other Sage libs...

vbraun commented 5 years ago
comment:24

Since its a common dependency its even more important that we err on the safe side

IMHO either

  1. the user has libraries installed in the standard location /usr, or
  2. the user explicitly configures the libray location, or
  3. we build it ourselves

Trying to poke around in the filesystem in sage configure is just going to be endless hurt as every third-party configure is going to do work a little bit different.

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

dimpase commented 5 years ago
comment:25

Replying to @vbraun:

Since its a common dependency its even more important that we err on the safe side

IMHO either

  1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

  1. the user explicitly configures the libray location, or

  2. we build it ourselves

Trying to poke around in the filesystem in sage configure is just going to be endless hurt as every third-party configure is going to do work a little bit different.

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

This is a lot of work for a relatively small crowd, and I don't see why automating it should not be considered.

vbraun commented 5 years ago
comment:26

Replying to @dimpase:

  1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

Its pretty much on every unix-like OS, including OSX. The only thing that OSX lacks here is a sane way to distribute software that is not an self-contained opaque application bundle.

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

And the sane way is to let the users explicitly pick their dependencies, not rummage around and see if we can randomly find something that isn't part of the OS/distribution. It should just be a matter of ./configure CPPFLAGS=-I/the/location/include LDFLAGS=-L/the/location/lib

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

This is a lot of work for a relatively small crowd, and I don't see why automating it should not be considered.

Surely there are many more homebrew developers than sage developers. Its also going to be much easier to maintain a homebrew package when there is a simple way to configure the sage dependencies, than to fix all possible Sage build pitfalls with whatever broken bits and pieces we may find in the filesystem.

dimpase commented 5 years ago
comment:27

Replying to @vbraun:

Replying to @dimpase:

  1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

Its pretty much on every unix-like OS, including OSX.

Not really; on FreeBSD most, but not all, libraries are in /usr/local, rather than in /usr

On Solaris things are just all over the place...

The only thing that OSX lacks here is a sane way to distribute software that is not an self-contained opaque application bundle.

Sanity on OSX is lacking in more than one place, just see above on many incarnations of zlib.h...

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

And the sane way is to let the users explicitly pick their dependencies, not rummage around and see if we can randomly find something that isn't part of the OS/distribution. It should just be a matter of ./configure CPPFLAGS=-I/the/location/include LDFLAGS=-L/the/location/lib

But /usr/local may be a part of OS/distribution, e.g. Homebrew puts everything in /usr/local, and FreeBSD has most things in /usr/local... Similarly, /sw used to be the place where Fink put all its stuff, IIRC.

And as you say, Conda can be put in /bar/foo, so it would suffice to replace /usr by /bar/foo, not fiddle around with every spkg separately.

Even better, Conda comes with pkg-config, so one could use appropriate autoconf macros, right?

jhpalmieri commented 5 years ago
comment:28

By the way, I still don't know if this problem is unique to my computer (did something go wrong in an Xcode update?) or if it is a situation shared by others.

embray commented 5 years ago
comment:29

Replying to @jhpalmieri:

By the way, I still don't know if this problem is unique to my computer (did something go wrong in an Xcode update?) or if it is a situation shared by others.

Well, it would help to find out if that's the case or not. Is this blocking building releases for OSX? The problem here would be that Sage's configure is detecting some zlib headers on your system, in such a way that Python's configure does not detect the same headers. Why don't you post the config.log for each of them so we can see the specifics, rather than just speculating. Also it's not clear to me what exactly makes you think /usr/include has to do with this. If you have a zlib.h in /usr/local/include it will detect that first, by default. If it isn't found in any of the default prefixes it searches (/usr/local, /usr, /opt/local, /sw) then it shouldn't detect any zlib at all.

In the meantime, you can configure Sage like

./configure --without-zlib

This is clearly counter-intuitive but the --without-zlib flag comes from the AX_CHECK_ZLIB macro and will just prevent it from trying to detect zlib, so it will fail to detect and thus install the spkg. You can give that a try.

Also, I've been meaning to update the SAGE_SPKG_CONFIGURE to add configure flags for explicitly forcing SPGKs/disabling checks for the system lib. But for this case --without-zlib ought already to work.

jhpalmieri commented 5 years ago
comment:30

Replying to @embray:

The problem here would be that Sage's configure is detecting some zlib headers on your system, in such a way that Python's configure does not detect the same headers. Why don't you post the config.log for each of them so we can see the specifics, rather than just speculating.

Okay, I'll work on that.

Also it's not clear to me what exactly makes you think /usr/include has to do with this. If you have a zlib.h in /usr/local/include it will detect that first, by default. If it isn't found in any of the default prefixes it searches (/usr/local, /usr, /opt/local, /sw) then it shouldn't detect any zlib at all.

See comment:18 for where zlib.h is on my system. I believe that the main configure script is finding a system version of zlib.h, but it's not in a location found by Python.

jhpalmieri commented 5 years ago

Attachment: sage.config.log

Attachment: python2.config.log

jhpalmieri commented 5 years ago
comment:31

Okay, here is the main config.log and also Python's. Maybe others will find them more illuminating that I do.

dimpase commented 5 years ago
comment:32

The only part where zlib is mentioned in Python's config log is 

configure:11237: checking for inflateCopy in -lz
configure:11262: clang -o conftest -g -O2 -Wl,-search_paths_first  -L/Users/jpalmier/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.6.beta0/local/lib -Wl,-rpath,/Users/jpalmier/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.6.beta0/local/lib  -Wl,-search_paths_first -L/usr/local/lib conftest.c -lz  -ldl  >&5
configure:11262: $? = 0
configure:11271: result: yes

Strangely, no sign of header check... Also, that -L/usr/local/lib is worrying - don't you by any chance have a copy of zlib in /usr/local ?

And no attempt to build zlib extension in logs/pkg/python2-2.7.15.p0.log.

A python bug? It's really hard to see why zlib extension wasn't even tried to be built...

jhpalmieri commented 5 years ago
comment:33

Replying to @dimpase:

Also, that -L/usr/local/lib is worrying - don't you by any chance have a copy of zlib in /usr/local ?

No. Not in /usr/local/include, and find /usr/local -name zlib.h -print returns nothing. There's only this:

$ find /usr/local -name zlib* -print
/usr/local/include/boost/iostreams/detail/config/zlib.hpp
/usr/local/include/boost/iostreams/filter/zlib.hpp

Edit: and if I move this stuff out of /usr/local, the main Sage config.log file still says "zlib-1.2.11.p0 not installed (configure check)" and Python 2 is still built without zlib support.

dimpase commented 5 years ago
comment:34
dnl Check if system zlib has *Copy() functions
dnl
dnl On MacOSX the linker will search for dylibs on the entire linker path
dnl before searching for static libraries. setup.py adds -Wl,-search_paths_first
dnl to revert to a more traditional unix behaviour and make it possible to
dnl override the system libz with a local static library of libz. Temporarily
dnl add that flag to our CFLAGS as well to ensure that we check the version
dnl of libz that will be used by setup.py. 
dnl The -L/usr/local/lib is needed as wel to get the same compilation 
dnl environment as setup.py (and leaving it out can cause configure to use the
dnl wrong version of the library)
case $ac_sys_system/$ac_sys_release in
Darwin/*) 
    _CUR_CFLAGS="${CFLAGS}"
    _CUR_LDFLAGS="${LDFLAGS}"
    CFLAGS="${CFLAGS} -Wl,-search_paths_first"
    LDFLAGS="${LDFLAGS} -Wl,-search_paths_first -L/usr/local/lib"
    ;;
esac

AC_CHECK_LIB(z, inflateCopy, AC_DEFINE(HAVE_ZLIB_COPY, 1, [Define if the zlib library has inflateCopy]))

This is a special casing of Darwin in python2 configure.ac

Can you remove it and try again? (I presume you do have zlib header in /usr/include)

dimpase commented 5 years ago
comment:35

If this works then it is an old upstream workaround that should be obsolete with OSX 10.13 and 10.14

vbraun commented 5 years ago
comment:36

Python's setup.py checks for the presence (header + library) and version of zlib, and will silently skip zlib if it can't be found. This all happens after the configure run.

        zlib_inc = find_file('zlib.h', [], inc_dirs)
        have_zlib = False
        if zlib_inc is not None:
            zlib_h = zlib_inc[0] + '/zlib.h'
            version = '"0.0.0"'
            version_req = '"1.1.3"'
            if host_platform == 'darwin' and is_macosx_sdk_path(zlib_h):
                zlib_h = os.path.join(macosx_sdk_root(), zlib_h[1:])
            fp = open(zlib_h)
            while 1:
                line = fp.readline()
                if not line:
                    break
                if line.startswith('#define ZLIB_VERSION'):
                    version = line.split()[2]
                    break
            if version >= version_req:
                if (self.compiler.find_library_file(lib_dirs, 'z')):
                    if host_platform == "darwin":
                        zlib_extra_link_args = ('-Wl,-search_paths_first',)
                    else:
                        zlib_extra_link_args = ()
                    exts.append( Extension('zlib', ['zlibmodule.c'],
                                           libraries = ['z'],
                                           extra_link_args = zlib_extra_link_args))
                    have_zlib = True
                else:
                    missing.append('zlib')
            else:
                missing.append('zlib')
        else:
            missing.append('zlib')
jhpalmieri commented 5 years ago
comment:37

Replying to @dimpase:

(I presume you do have zlib header in /usr/include)

I remind you of the ticket title.

dimpase commented 5 years ago
comment:38

Replying to @vbraun:

Python's setup.py checks for the presence (header + library) and version of zlib, and will silently skip zlib if it can't be found. This all happens after the configure run. [...]

OK, so the only role of ./configure is to write out the approprite #define in the generated header.

From reading setup.py it looks as if a reasonable attempt is made on OSX to locate the headers in OSX SDK directories, but I cannot guess which of the several checks in this code fragment fails. This needs debugging on OSX 10.14, which I don't have access to. Could someone have a look, just by adding prints in the right places?

jhpalmieri commented 5 years ago
comment:39

I've added print statements, I don't know if they're in the right places, but as far as I can tell, Python is not looking anywhere in /Applications for zlib.h, and in particular, it is not looking in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/....

Do you have particular requests for where to put print statements, or particular information you would like to see?

jhpalmieri commented 5 years ago
comment:40

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

dimpase commented 5 years ago
comment:41

Replying to @jhpalmieri:

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

I don't see how not installing the headers is normal, unless you want to abandon classic command-line workflow and do everything via Xcode.

The OSX-specific workaround involving adding -L/usr/local/lib to various places seems to indicate that they specifically support, on OSX, libraries installed in /usr/local.

dimpase commented 5 years ago
comment:42

Replying to @jhpalmieri:

I've added print statements, I don't know if they're in the right places, but as far as I can tell, Python is not looking anywhere in /Applications for zlib.h, and in particular, it is not looking in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/....

I think is_macosx_sdk_path() expects SDK to be in /System or /Library, but not in /Applications - perhaps this restriction is obsolete, and should be removed?

Do you have particular requests for where to put print statements, or particular information you would like to see?

Does zlib_inc compute as None? What is inc_dirs it is called with? What is the result of macosx_sdk_root() call?

ditto - with is_macosx_sdk_path() hacked to allow /Applications to contain SDK.

jhpalmieri commented 5 years ago
comment:43

Replying to @dimpase:

Replying to @jhpalmieri:

I've added print statements, I don't know if they're in the right places, but as far as I can tell, Python is not looking anywhere in /Applications for zlib.h, and in particular, it is not looking in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/....

I think is_macosx_sdk_path() expects SDK to be in /System or /Library, but not in /Applications - perhaps this restriction is obsolete, and should be removed?

Do you have particular requests for where to put print statements, or particular information you would like to see?

Does zlib_inc compute as None?

Yes.

What is inc_dirs it is called with?

['.', 'Include', './Include', '/usr/local/include', '/Users/jpalmier/Desktop/Python-2.7.15/Include', '/Users/jpalmier/Desktop/Python-2.7.15']

(I'm building in /Users/jpalmier/Desktop/Python-2.7.15.)

What is the result of macosx_sdk_root() call?

'/'

ditto - with is_macosx_sdk_path() hacked to allow /Applications to contain SDK.

No change.

I also tried

CFLAGS="-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" ./configure

which changed the return value of macosx_sdk_root to that directory, but the zlib extension still wasn't bult.

dimpase commented 5 years ago
comment:44

Can you try calling xcrun (?) with appropriate options to get correct SDK root in macos_sdk_root() ?

vbraun commented 5 years ago
comment:45

Should be xcrun --show-sdk-path

jhpalmieri commented 5 years ago
comment:46

Turns out I had guessed the SDK root correctly: xcrun --show-sdk-path returns

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
dimpase commented 5 years ago
comment:47

that part of setup.py in Python is quite dated, they should really be using xcrun, instead of fooling around with parsing of CFLAGS.

embray commented 5 years ago
comment:48

Replying to @dimpase:

Replying to @jhpalmieri:

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

I don't see how not installing the headers is normal, unless you want to abandon classic command-line workflow and do everything via Xcode.

Knowing Apple that's probably what they want you to do. Why, after all, would you be using Apple products except if you want to use software written specifically for Apple with Apple's toolchain :)

The OSX-specific workaround involving adding -L/usr/local/lib to various places seems to indicate that they specifically support, on OSX, libraries installed in /usr/local.

You can also set PYTHONCONFIGURE=... to add additional configure flags for configuring Python, if indeed that helps at all.

embray commented 5 years ago
comment:49

Replying to @jhpalmieri:

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

It did come up here: https://github.com/pyenv/pyenv/issues/1219

There's also discussion on the Python bug tracker here; it came up in the context of building _tkinter but the issue is similar: https://bugs.python.org/issue34956

With an interesting comment from Ned Deily (emphasis mine):

Python has never fully supported building only from an Xcode-only installation, i.e. we have required at a minimum installing system header files into /usr/include et al by using "xcode-select --install". As of 10.14, xcode-select no longer installs header files in /usr/include. So that further cripples Python builds in that header files for several third-party libraries shipped with macOS are no longer found, like zlib and sqlite3. When using an Xcode-only installation (no Command Line Tools) in previous releases of macOS, I believe it was the case that essentially the system root directory ('/') was searched by the compiler tool chain for header files, for libraries, and for frameworks, and for frameworks the long standard fallback framework search path order was honored by default: first search !/Library/Frameworks, then !/System/Library/Frameworks. That worked fine if you were installing frameworks like Tcl and TK into !/Library/Frameworks and wanting them to override the system ones. If you did not install the Command Line Tools, then the tool chain used the MacOSX.sdk embedded in Xcode.app as the system root for searching. By default, that's found in: !/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk