gdal --with-python fails

[rddesmond]

I'm escalating a comment from @dschneiderch on #36744 to a full issue. In it, he mentioned that he was no longer able to install gdal with python support. That issue is about removing the sandboxed version of numpy in favor of one that is normally installed, and whatever change broke the build doesn't appear to be related (though I may be OBE if the proposed patch in that issue is applied). The shell output from that comment is:

brew install gdal --with-complete --with-python --with-libkml --env=std
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading http://download.osgeo.org/gdal/1.11.2/gdal-1.11.2.tar.gz
Already downloaded: /Library/Caches/Homebrew/gdal-1.11.2.tar.gz
==> Downloading https://downloads.sourceforge.net/project/numpy/NumPy/1.8.1/numpy-1.8.1.tar.gz
Already downloaded: /Library/Caches/Homebrew/gdal--numpy-1.8.1.tar.gz
==> python setup.py build --fcompiler=gnu95 install --prefix=/usr/local/Cellar/gdal/1.11.2_3/libexec
File "numpy/core/setup.py", line 42, in check_types
out = check_types(a, *kw)
File "numpy/core/setup.py", line 289, in check_types
"Cannot compile 'Python.h'. Perhaps you need to "\
SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

READ THIS: https://git.io/brew-troubleshooting

These open issues may also help:
gdal 2.0.0 #42845
gdal 1.11.2 on OS X 10.6 #42056
Latest update to poppler, 0.34.0 breaks gdal --with-complete #42031
gdal --with-python always installs numpy into prefix #36744
Khione:~

[rddesmond]

I was able to overcome the problem with numpy by applying the patch at: https://github.com/Homebrew/homebrew-python/issues/224. (I ran brew install gdal --with-python --debug and manually applied the linked patch in the shell that appears as numpy fails.)

It's not clear to me if homebrew can apply patches to a resource, if that patch is hiding something else, or if that's desired -- since #36744 aims to remove the resource.

[rddesmond]

After overcoming the problem with numpy, the swig/python additionally fails to build with:

==> python setup.py install --prefix=/usr/local/Cellar/gdal/1.11.2_3 --record=installed.txt --single-version-externally-managed
running install
running build
running build_py
...
clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/opt/sqlite/include -arch x86_64 -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/usr/local/Cellar/gdal/1.11.2_3/libexec/lib/python2.7/site-packages/numpy/core/include -I/private/tmp/gdal20150918-30019-q12g3y/gdal-1.11.2/include -c extensions/gdal_wrap.cpp -o build/temp.macosx-10.10-x86_64-2.7/extensions/gdal_wrap.o
extensions/gdal_wrap.cpp:2855:10: fatal error: 'cpl_port.h' file not found
#include "cpl_port.h"
         ^
1 error generated.
error: command 'clang' failed with exit status 1

I'm not sure about this part yet. That file is in the ../../port, which is included by setup.cfg:

# You can override default build options here

[build_ext]
#include_dirs = ../../port:../../gcore:../../alg:../../ogr/
#library_dirs = ../../.libs:../../
#libraries = gdal
gdal_config=../../apps/gdal-config

[DomT4]

Can you provide a full brew gist-logs of the build? Thanks.

[rddesmond]

Sure, running brew gist-logs gdal --with-python output to https://gist.github.com/5fa9b27b3d4e49db3112. Does that capture the --with-python flag?

[rddesmond]

I see that it did, very good. That gives back the main problem, since my current fix for the main problem is addressed by a manual step during the build (see my first comment), I'm not sure how to get to the secondary problem in my second comment.

[jiegec]

That is fixed in numpy v1.10.0b1.

[rddesmond]

So, to complete the circle -- here is where I am. I built this successfully by updating the resource to numpy 1.10.0b1 and uncommenting out the items in build_ext under setup.cfg (with --debug; I jumped to the shell during the build -- I didn't make a proper patch).

I'm not sure how to proceed -- I am not exactly sure what change caused the problem. Submitting a patch to use the new version to numpy is doable -- but it seems odd to use a beta version randomly here (and -- at least for me -- it complains that I am building and running on different numpy api versions). I see some other formula where a patch (e.g. the one in https://github.com/Homebrew/homebrew-python/issues/224) is applied to a resource (#31508), but that is an ongoing discussion. Additionally, there is discussion about how numpy should be treated in general (#36810).

For the patch to batch to gdal itself (uncommenting the comment things in build_ext)... it would be nice to know the delta that made this problem cause before proposing the solution. A patch is easy enough to provide though (once I learn how to do it).

[rddesmond]

And the gist-logs for the second part of this. The gist-logs after updating the resource to numpy-1.10.0b1 (which makes it run through until the GDAL failure):

https://gist.github.com/d33add29118b7318196a (the error is in 05.python)

[rotten]

FWIW, I was able to finally get it (gdal 1.11.3) to build on my brand new Yosemite system from source by going into swig/python/setup.cfg and uncommenting the three lines in the [build_ext] section that were commented out (by the configurator?) for some reason. I could not get brew to build it for me when I asked for either --with_postgresql or --with_python.

[tdsmith]

numpy 1.9.3 resolves the build issue.

[rotten]

It didn't for me...

[tdsmith]

Sorry, that was ambiguous; I meant the build issue we were patching for in homebrew-python. No idea what issue you're encountering!

[rddesmond]

@tdsmith That's great! Thank you for submitting the patch upstream. For me, that fixes the first half of the problem in this issue (and makes it so the solution doesn't have to include patching a resource, phew). (The second half being in the gdal python build doesn't include all of the required directories in the build path.)

[tdsmith]

I don't understand why the setup.cfg is only sometimes an issue (I see it building for python 3 but not for python 2) but I guess it doesn't matter. I opened #44345 to address this.

[rddesmond]

Building from your branch works great for me! Thank you for the patch. It appears that it now builds python by default, correct?

You might consider adding a test to make sure that python is linking correctly (the "if", of course, assumes that python isn't default -- if probably isn't valid if it is)

if build.with?("python") || build.with?("python3")
  system "#{bin}/gdal2tiles.py", "--help"
end