Linking against anaconda's hdf5 dylib causes an error in ld on OS X

[ngoldbaum]

This happens when I try to pip install yt on anaconda. Interestingly, when I try to build the conda recipe for yt on conda-recipes, I don't get an error, even though it's linking against the same dylib.

The full error is pasted below:

building 'yt.utilities.hdf5_light_reader' extension
compiling C sources
C compiler: gcc-4.8 -fno-strict-aliasing -I/Users/goldbaum/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

compile options: '-DH5_USE_16_API=True -I/usr/local/include -I/Users/goldbaum/anaconda/lib/python2.7/site-packages/numpy/core/include -I/Users/goldbaum/anaconda/include/python2.7 -c'
gcc-4.8: yt/utilities/hdf5_light_reader.c
gcc -bundle -undefined dynamic_lookup -L/Users/goldbaum/anaconda/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/yt/utilities/hdf5_light_reader.o -L/usr/local/lib -lm -lhdf5 -o build/lib.macosx-10.5-x86_64-2.7/yt/utilities/hdf5_light_reader.so
ld: mach-o string pool extends beyond end of file in /Users/goldbaum/anaconda/lib/libhdf5.dylib for architecture x86_64
collect2: ld returned 1 exit status
ld: mach-o string pool extends beyond end of file in /Users/goldbaum/anaconda/lib/libhdf5.dylib for architecture x86_64
collect2: ld returned 1 exit status
error: Command "gcc -bundle -undefined dynamic_lookup -L/Users/goldbaum/anaconda/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/yt/utilities/hdf5_light_reader.o -L/usr/local/lib -lm -lhdf5 -o build/lib.macosx-10.5-x86_64-2.7/yt/utilities/hdf5_light_reader.so" failed with exit status 1

Cursory googling returns the c++ file that generates the error: https://github.com/Apple-FOSS-Mirror/ld64/blob/master/src/ld/parsers/macho_dylib_file.cpp

When I link against homebrew's hdf5 and hdf5 compiled from source, I do not have the same issue.

[asmeurer]

I get the same error from the recipe. Note that my machine is using clang to build. I pasted the conda build output and the ld stacktrace at https://gist.github.com/asmeurer/6384735.

[ngoldbaum]

Hmm, not quite the same error - you got a seg fault and I got a 'mach-o string pool extends beyond end of file' error. I'm on 10.8.4 using xcode 4.6.3.

[ngoldbaum]

Just to demonstrate that this issue isn't specific to yt, I just tried building h5py from source and got the same error. For reference, the compilation session is pasted at https://gist.github.com/ngoldbaum/6392792.

[ngoldbaum]

I've run into this again in the context of our new conda-based distribution of yt: https://bitbucket.org/yt_analysis/yt/pull-request/591/first-draft-of-a-new-get_yt-script-based/diff

Is there a workaround for this issue? One thought I had was to try to build my own hdf5 package using the 1.9.1 release of conda, but there isn't an hdf5 recipe available on conda-recipes and I'm not sure how to write a recipe for a binary-only package.

[asmeurer]

I added the hdf5 recipe from Anaconda to conda-recipes. It looks like the url is broken, so I wasn't able to test if it works with conda build.

[ngoldbaum]

Thanks a ton for uploading the recipe! I've PRd a fix for the URL. I'm able to successfully link against the dylibs produced by conda build hdf5 on my laptop :)

I'm happy to close the issue, but this won't be solved for anyone besides me until a new build of hdf5 is uploaded for anaconda. @asmeurer, @ilanschnell will a new build of hdf5 be available with the new anaconda release due out tomorrow?

[ilanschnell]

Thanks for updating the source tarball URL, I've merged yout PR. But I don't see how the new URL would fix anything. I just checked that we build HDF5 using a tarball which is identical to the one located at the new URL.

• Ilan

On Mon, Sep 9, 2013 at 12:52 PM, ngoldbaum notifications@github.com wrote:

Thanks a ton for uploading the recipe! I've PRd a fix for the URL. I'm able to successfully link against the dylibs produced by conda build on my laptop :)

I'm happy to close the issue, but this won't be solved for anyone besides me until a new build of hdf5 is uploaded for anaconda. @asmeurerhttps://github.com/asmeurer, @ilanschnell https://github.com/ilanschnell will a new build of hdf5 be available with the new anaconda release due out tomorrow?

— Reply to this email directly or view it on GitHubhttps://github.com/ContinuumIO/anaconda-issues/issues/18#issuecomment-24099721 .

[ngoldbaum]

@ilanschnell, the issue isn't with the source tarball, it's that the current build of hdf5 on anaconda includes dylibs that need to be regenerated using the latest conda release.

[ilanschnell]

Sure, but the hdf5 package itself has already been updated since the last Anaconda release, so

$ conda update hdf5

should fix it. And this is also the hdf5 package which will go into the release tomorrow.

[ngoldbaum]

@ilanschnell, nope, this is an issue with the hdf5-1.8.9-1 build. To see it trigger, try building h5py on an OS X machine (I'm on 10.8, haven't tried it on 10.7 or earlier) from source using setup.py install rather than conda build. In the latest conda release, there was a fix for the issue that produced the bad dylibs in the hdf5-1.8.9-1 build, but since hdf5-1.8.9-1 was not rebuilt since then, there will still be linking issues on OS X unless hdf5 is rebuilt.

[ilanschnell]

Is this the error you are getting?

ld: mach-o string pool extends beyond end of file in /Users/ilan/python/lib/libhdf5.dylib for architecture x86_64

Which was the fix in conda that fixed the problem? Sorry, I'm a bit confused now.

[ngoldbaum]

Yes, that's right. I believe this was fixed in this PR although I'm not 100% sure which change fixed it. This issue is also relevant.

[ilanschnell]

I'd be surprised if the PR you refer to has fixed it. I'm assuming that the compile problem has nothing to do with conda, but with the fact that we build HDF5 on OSX 10.5, which is is not the same OSX version you or I build h5py against. h5py (in Anaconda) is also build on 10.5.

[ngoldbaum]

@ilanschnell maybe there's a way to fix this in a portable way. One thing that does work with the current hdf5 build on anaconda is conda build h5py or conda build yt. Is there something special about conda build that avoids this issue? Could it be incorporated as part of conda install or as part of the conda installer?

[ilanschnell]

Ah, so you are saying that with the current build of hdf5 actually work with conda build h5py. in that case it should be easy to see what causes the problem. conda build adds environment variables to build subprocess, and if you add the env`` command to yourbuild.shyou can see what they are. Finding the environment variable(s) which is needed to makesetup.py install``` work shouldn't be hard then.

[ngoldbaum]

Unfortunately there isn't an h5py recipe on conda-recipes, so I'm using the yt recipe.

Here is the output of env from inside the yt recipe's build.sh script: http://paste.yt-project.org/show/3857/

I've tried setting CFLAGS, CXXFLAGS, LDFLAGS, OSX_ARCH, PKG_CONFIG_PATH, and MACOSX_DEPLOYMENT_TARGET to match the settings in the conda build environment, but am still getting the same mach-o string pool error when I try to setup.py install yt.

@ilanschnell am I not setting a necessary environment variable? I believe the rest are particular to conda build, per the docs or control things on the python side and shouldn't affect compilation.

[matthewturk]

@ilanschnell it looks like we're blocked on this for using Conda as a build environment for yt. Any ideas about what we could do? It seems that similar issues crop up when the GLIBC is not the same on the system as in Conda, which shows up often on supercomputers. Thanks for any hints ...

[asmeurer]

I added h5py from Anaconda if that will help.

[ngoldbaum]

Just looked at this again using the h5py recipe and have discovered a possible reason why builds succeed under conda build but fail under pip install or setup.py install builds.

Specifically, take a look at this gist: https://gist.github.com/ngoldbaum/6674064

The first file is the output of env under conda build, just before setup.py install is invoked in the build.sh script. I still don't see anything special here and after messing around with setting a few of these environment variables, I'm not able to get the build to succeed outside of conda build.

The second file shows two invocations of gcc. The first succeeds and occurs under conda build h5py, the second fails and occurs under pip install h5py. I see that in the first call, there are explicit references to library locations inside anaconda that are not present in the pip install gcc call. Could this be the base issue? It's not clear to me why the two gcc calls are different. I'd appreciate any suggestions on how to make these two build environments as similar as possible.

[ngoldbaum]

I just tried again with Anaconda 1.8 and XCode 5 command line tools - same issue:

ld: mach-o string pool extends beyond end of file in /Users/goldbaum/anaconda/lib/libhdf5.dylib file '/Users/goldbaum/anaconda/lib/libhdf5.dylib' for architecture x86_64

[teoliphant]

It sounds like we should close this issue because conda build h5py works. Conda intentionally creates a clean and controlled build environment within which to build packages. Getting pip install to compile correctly a binary without this clean environment can be difficult and not directly related to conda or anaconda.

One thing you could try is to use conda create -n build_env <pkg_list> to build a clean environment with the build dependencies of h5py and then try to pip install it in that environment. You might need to be sure you are using the same compiler as the one conda build uses --- setting appropriate environment variables.