Building Corrfunc with conda-build

[nickhand]

We (@rainwoodman and I) would like to use Corrfunc for MPI-enabled correlation function calculations in nbodykit (see bccp/nbodykit#355, for a start). We prefer to have all of the dependencies available on conda, either through the defaults channel or our own bccp channel.

We've built Corrfunc on Linux no problem but are having some issues on Mac when conda-build tries to set the linking paths properly. Have you built on Mac with conda-build before? The issue we are seeing is here:

https://travis-ci.org/bccp/conda-channel-bccp/jobs/247771299#L1451

otool -L spits out that the countpairs_mocks.so object is linked against countpairs_mocks, which is creating issues because that isn't a valid file name and conda-build can't find it.

[manodeep]

Thanks for reporting the issue. First of all, we would really love to have a conda install option for Corrfunc and all my previous attempts have failed for OSX. I do not know how to fix it but I am happy to help in any to diagnose and find a solution.

What I recall was that the _countpairs_mocks.so needed to be copied into a different path. Ultimately, this might be caused by the fact that I have tried to do all the installation via make even when python setup.py has been called.

What do you mean that _countpairs_mocks.so is not a valid filename? I am not sure I understood that -- is there some naming convention that you are referring to?

[rainwoodman]

Is it possible to get rid of make? We had pretty good experience with that when wrapping class and other packages.

[manodeep]

@rainwoodman Unlikely to get rid of make. Currently python isn't a dependency for Corrfunc and I would like to keep it that way.

There are future plans to create a separate and more generic python-only package. But that's a different story :)

[rainwoodman]

I mean setup.py doesn't necessarily need to call make if you are willing to flatten the directory structure a bit. Some of my similar packages follow this approach. e.g. bigfile has cmake, make and setup.py:

https://github.com/rainwoodman/bigfile

[nickhand]

I got it to build with conda on Mac by renaming the install_name to include the .so extension. See e957ce52. I don't think this breaks anything else.

[rainwoodman]

I looked at the directory structure. I see two possibilities:

• approach 1: add a Makefile target that prints the paths of all of the dependency files (source and header, separately) for the libs target, recursively. Then we can run that target once and decide which files to pass to setuptools for building a single extension so file.

• approach 2: add a target (recursively) insist all libraries are built with static (.a) and -fPIC. Then link the python extension with the .a files. This should expose most of the linking problem at the linking stage, creating a single .so for the python extension. an example is https://github.com/rainwoodman/pfft-python/blob/master/setup.py This could be done by fiddling with CFLAGS and LDFLAGS, too I suppose.

[rainwoodman]

I still think a single .so file is easier to access than many .so files for the long run. That being said, since your minimal fix works I won't pursue this further.

[manodeep]

@rainwoodman Thanks again for the suggestions. Under the current scheme, every Corrfunc API creates a static library (i.e., files are always compiled with the -fPIC option). These static libraries then linked to produce two shared libraries (the python extensions) -- one for theory routines and one for mocks. Keeping the python routines in separate shared libraries helps maintain different mental contexts for the mocks and theory routines.

@nickhand Would it be possible to update the meta.yaml file and/or send a PR?

Thanks so much to both of you for checking up on this.

[nickhand]

@manodeep yep, I'll put it in a PR

[manodeep]

See PR #131.

@nickhand @rainwoodman Are you guys okay with closing this issue?

[nickhand]

sounds good to me!