Use out-of-source build and install extensions in Python source tree / Use in-source build and clean up without removing cythonized extensions

[ashwinvis]

Motivation for using CMake

I am developing this python package called Fluidsim which use some Cython extensions as wrappers for FFTW libraries. Therefore while building these extensions (say for eg: fftw2dmpiccy) it becomes an issue when the following are not detected:

• CC: The machine uses a custom executable for the MPI C compiler. ie. it could be anything of these: cc/gcc/mpicc/icc. Also sometimes if one uses an anaconda Python distribution it might provide its own MPI compiler.
• libmpi.so : The MPI library is in a non-standard location
• libfftw3.so and libfftw3_mpi.so : The FFTW3 library is in a non-standard location.

Of course I could handle this by installing like this:

CC="cc"   \
CFLAGS="-I/path/to/include"  \
LDFLAGS="-L/path/to/lib"    \
LDSHARED="cc -shared -pthread" \
python setup.py develop

Then this becomes a labourious process of finding include paths and library paths, especially when you have multiple MPI libraries installed.

I was looking for something more generic and automatic, a one click solution.

I tried to use CMake, by following this example. It partially solves this issue with some help from FindMPI.cmake, FindFFTW3.cmake and FindFFTW3MPI.cmake modules (the latter two found in Github).

TLDR: The issue is..

I understand that CMake works best when the extensions are built in a separate build directory. and this requires copying all python source files into it (an out of source build). This is a clutter of files and deploying the project in development mode (python setup.py develop) is a nightmare. Running make clean would remove all the built extensions as well.

Possible solutions: (Not sure how to do it)

• Ideally, I would prefer an in source build. This would help to deploy the python project in development mode. Can CMake clean up after the build operation, without removing the built *.so files themselves? Also I would need CMake not to replace any existing Makefile.
• Can I use an out of source build and instruct Cmake to install the *.c and *.so files back into the source tree.

[thewtex]

Yes, CMake will do a much nicer job of configuring your build and doing system introspection.

Ideally, I would prefer an in source build.

A clean separation of source code and build artifacts is a CMake "feature".

Can I use an out of source build and instruct CMake to install the .c and .so files back into the source tree?

It is possible to change to the build location of a property with the LIBRARY_OUTPUT_DIRECTORY target property:

https://cmake.org/cmake/help/v3.4/prop_tgt/LIBRARY_OUTPUT_DIRECTORY.html

Set it with set_target_properties:

https://cmake.org/cmake/help/v3.4/command/set_target_properties.html

This would be called on the target created by cython_add_module.

Please give it a try. If it works well, we could consider adding a CMake option like PYTHON_MODULE_IN_SOURCE_BUILD that will always output the Python C-extension to the directory of the first source file, ...