(Optionally) use a proper build system

[enkore]

setuptools' Extension with custom Cython steps "kind of" works, but has many restrictions and annoyances:

• No parallel compilation
• Wrong / incomplete change detection: Editing a C file does not trigger recompile, need to manually delete target or touch Cython source
• Poor compiler support (commercial UNIX compilers like SunCC)
• No library detection, requiring custom search code
• No support for feature detection at all, requiring the ugliest of hacks to support ISA extensions

Problems using non-setup.py setup:

• Harder to integrate into "pip"-based installation (may need similar kludge like now with Cython - e.g shelling out to cmake)

[RonnyPfannschmidt]

if you set up a way to do setup.py build with the shell out it should be fine - for a good measure try to get to the point where you can build wheels

[enkore]

Turns out that there already is tooling for glueing CMake into setup.py: http://scikit-build.readthedocs.io/en/latest/

The scikit-build package is fundamentally just glue between the setuptools Python module and CMake. Currently, the package is available to perform builds in a setup.py file. In the future, the project aims to be a build tool option in the currently developing pyproject.toml build system specification.

In that case it may be possible to just always build using CMake, which would be rather nice (much better than Extension anyway). Having CMake as a build requirement is no big deal.

[enkore]

Note: A proper build system would allow us to use the better CRC32 routines (crc32_clmul.c) on more platforms than just GCC (sans OpenBSD, not including anything clang!).

The same argument would apply to shipping other routines, e.g. there are BLAKE2 implementations using AVX2 which are 30-40 % faster on Haswell and later than the scalar (but ILP-heavy-by-design) code we use now; but it's impossible™ to ship that using setup.py. Or at least overly difficult and annoying.