Advice for packaging wheels built for different CPUs (i.e. --march=native or avx)

[viblo]

Problem description

Hi! I wonder if there's some examples / ideas how to (if possible) package cffi libraries compiled against specific CPU instructions? (I imagine the same issue comes up with other non-pure packages as well, not just with cffi)

I have a library, Pymunk, which uses the Chipmunk2D c library. I did some tests, and by compiling with -march=skylake for my CPU (an Intel skylake laptop cpu) I get a 5% performance boost under Ubuntu in WSL. I would not be surprised if there's more improvements to have by newer CPUs as well. I have also considered testing out if SIMD instructions in the Chipmunk2D library could increase performance.

But this is all good when compiling locally. However, can this be packaged up for Pypi/conda?

Some ideas I have:

• Let end users (which require high performance) compile themselves, maybe add some easy option to setup to build for native?
• Do runtime detection for SSE/AVX etc, for parts where it matters most.
• More options?

(Im not sure how much this is about packaging python or if its more of a general package binary libraries problem regardless of language. Feel free to close if out of scope)

[jeanas]

Have a look at https://pypackaging-native.github.io/key-issues/simd_support/

The TL;DR is that with PyPA packaging tools, this is really difficult to do.

[viblo]

Thanks! That page is exactly what I was looking for!

I did look at the general Pypa Packaging guide at https://packaging.python.org/en/latest/ (which is also how I found I could open an issue here), but I didnt know there's a separate site with information around native packaging. Even if that site is outside the Pypa I think it would be nice with a link somewhere on the Packaging guide, to help people like me.