Cover Sentry's milksnake on the extension modules page

[ncoghlan]

Updated to refer to https://github.com/getsentry/milksnake instead

======== Original text: @mitsuhiko's https://github.com/mitsuhiko/snaek is an elegant approach to writing & distributing Rust-accelerated extension modules that minimises the number of binary wheels you need to create by relying on cffi to manage the integration with the Python runtime.

[ncoghlan]

Context for the recommendation:

• https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html
• https://kushaldas.in/posts/writing-python-extensions-in-rust.html

[pfmoore]

Does this technique (and the one for Go referenced in #360) work on Windows? I believe (but that was from a quick check, some time ago) that Rust uses the mingw behind the scenes, and so won't link with the correct CRT for Python on Windows.

If this is only a suitable option for Unix-based platforms, then we need to explicitly call that fact out as it's an important limitation (particularly as it's Windows where building binary extensions is most problematic, in practice).

[mitsuhiko]

Rust uses the mingw behind the scenes

Rust compiles with msvc. However it compiles with a different version than Python obviously. However why does it matter if its compiled with a different version of libc than Python?

[pfmoore]

@mitsuhiko Ah, that's good to know.

why does it matter if its compiled with a different version of libc than Python?

It only really matters because python-dev have never supported using extensions compiled with a different MS C runtime than the Python binary.

The specific issue is that if the .pyd that's loaded into the Python interpreter uses a different C runtime than Python uses, there's the potential for handles managed by one CRT to get passed to a different CRT. There's also different startup code invoked, AIUI. My understanding is that this is a theoretical issue, and in practice there's little evidence of actual crashes caused by this, but the advice generally given is that you must build extensions with the same C compiler[1] as your Python implementation was built with.

I'm more than happy if this works fine - and I don't even have a problem with people taking a "works for me" attitude to it. All I was flagging up was that we need to be careful if the PUG is making recommendations that conflict with the official python.org stance, and even then all I think is needed is a warning in the docs that users need to be aware of the issue.

[1] With the advent of the "Universal CRT" that's used by Python 3.6, that becomes "the same or a later version", I believe.

[mitsuhiko]

@pfmoore traditionally the reason that matters is because the c extension uses the same libpython as the main interpreter and assumes memory management between them is compatible. However with snaek all allocation and deallocation happens on the rust side of things and libpython is not even involved. That's why I don't think an actual issue can happen with a compiler mismatch.

[mitsuhiko]

Also there is no startup code involved in snaek.

[pfmoore]

@mitsuhiko OK, cool. (As the author of the code, I'm completely happy if you say it's OK on Windows - no need to prove it to me :smile:)

[mitsuhiko]

@pfmoore it does not work currently on windows because i did not care but it should work fine there if i get around fixing it.

[ncoghlan]

Right, the compiler mismatch problem relates to the use of CPython APIs that are outside the "stable ABI guarantee" (since the full API not only includes some APIs based on C runtime defined structs like FILE, but also allows extension-module allocated objects to be passed over to the core runtime to be deleted, and vice-versa).

Relying solely on cffi at runtime the way snaek does avoids that problem, since it's only cffi that's talking directly to CPython, not the Rust modules that cffi ends up loading.

[ncoghlan]

Updated based on the snaek -> milksnake change.