Precompilation support

[josevalim]

Hi @warmwaffles!

A couple months ago, the amazing @cocoa-xu added support for precompilation in elixir_make. Is this something of interest to you and the project? This can help specially with Windows users.

If you are, we would be happy to fund towards making this happen, or getting someone to make it happen. :)

Thank you.

[josevalim]

Btw, once precompilation is operational, the workflow for releasing new versions is slightly different. You would tag, push the tag, wait for GH Actions to precompile (already automated), and then run a command to download the precompiled artefacts and checksum them (also already automated), and publish the package.

There is an example with everything hooked up, including CI here: https://github.com/cocoa-xu/cc_precompiler_example

[warmwaffles]

@josevalim I'll take a look at this. The only thing this would conflict with is https://github.com/elixir-sqlite/exqlite#advanced-configuration specifically the "Database Encryption" portion.

I do like the potential speedups for installation.

[josevalim]

@warmwaffles you have full control over when to use the precompiled stuff. So if that env var is set, you disable precompiled download. If something is missing, we can add the missing features to make this possible. :)

[warmwaffles]

Oh that's brilliant. I'll take a stab at this.

[cocoa-xu]

Hi @warmwaffles, also please feel free to ask me anything, I'll be very happy to help you! :)

[fire]

I would like to also enable this for https://github.com/fire/elixir_mvsqlite. Rust complication is difficult

[josevalim]

@fire for Rust there is https://github.com/philss/rustler_precompiled/

[fire]

I am not using rustler, it is the build make system of exqlite + a library with c abi built by rust.

[warmwaffles]

@cocoa-xu and @josevalim Just trying to figure out how precompiled binaries work. When you do a mix deps.get will that also pull the precompiled binaries from github, or when I invoke mix hex.publish locally, those are pulled down and pushed up to hex.pm?

[warmwaffles]

Or you know, I could read https://github.com/cocoa-xu/cc_precompiler/blob/main/PRECOMPILATION_GUIDE.md 😆

[warmwaffles]

Since I want to support as best as possible the last 3 versions of OTP (right now 25, 24, and 23), I'll need to build precompiled binaries for each version right? That shouldn't be difficult to do with a matrix, it just makes the precompiled binaries balloon.

[cocoa-xu]

Hi @warmwaffles

When you do a mix deps.get will that also pull the precompiled binaries from github,

No, these precompiled binaries will only be pulled when the user does a mix compile (either directly or not, e.g., execute mix test). And only the precompiled binary that matches the user's machine will be pulled from GitHub. If none of these precompiled binaries matches the user's machine, then we will fallback to compiling from source.

or when I invoke mix hex.publish locally, those are pulled down and pushed up to hex.pm?

No, there were no changes made to mix hex.publish. These precompiled binaries will only be pulled when you do a mix elixir_make.checksum, and then a checksum.exs file will be generated (this file should be specified in package/0 in mix.exs!). These precompiled binaries will not be pushed to hex.pm: when the user does a mix compile, they will be pulled from GitHub. :)

[cocoa-xu]

support as best as possible the last 3 versions of OTP (right now 25, 24, and 23), I'll need to build precompiled binaries for each version right?

For these shared libraries (NIFs), as long as erlang:system_info(nif_version) stays the same then we don't have to build the precompiled binaries for each OTP version (instead we need to build for each nif_version that you'd like to support). ;)

[warmwaffles]

Precompilation support added!