Release pre-built binaries

nathan818fr commented 7 months ago

The purpose of this pull request is to add an automatic way to release pre-built binaries for libmem. (it follows the discussions on #142)

Considerations

For Linux

GNU libc and libstdc++ are backward compatible, but not forward compatible. To maximize compatibility, libmem should be built with the oldest version of these libraries. So compiling on the oldest Debian version that is still supported seems a good choice (v10 "Buster" currently).

It's the same for musl. So compiling on the oldest Alpine version that is still supported seems a good choice (v3.16 currently).

References:

For Windows

Since MSVC 2015, C++ binaries are backward compatible (/GL and /LTCG should not be used), but not forward compatible. To maximize compatibility, libmem should be built with the oldest compatible version of MSVC (v141 toolset for i686 and x86_64, v143 toolset for aarch64).

References:

https://learn.microsoft.com/en-us/cpp/porting/binary-compat-2015-2017?view=msvc-170

MSVC also has another compatibility constraint: All modules passed to a given invocation of the linker must have been compiled with the same run-time library (/MD, /MDd, /MT or /MTd). So libmem should be built with all runtimes, to allow users to choose the one they want.

References:

Changes

Improve static library bundling

Enhancement of the Linux static library bundling process: It now automatically gathers the linked libraries, and invoke ar from CMake.

Additionally, a similar bundling process has been introduced for Windows (using lib.exe).

Add release script

Addition of tools/build-release.sh, which is used to build and package a release on all platforms:

linux-gnu-x86_64
linux-gnu-aarch64 (compilation fail, ARM on Linux seems not supported by libmem)
linux-musl-x86_64
linux-musl-aarch64 (compilation fail, see above)
windows-msvc-i686
windows-msvc-x86_64
windows-msvc-aarch64

The platform naming is inspired by the rustc target names.

It is important for me to have a release script that doesn't rely only on GitHub Actions, as it:

allows testing the release process locally more easily;
prevents vendor lock-in.

For linux platforms, the script uses Docker to build (see tools/docker-env/linux-*.Dockerfile). This allows compilation with older versions of gcc and glibc/musl (see compatibility considerations above). Multi-arch support is provided by Docker+qemu-user-static, which allows to run ARM containers on x86_64 hosts.

For Windows platforms, the script must be run from a Windows host (using Git Bash, Msys2 or Cygwin, see tools/local-env/windows-msvc.sh). It requires vcvars-bash (disclaimer: I'm the author of this utility) to setup the Microsoft C++ toolset from Bash. This avoids the need for a non-Bash script to build on Windows. It was added as a submodule in extern/vcvars-bash. Multi-arch support is provided by the Microsoft C++ toolset, which allows to compile for ARM and i686 from x86_64 hosts.

external/CMakeLists.txt has been modified to respect the CMAKE_MSVC_RUNTIME_LIBRARY variable:

CMAKE_MSVC_RUNTIME_LIBRARY is forwarded to external projects;
for capstone, CAPSTONE_BUILD_STATIC_RUNTIME is set accordingly;
for keystone, LLVM_USE_CRT_* is set accordingly.

This allows building libmem with any runtime library (/MD, /MDd, /MT or /MTd) on Windows (see compatibility considerations above).

The script generates the following artifact structure:

linux-gnu

``` ├── include/ │ └── libmem/ │ ├── libmem.h │ └── libmem.hpp ├── lib/ │ ├── shared/liblibmem.so │ └── static/liblibmem.a ├── licenses/[...] └── GLIBC_VERSION.txt ```

linux-musl

``` ├── include/ │ └── libmem/ │ ├── libmem.h │ └── libmem.hpp ├── lib/ │ ├── shared/liblibmem.so │ └── static/liblibmem.a ├── licenses/[...] └── MUSL_VERSION.txt ```

windows-msvc

``` ├── include/ │ └── libmem/ │ ├── libmem.h │ └── libmem.hpp ├── lib/ │ ├── shared-MD/libmem.dll │ ├── shared-MDd/libmem.dll │ ├── static-MD/libmem.lib │ ├── static-MDd/libmem.lib │ ├── static-MT/libmem.lib │ └── static-MTd/libmem.lib ├── licenses/[...] ├── MSVC_VERSION.txt └── WINSDK_VERSION.txt ```

Add release workflow

Addition of .github/workflows/release.yml, to automate the release process using GitHub Actions.

The workflow is triggered when a tag is pushed, it:

runs tools/build-release.sh on all platforms;
create a GitHub Release with the generated artifacts (marked as pre-release, so it can be edited before being published).

The workflow can also be triggered manually for testing purposes (screenshot). It's possible to specify the branch/tag to build, and also to choose platforms to build for. In this case, there is no GitHub Release created (but the artifacts are still uploaded as workflow artifacts).

Here is the workflow result for tag TEST1:

Potential issues / Interrogations

Windows artifacts are huge once uncompressed (120 MiB compressed -> 700+ MiB uncompressed), because they are compiled for multiple runtime libraries:

shared (.dll): /MD and /MDd runtimes (/MT seems less common for shared libraries, so I didn't include it)
static (.lib): /MD, /MDd, /MT and /MTd runtimes

Maybe all runtimes are not needed? I already removed /MT for shared libraries, but others seem really common.

Maybe windows artifacts should be split between MD and MT runtimes (e.g. windows-msvcMD and windows-msvcMT)? But it seems confusing for users.

Or maybe all artifacts should be split between shared and static libraries (e.g. linux-gnu-shared, linux-gnu-static, windows-msvc-shared, windows-msvc-static, etc.)?

--

Since the workflow is relatively long to finish, it is not run on every push. The ability to trigger it manually seems enough for testing purposes. But maybe a scheduled nightly run on master would be useful?

--

Maybe a README file should be added in the release artifacts? With a link to this repository, a brief explanation of the artifact structure and a note about compatibility considerations.

--

I'm disposed to make changes. :smile:

rdbo commented 7 months ago

Great commit :100: Notes:

As for the potencial issues:
- I'll try to figure out about the runtimes issue on Windows
- I think automatic builds on release + manual builds for testing is good enough
- If no solution is found about the runtime issues, we can add a README to inform the user

nathan818fr commented 7 months ago

A final comment, even if the commit has been merge, if ever some comes here...

In the end: The term "platforms" has been changed to "targets", and artifacts have been split to include either the static version or the shared version. And for Windows, there are even 3 targets to divide by the used runtime library: shared-md, static-md and static-mt (shared-mt is not required as you can safely link to it from an MT project).

Consequently, there are more concurrent runners and the execution time has dropped from ~32 mins to ~12 mins.

rdbo / libmem