add a CMake build system for the C code?

[oconnor663]

Is CMake the closest thing to a cross-platform standard for building C these days?

[k0001]

I've been using this in a project of my own:

CMakeLists.txt:

cmake_minimum_required(VERSION 3.9)

project(libblake3 VERSION 0.1.0 DESCRIPTION "BLAKE3 C implementation")
enable_language(C ASM)

include(GNUInstallDirs)

set(blake3_SOURCES)
list(APPEND blake3_SOURCES
  blake3.c
  blake3_dispatch.c
  blake3_portable.c)

if(CMAKE_SYSTEM_PROCESSOR STREQUAL x86_64)
  list(APPEND blake3_SOURCES
    blake3_avx2_x86-64_unix.S
    blake3_avx512_x86-64_unix.S
    blake3_sse41_x86-64_unix.S)
elseif(CMAKE_SYSTEM_PROCESSOR STREQUAL i686)
  list(APPEND blake3_SOURCES
    blake3_avx2.c
    blake3_avx512.c
    blake3_sse41.c)
  set_source_files_properties(blake3_avx2.c PROPERTIES COMPILE_FLAGS -mavx2)
  set_source_files_properties(blake3_avx512.c PROPERTIES COMPILE_FLAGS "-mavx512f -mavx512vl")
  set_source_files_properties(blake3_sse41.c PROPERTIES COMPILE_FLAGS -msse4.1)
elseif((ANDROID_ABI STREQUAL armeabi-v7a) OR
       (CMAKE_SYSTEM_PROCESSOR STREQUAL aarch64))
  list(APPEND blake3_SOURCES blake3_neon.c)
  set_source_files_properties(blake3_neon.c PROPERTIES COMPILE_FLAGS -mfpu=neon)
  set_source_files_properties(blake3_dispatch.c PROPERTIES COMPILE_FLAGS -DBLAKE3_USE_NEON=1)
endif()

configure_file(libblake3.pc.in libblake3.pc @ONLY)
install(FILES ${CMAKE_BINARY_DIR}/libblake3.pc
  DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)

add_library(blake3-shared SHARED ${blake3_SOURCES})
set_target_properties(blake3-shared PROPERTIES VERSION ${PROJECT_VERSION})
set_target_properties(blake3-shared PROPERTIES SOVERSION 0)
set_target_properties(blake3-shared PROPERTIES PUBLIC_HEADER blake3.h)
target_include_directories(blake3-shared PRIVATE .)
install(TARGETS blake3-shared
  LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
  PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})

add_library(blake3-static STATIC ${blake3_SOURCES})
set_target_properties(blake3-static PROPERTIES OUTPUT_NAME blake3)
set_target_properties(blake3-static PROPERTIES VERSION ${PROJECT_VERSION})
set_target_properties(blake3-static PROPERTIES SOVERSION 0)
set_target_properties(blake3-static PROPERTIES PUBLIC_HEADER blake3.h)
set_target_properties(blake3-static PROPERTIES COMPILE_FLAGS -fPIC)
target_include_directories(blake3-static PRIVATE .)
install(TARGETS blake3-static
  LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
  PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})

libblake3.pc.in:

prefix=@CMAKE_INSTALL_PREFIX@
exec_prefix=@CMAKE_INSTALL_PREFIX@
libdir=@CMAKE_INSTALL_LIBDIR@
includedir=@CMAKE_INSTALL_INCLUDEDIR@

Name: @PROJECT_NAME@
Description: @PROJECT_DESCRIPTION@
Version: @PROJECT_VERSION@

Requires:
Libs: -L${libdir} -lblake3
Cflags: -I${includedir}

[oconnor663]

(Just wanted to drop in a quick note and say that this is super useful, and thank you. I should have more time for these PRs after I get back from some travel.)

[xnox]

Is CMake the closest thing to a cross-platform standard for building C these days?

not really, meson is. which can generate ninja, xcode, vs-code, etc. Plus I find meson a lot easier to use. Many projects seem to be moving to meson and converging on it to be the cross-platform build system to use.

Although maybe CMake has larger marketshare still due to inertia, but declining.

[k0001]

J ust wanted to drop in a quick note and say that this is super useful, and thank you. I should have more time for these PRs after I get back from some travel.

Hey, no worries. I might take a stab myself at preparing this myself at some point, if you are interested.

[xnox]

If we are adding a proper build system, we should declare stable ABI of the C-library and make a soname, and have its symbols versioned.

[oconnor663]

I suppose there's no reason we can't have more than one build system, as long as someone's willing and able to maintain each of them, and we test each of them in CI :)

[k0001]

Hmmm… I'd rather make sure we have one thing that works well everywhere, is able to build static and shared libraries, and also makes it easy to directly include the C sources or object files directly in downstream projects if necessary. I know CMake can be used for this. I don't know about Meson.

[xnox]

Hmmm… I'd rather make sure we have one thing that works well everywhere, is able to build static and shared libraries, and also makes it easy to directly include the C sources or object files directly in downstream projects if necessary. I know CMake can be used for this. I don't know about Meson.

meson does all of that, and better. CMake actually is a lot harder to do all of that, in a correct manner, cross-platform, without odd toolchain files and quirks all over the place.

[oconnor663]

I've gotten the impression that this is something of a cultural split in the C/C++ world, and if there's a way to avoid taking a side on it in this project, it might be worth doing a moderate amount of extra work.

[k0001]

I wouldn't mind checking out Meson. Thanks @xnox for the hint.

These are some things that I think should be addressed in terms of packaging for a low level library such as this, that would ease its adoption:

• No external dependencies beyond the build toolchain.
• Widely available build tool.
• Fully deterministic and offline builds.
• Easy for Linux distributions to package.
• Easy for other languages to write bindings for.
• Shared library readily available.
• Static library readily available.
• Making it easy for downstream users to combine the objects in the static library with other objects of their own to create a new static or shared library of their own.
• Objects and source code themselves readily available. Usually this means that a downstream Meson/CMake project should be able to “include” this project, rather than just link against the built shared/static library.
• Linux, Darwin and Windows support.
• Android and iOS support.
• pkg-config support.

[oconnor663]

Once we start thinking about publishing precompiled shared libraries, one thing to consider is that a thin layer of C bindings for the Rust crate could compete for the same use case. Toolchain-wise, I think that has benefits and drawbacks. (Drawback: Now you need a Rust toolchain, and Rust doesn't support every system under the sun. Advantage: It's a lot easier to get modern Rust running on old distros than it is to get modern GCC, for e.g. AVX-512 support.) But feature-wise, the Rust code supports multithreading, which is a major advantage in large-input use cases.

Another aside: As we start investing more in C build tooling, it might make sense to reorganize the top level repo from "Rust code with a c/ subdirectory" to something more like "rust/ and c/ subdirectories".

[concatime]

Even if I recognize that Meson is easier to use than CMake, libblake3 is a fairly low level dependency (like zlib), and I would not be surprised if Python would integrate a module for Blake3 (like zlib..). The thing is, Meson is built on Python. So, to build Python, we would need Blake3, but to build Blake3, we would need Meson, and transitively Python. Welcome to the world of circular dependencies, the nightmare of all maintainers. Almost all low level libraries either use GNU Autotools or CMake (e.g. libzstd and zlib-ng). I see Meson relevant for high levels programs, like Sway, Mesa or GTK.

PS: As you can see here, Python already has a Blake2 module. So I expect Blake3 soon.

[concatime]

@k0001 I simplified your version and generated a WIP version: https://gist.github.com/concatime/e9b7c342fd348a8146aeaa026b743f61

[emmenlau]

Is CMake the closest thing to a cross-platform standard for building C these days?

At least it is really really wide spread, it supports quite a lot of platforms, and its quite actively developed. The cmake configuration from @concatime looks pretty decent, and other projects have started to create their own cmake instructions for blake3 already too. Even though it is simple enough, I would much prefer an "official" cmake build file from this project, because it adds more trust to the build.

[eli-schwartz]

The thing is, Meson is built on Python. So, to build Python, we would need Blake3, but to build Blake3, we would need Meson, and transitively Python. Welcome to the world of circular dependencies, the nightmare of all maintainers.

For the record, https://sr.ht/~lattis/muon now exists and is viable for many projects (but is still expanding support for the full meson DSL).

It bootstraps with nothing other than a c99 compiler and a posix shell.

So it is potentially practical to use meson, but make sure that either you stick to the ever-growing subset of meson that muon supports, or contribute whichever functions muon is lacking.

[concatime]

@eli-schwartz made me discover muon project, and I must admit I fell in love with ! (the project, not eli...) I made a fork to add a Meson build which works with muon too: https://github.com/concatime/BLAKE3/tree/meson-build. Right now, I only have access to a x86-64 machine. It would be awesome if someone tests my branch on x86 and/or AArch64. AArch64 is probably broken due to BLAKE3_USE_NEON behaving differently. Will fix it later. The only thing missing I think is Android support.

[eli-schwartz]

Neat, happy to be of service. (And also to submit patches making muon better -- with my Meson dev hat on, it's great to have multiple implementations.)

A couple of small notes:

• in order to use as a subproject you would need to make the c/ directory into its own standalone release or else make a top-level meson.build file, since subprojects must have a top-level meson.build.
• you may want to take a look at https://mesonbuild.com/Pkgconfig-module.html#pkggenerate (muon stubs this out)
• you can use declare_dependency() to create an interface object for use as a subproject