search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.7k stars 1.76k forks source link

Meson port #11723

Open Salamandar opened 3 years ago

Salamandar commented 3 years ago

I suggest to use the Meson build system.

This is a modern and explicit build system (meaning it doesn't do anything we didn't ask), that uses a readable typed language. It's also backed by the Ninja build tool, that is way faster than Make.

One of the features of Meson is that the configuration options are splitted into a specific file (meson_options.txt) that can easily be read by users/packagers. With Autotools, you have to read the whole configure.ac file to find where the options are defined.

Also, Meson provides easy support for compiler warning levels, LTO and such.

Meson is already provided by all main Linux distribution repositories (CentOS included, even if it's an old version). It builds out-of source, and supports cross-compilation.

Meson won't be able to build kernel modules, it will only call kbuild (for now, I think), but will at least provide feature detection, and configuration header generation.

Package generation (rpm/deb) can be done, even if it will be a bit tricky (I didn't read how it's currently done).

I already ported some softwares to Meson, so if the maintainers are interested, I could gladly work on it !

Kokokokoka commented 3 years ago

Please, no. This adds python as a hard build dep. Why not switching to perforce jam or redo then? (Don't take this advice too seriously, even though it could work and would solve some complexity problems and doesn't add (your fav lang here) as a build dep)

Salamandar commented 3 years ago

This adds python as a hard build dep.

Kokokokoka commented 3 years ago

let's take a look at linux-headers from gentoo:

DEPEND="app-arch/xz-utils
    dev-lang/perl"
RDEPEND=""

where is python? gdb:

RDEPEND="
    dev-libs/mpfr:0=
    >=sys-libs/ncurses-5.2-r2:0=
    >=sys-libs/readline-7:0=
    sys-libs/zlib
    lzma? ( app-arch/xz-utils )
    python? ( ${PYTHON_DEPS} )
    xml? ( dev-libs/expat )
    source-highlight? (
        dev-util/source-highlight
    )
    xxhash? (
        dev-libs/xxhash
    )
"
DEPEND="${RDEPEND}"
BDEPEND="
    app-arch/xz-utils
    sys-apps/texinfo
    virtual/yacc
    nls? ( sys-devel/gettext )
    test? ( dev-util/dejagnu )
"

so, no python as a hard build dep please, don't use your system to justify something in totality.

Salamandar commented 3 years ago

Uh… Well maybe Gentoo splits packets into smaller packets, I don't know. But here on archlinux,

Anyways, my point was that Python is now installed on most (if not all) setups nowadays. It's not really a restrictive dependency. I'll wait for the developers' opinion on this.

bghira commented 3 years ago

it's not a hard dep, just required for building. don't let python get in the way of replacing a miserable build system.

gmelikov commented 3 years ago

Can we support every OS with it (FreeBSD, Linux, MacOS, Windows)?

Does FreeBSD already have it as a dep system-wide? If no, then it may be a big no-go here, last time I told about python as a build dep for them - it was a definite no.

Is there a strong pros for a project?

Kokokokoka commented 3 years ago

It doesn't add anything, but complexity. It doesn't solve anything. "faster" is not better. There is no need to introduce new build deps, because arch/etc is using python everywhere. There are other distros, there are other OS.

bghira commented 3 years ago

Can we support every OS with it (FreeBSD, Linux, MacOS, Windows)?

https://docs.freebsd.org/en_US.ISO8859-1/books/porters-handbook/building.html#using-meson

Does FreeBSD already have it as a dep system-wide? If no, then it may be a big no-go here, last time I told about python as a build dep for them - it was a definite no.

well, no, but zfs tests already require python, and building/running those should be a part of their merge flow.

faster is better, because repeatedly building ZFS is a hindrance to development.

gmelikov commented 3 years ago

well, no, but zfs tests already require python, and building/running those should be a part of their merge flow.

IIRC zfs tests is not a mandatory part of base system.

EDIT: in addition, you can build ZFS without python support now.

gmelikov commented 3 years ago

As a contributor, build time is not a critical problem for me now, it's up to 10 minutes on a slow machine.

For now I don't see big gains in this idea, we'll have a lot of problems on every OS we want to support soon (it's not only Linux and FreeBSD), so using conventional tools looks better for me.

Salamandar commented 3 years ago

Can we support every OS with it (FreeBSD, Linux, MacOS, Windows)?

Sure, and more than that.

Does FreeBSD already have it as a dep system-wide?

Looks like it's the case : https://docs.freebsd.org/en_US.ISO8859-1/books/porters-handbook/building.html#using-meson

EDIT: in addition, you can build ZFS without python support now.

Python would only be a build dependency, not a runtime dependency.

By the way, other low level stuff like ZFS already use Meson :

As a contributor, build time is not a critical problem for me now, it's up to 10 minutes on a slow machine.

Build time is not the main argument here (although meson is a huge speed bump). Autotools is difficult to read and maintain, and as I said before it does not provide a centralized place for build options like meson does.

For now I don't see big gains in this idea, we'll have a lot of problems on every OS we want to support soon

Well, more OSes means more complicated build system code. And Meson usually divides the number of lines of code by 3 or more. And the language is way easier to read.

bghira commented 3 years ago

As a contributor, build time is not a critical problem for me now, it's up to 10 minutes on a slow machine.

including ARM SoC like Rpi3? for me it takes more than 40 minutes with ccache and other fun toys.

we'll have a lot of problems on every OS we want to support soon (it's not only Linux and FreeBSD)

this is a premature optimisation.

until you introduce that "other (unnamed) OS" support to the repo, it shouldn't be a consideration. what OS is it, anyway? Windows? NetBSD? OpenBSD? someone else who hasn't been a part of the discussion at all until this moment?

Kokokokoka commented 3 years ago

Why not cmake though? It can use ninja, so your build will be "faster" and no python hard build dep.

Salamandar commented 3 years ago

Why not cmake though ?

CMake has multiple drawbacks and Meson was created with those in mind, mostly related to the language (non-typed language). Also, CMake is not really LFS (Linux file system) friendly. For example, FreeCAD still installed libraries and icons into /usr/share/freecad/{lib,icons} because of CMake, until a patch recently.

I heavily used (and still use a bit) CMake some years ago (imagine porting a whole company to CMake), and it's appropriate in some really complex cases where Meson is not adapted (but on those really complex cases, you tend to think the whole architecture is flawed...). So I don't hate CMake. But using Meson is a breath of fresh air in comparison.

And, once again, my proposition of porting to Meson is NOT about speed. (or, not only)

Kokokokoka commented 3 years ago

Why one can think of this as a bad idea: it changes one flawed thing to another. There is no magic bullet. And when someone advertises changing the whole build base to solve some of the autotools problems, this person introduces new problems and new complexity. While autotools just work. Speaking of premature optimisation, I think, this is the case. You change something that works to something you personally like more. For now meson is trending, the next day there will be just another favourite build tool. This is not about engineering decision, this is about trend. And it introduces python as a hard build dep. While now nullself can build zfs without python, and I really hope it will stay that way.

IvanVolosyuk commented 3 years ago

kbuild uses make. Can meson generate makefiles for kbuild? Can this tree be copied over to kernel using something like 'copy-builtin'?

Salamandar commented 3 years ago

it changes one flawed thing to another. There is no magic bullet.

Well let's keep using FORTRAN on MIPS then ;)

This is not about engineering decision, this is about trend

That's partly true. But if Meson is trending, maybe that's because the engineering behind Meson is correct and solves the issues present in Autotools (and in a smaller extend, in CMake).

Can meson generate makefiles for kbuild ?

While this is not "standard" for Meson (meaning that Meson usually does not generate makefiles), yes this is 100% possible. Auto-generated files are a standard process (whether it's using the meson functions configure_file or custom_target) and we can install them right where they are needed, or call kbuild to build kmodules.

adamdmoss commented 3 years ago

TBH I think that any build system overhaul (whatever that looks like - Meson or CMake or whatever) has to wait until we have a good handle on the requirements of the incoming OSX ZFS port (perhaps Windows too), to see what the real pain points are with the current system.

Would hate to see this all overhauled just to have to do it again soon.

Salamandar commented 3 years ago

TBH I think that any build system overhaul (whatever that looks like - Meson or CMake or whatever) has to wait until we have a good handle on the requirements of the incoming OSX ZFS port (perhaps Windows too), to see what the real pain points are with the current system.

That's fair. Although, porting an autotools software to windows is harder than with meson ;)

You know what, I'll slowly start working on a POC, then open a WIP merge request.

bghira commented 3 years ago

having a working replacement build system is certainly a lot more helpful than what people who don't like it have offered - nothing.

bghira commented 3 years ago

let's take a look at linux-headers from gentoo:

i can't believe you used gentoo as an example of why not to use python.

that whole distribution is based on python. try uninstalling it, see what happens.

most end-user Linux systems don't even have a compiler on them - nor do most Linux server installations. should we ensure ZFS builds without a compiler? all these different compilers with build options and quirks, it'd be better to just write machine code directly and never need to worry about compiling.

tying ZFS to libc at all kind of limits us in general. what if there's a different libc library on an OS we want to support? Linux and FreeBSD have two very different implementations. maybe we should rewrite ZFS so that it runs on its own kernel and no longer relies on the existence of any OS-specific libraries or kernel facilities.