proposal: transition OpenRC to using the Meson build system

[williamh]

All,

I am interested in transitioning away from the current makefiles OpenRC uses to a build system based on meson [1].

In 0.24, my plan is to make both build systems available and document building OpenRC via meson. Then, at some point in the future, I want to retire the Makefiles.

The possible down side I see is that this will add python 3, ninja and meson as build time dependencies.

The upside is a build system which will handle parallel builds and portability to multiple operating systems better than our current makefiles. Also ninja is much faster than GNU make, so it will decrease build times.

I am opening this issue to gather ideas both for and against this proposal.

[1] http://mesonbuild.com

[awilfox]

Any reason to not use CMake? It is just as portable, can output Ninja, and requires only a C++ runtime and expat (which are going to be on almost every Linux and BSD computer as well). It is also used by a large number of other packages including LZ4, Poppler, Doxygen, Conky, librpc, MySQL/MariaDB, most of KDE, all of LXQt... so it is very widely used and would have a much higher chance of already being installed on another system.

Note that I have nothing against Meson (it seems fine), and IMO Python 3 is something every home should have; nevertheless, I'm just putting it out there that CMake would be another alternative with great portability and flexibility without pulling in Python and "Yet Another Build System".

[skarnet]

I am concerned that all those new build-time dependencies are themselves difficult to build and would make bootstrapping a lot more difficult. OpenRC is a service manager, i.e. low-level infrastructure, and should be easy to build from scratch; if you have to build some heavy environment before you can build OpenRC, people who attempt to make their own distribution will suffer. (I don't use OpenRC, but I do build my own environments and spaghetti dependencies are the single most time-consuming issue I face.)

[kaniini]

OpenRC is a core bootstrap dependency for Alpine, Python and Ninja are not. While Python and Ninja would be $makedepends_host dependencies, it may lead us to avoid upgrading OpenRC.

[williamh]

@ncopa What do you think about what @kaniini says? Would it stop Alpine from upgrading OpenRC if we switch to the meson build system in the future?

@skarnet If you are referring to folks using something like Linux from scratch, I'm at a bit of a loss, I'm not interested in autotools, so that would leave either cmake or meson.

@awilfox I haven't really looked at cmake, but I am willing to research it.

[williamh]

@awilfox I have heard that the cmake syntax is difficult at best, so unless it has some significant feature that meson doesn't have, I think I want to stik to meson for now if this goes forward.

[williamh]

@skarnet Do you know of something besides meson that you think works well and does not require generated things in the tarball e.g. no configure scripts.

[kaniini]

@williamh to clarify: i did not say it would or would not unilaterally stop us from upgrading in the future, just that it may cause us problems bootstrapping. for example: it is not known what support meson has for cross builds, which continued improvement of crossbuild support in alpine is an ongoing goal, and crossbuild support is a requirement for bootstrap packages, as we crossbuild to bootstrap new architectures. if such a switch were to cause regressions with crossbuilds, it would put us in the position of having to avoid new releases until the regression were fixed.

[kaniini]

to provide further clarification (in case it wasn't obvious): we don't really care per se as long as you don't break crossbuilds

[skarnet]

I do not know the current state of OpenRC, but:

• Is the current situation so unworkable that something really needs to change?
• It's 2017, autotools is entirely unnecessary. Clean, simple Makefiles are generally enough. AFAIK, OpenRC is a lot of shell and a bit of C; I do not see why you would need anything more than a Makefile. git is an order of magnitude more complex than OpenRC, and it doesn't use a configure script, just a Makefile and make variables.
• Unfortunately, all "higher-level" build system alternatives have problems, one way or another; those problems are more subtle than the problems with autotools, but they're there. Spaghetti dependencies is only one of them; support for cross-compilation (i.e. build-time test bypassing), configurably static builds (I want libX.so and libY.so, but libZ.a), are examples of others. For big projects, the tradeoff may be worth it, if using a high-level build system really simplifies the build configuration; for something as basic as OpenRC, I don't think it is the case.

Some people like to say that autotools is a disease projects catch. I tend to think the same of most high-level build systems.

[williamh]

@skarnet I know for a fact that parallel builds are broken [1]. I can't reproduce the bug, but with parallel make, that doesn't mean it doesn't happen, it just means I'm getting lucky.

The way I see it, the possible solutions are:

1) force '-j 1' everywhere (very hacky) 2) rewrite the makefiles (if I do that I want a non-recursive make structure) 3) go to a high level build system which handles all of this and portability.

[1] https://bugs.gentoo.org/show_bug.cgi?id=374191

[kaniini]

@williamh if we contribute a patch fixing the makefiles would that be an acceptable solution? Alpine policy requires that the OpenRC version we ship be cross-buildable. There is very little documentation about Meson and cross-builds, leading me to believe it is not a formally supported option. The current build system works fine for cross-building.

[williamh]

@kaniini Ok, cool, like I said above, I'm not going to kill the Makefiles right away. Is this the documentation you are talking about for cross-builds? I've not used this functionality. https://github.com/mesonbuild/meson/wiki/Cross%20compilation

[kaniini]

@williamh yes, and while we can generate those files the process seems really half baked compared to the current cross-build situation for openrc :)

[awilfox]

CMake uses a similar type of file for cross-compilation. The only significant feature it has that Meson doesn't is that it doesn't have any complex [R]DEPEND on most platforms, just C++ and expat. (Python's XML libraries also require expat, as do git, gettext, gdb, polkit, dbus, and LVM, so expat is more than likely going to present everywhere.) So, no, if you aren't comfortable with the CMake syntax, it probably won't benefit you at all.

If I may further elaborate on @skarnet's question: what does OpenRC need done that is hard to do portably using regular Make? I'm guessing it has something to do with either SELinux, FreeBSD, or Gentoo Prefix? Just curious.

[awilfox]

I can additionally add from my experience as a developer of Adélie Linux (and various other embedded projects) that Python is a horrible mess to cross-compile unless you use 3.5 or 3.6 only. In Python 3.5 they have actively worked on the build system to support cross builds. Even still, not all modules will build in that configuration, so you are always wasting work (because you will have to rebuild it again natively). Again, that is not always a bad thing (and is required to bring up Portage on a new platform anyway), but it is something I thought was important to note.

(If you really want 3.4, you will need at least one commit from 3.5 to support it.)

[kaniini]

@awilfox meson is a host tool and therefore does not require cross-compiled python

[alonbl]

Hi,

I am old school, if you like I can help migrate to autotools.

I find these the best build system even after the emerge of new ones. The only one that is flexible, supports cross-compile properly and have no build time dependencies.

I've done this to many projects, usually the build system turns out very clean. The bad name of autotools is mainly due to people that misuse them.

[williamh]

@alonbl Thanks for the offer. Besides the difficult m4 syntax, My concern is that I don't want to add generated content to the release tarballs. If we convert to autotools, the configure script would have to be checked into git, or I would have to require the user to generate it. I would opt for requiring the user to generate it. That means autotools would have to be on their systems to build OpenRC.

[kaniini]

Right now to do a cross build of openrc, all we need to do is set the CC environment variable to the cross compiler. My main concern is that with Meson, it will be considerably more complex.

[alonbl]

@williamh: I understand that m4 is difficult, but as I wrote, experienced people can produce a very simple notation, especially for simple projects like openrc.

However, I do not understand the generate issue... The major advantage of autotools is that you checkin only your sources, while separating between build machine and packaging machine. The packaging machine should contain additional dependencies (autoconf/aclocal/m4/automake/libtool), while once packaged, the package can be transferred to the build machine with minimal dependencies. When you run "make dist" autotools provides a tarball with all required additional files to enable build without user generate the autotools helpers nor build machine has extra dependencies.

As I am sure you aware of that, there is probably other issue in your mind :)

[williamh]

@kaniini Thanks for explaining what your concern is, I'll see if I can get an answer about how that works.

[williamh]

@alonbl I don't create tarballs manually at all. They are now generated by GitHub for the releases, and the current make dist target generates the same tarball by using the "git archive" command. To keep doing that, I would have to check the configure script into git since "git archive" doesn't support adding generated content to the tarballs.

[alonbl]

@williamh: I believe you are confusing between github service of exporting a tag and packaged tarball to be used at build machine. You can upload additional artifacts for a tag, including a packaged tarball for the release into github to be used by people. There is nothing wrong in packaged tarball to reduce the requirement of build machine and ease build, especially for core package like openrc.

On 31 January 2017 at 18:56, William notifications@github.com wrote:

@alonbl I don't create tarballs manually at all. They are now generated by GitHub for the releases, and the current make dist target generates the same tarball by using the "git archive" command. To keep doing that, I would have to check the configure script into git since "git archive" doesn't support adding generated content to the tarballs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ncopa]

I share @kaniini's concern. Would be nice to not make bootstrapping a new arch more complex than it already is. Only depending on gnu make would be really nice, but I don't think ninja and meson will block us.

I can also mention that we have looked at ninja for apk-tools. ninja seems really nice, but you need something that generate the ninja files. We were thinking of making some shell script for that to avoid python dependency.

[kaniini]

Yes, ultimately it comes down to what we can do about bootstrapping. If we can make it work, great, if we can't, then we will deal with it when the time comes.

[williamh]

@alonbl No, I'm not confusing them, I'm not interested in uploading additional artifacts. If I converted to autotools, I would not include any generated content.

[alonbl]

Hi @williamh,

So let me summarize... you have plain make build system but it is too difficult to maintain especially because parallel build. You reject autotools concepts per its packaging methodology that reducing build dependencies (please do not use autotools as long as you reject their methodology). You consider a build system that has dependencies such as python for a very basic system package.

As I wrote, I am old school. I believe a package, especially base packages such as openrc which I love, should use minimal build requirements. Had it used python it would have logical to use python as a build system, in the past the trend was scons now it is something else. This leaves us with plain make and autotools (with proper packaging).

I am not trying to enforce anything, just set the facts right and use tools in the way they were designed to. I offer my help to port and maintain openrc build system using autotools if you choose to go this route.

[nirbheek]

Hello, I'm a former Gentoo developer and a contributor to Meson. @williamh asked me to talk a bit about Meson (specifically the cross-compilation support) so I thought I'd pop in and say a few words.

My go-to build system of choice has always been either plain make (for small self-contained projects) or GNU Autotools. However, as we all know, Autotools is something you use not because you love it, but because everything else is (usually) much worse. Autotools is an architectural marvel, but you can't spend 9 years working with it and retain affections for it.

However, Autotools and Make have set a series of standards that are critical to building and packaging on Linux, and nothing else has managed to replace Autotools precisely because no other build system so far has understood the problems that it solves and how best to solve them.

The work we've been doing in Meson is meant to learn from Autotools, undo the mistakes that are baked into it, and do things it never could. I gave a talk last year about this too. For instance, pkg-config files are the preferred way to find and use dependencies.

I'll try to address some of the issues raised here.

Spaghetti dependencies is only one of them;

1. In my opinion, the dependency system in Meson is much more robust and specific than something like Make or Autotools where you have to pass everything as compiler flags in environment variables, and if you're not careful you run the risk of having your project's targets compile/link with the system-installed version of your headers/library rather than the one you just built!

support for cross-compilation (i.e. build-time test bypassing)

2. Meson has well-integrated support for cross-compilation. If your project does not need to be aware that it is being cross-compiled, all you have to do is pass a cross-file that tells Meson what to do. It is really no more difficult than setting CC/CXX/etc. The same file also specifies the values for build-time tests that must be bypassed. Another good example is the cross-file used by libepoxy. The advantage of this approach is that it allows you to also support much more complicated cross-build setups. For instance, running tests under qemu or wine (exe_wrapper). Another example is if you build a tool and then use it to generate something else, Meson handles this case much better than Autotools. All you have to do is pass the native : true keyword argument while defining the build target for the tool and it will be built with the native compiler.

configurably static builds (I want libX.so and libY.so, but libZ.a), are examples of others

3. Meson makes it easy to switch your entire project between static or shared libraries. You can also force a specific library to be either static, shared, or a module.

Python

4. Meson is written in Python for ease of development, but it doesn't have (and will never have) any dependencies outside of the Python standard library. It requires Python 3.4.x and Ninja 1.5.x, and you can rest assured that we will never require anything else.

5. Meson and Ninja are an order of magnitude faster than Autotools/Make, and this is especially true while building on slow machines (mips, arm, etc). This is despite having done no speed-related optimizations to the Meson parser at all. As a data-point, in the GStreamer project, we found a 2-3x speedup when using the Meson build system compared to Autotools on fast machines, a ~5x speedup on a Raspberry Pi 3, and a 10x speedup on MinGW/Windows. Ninja is also much faster than recursive (and even non-recursive) Make, so this is also relevant for OpenRC. :)

6. Most importantly, Meson is not a static target. We want to constantly improve and evolve (while being backwards-compatible!) based on people's use-cases. If you find yourself having to jump through hoops to do something, or if you have issues using Meson, please pop into #mesonbuild on FreeNode or file a bug and we'll get right on it!

Hope this helps. :)

[kaniini]

Meson and Ninja are an order of magnitude faster than Autotools/Make, and this is especially true while building on slow machines (mips, arm, etc). This is despite having done no speed-related optimizations to the Meson parser at all. As a data-point, in the GStreamer project, we found a 2-3x speedup when using the Meson build system compared to Autotools on fast machines, a ~5x speedup on a Raspberry Pi 3, and a 10x speedup on MinGW/Windows. Ninja is also much faster than recursive (and even non-recursive) Make, so this is also relevant for OpenRC. :)

raccoon:~/openrc$ time make -j8
[...]
real    0m 3.59s
user    0m 7.07s
sys 0m 0.78s

How is build time relevant when it takes 3 seconds on a typical machine (a Core i5-6600)? It only takes 1 minute on even Alpine's slowest builder...

I am not ripping on Meson here, I just question how build time is relevant to a project that takes literally 3 seconds to build.

[nirbheek]

How is build time relevant when it takes 3 seconds on a typical machine

I used to think the same thing when I used Autotools in GStreamer. Making a few changes and rebuilding would usually take 3-5 seconds, which I was ok with. But now with Meson/Ninja it is almost instantaneous and it's sort of addictive. :)

You are absolutely right though that it is not a major concern. Just thought I'd put it out there because it's something that I like about Ninja.

[kaniini]

OpenRC builds the entire thing in literally 3 seconds from scratch though. Building just the changed files likely takes much less time.

Again, not trying to rip on Meson here: I think Meson has a lot of potential for a lot of projects due to it's good engineering decisions (such as embracing pkg-config / .pc files heavily) -- I just don't see it being useful for OpenRC. Again, it takes seconds to build, and not even a minute on my slowest builder.

[kaniini]

Also I fixed the "parallel build" issue in #117 (it had nothing to do with parallel builds).

[skarnet]

I can't speak for or against Meson because I don't know it, but the simple fact that it's using Python makes me say that it's not a good choice for OpenRC: bootstrapping OpenRC should not require bootstrapping Python (which is an order of magnitude more difficult to do, and two if it's a cross-build).

[williamh]

Sorry if I misstated my intent. As I've said the plan is not to get rid of Makefiles yet, but to try out Meson as an experiment. I opened this issue so that I can get some more testing for this in Alpine Linux and other distros that use OpenRC. I think it would be interesting to see how well it works.

[kaniini]

If you want to try it as an experiment, that's fine, but we're really not interested in using anything other than the makefiles which already work for us. The last time an upstream broke our build process, pkgconf happened.

[Elizafox]

As someone who's developed software that works across the BSD's, Linux, Windows, etc.:

Unless you need runtime detection of OS features (what features does OpenRC need to detect?) and linkage against libraries (the init system should only need libc ideally), this is a pointless endeavour. A simple makefile is plenty sufficient for OpenRC afaict. Why break what already works?

[williamh]

@elizafox OpenRC isn't that minimal. We link to libc, libutil, libdl, libpam, libpam_misc, libncurses/libtermcap,libselinux, libcrypt, libaudit, and libkvm. From what I'm seeing , Some of these libraries are O/S specific (libkvm is only referenced on *bsd), some are libc specific (libdl and libutil are part of glibc but I don't see references to them in musl upstream), some are completely optional, and some are o/s specific and optional. Right now, we definitely do not have a "simple" makefile. See my previous comment for my current thoughts on this issue.

[awilfox]

Just for what it's worth:

x86_64-builder01 ~ # ls /usr/lib/lib{dl,util}*
/usr/lib/libdl.a  /usr/lib/libutil.a

We use musl libc at Adélie, and they ship empty .a files for compatibility with glibc.

[stalkerg]

Hello! I am voting to CMake. CMake syntax is not really pain, it's normal language. Also, CMake support not only Ninja and you can generate standard Makefiles. Besides I can help with CMake because:

1. I use OpenRC every day on my Gentoo systems.
2. I know CMake very well, and support project for porting PostgreSQL to CMake. ( https://github.com/stalkerg/postgres_cmake )

[kaniini]

fwiw, we plan to use meson as default build system in pkgconf 1.4 series (1.3.x is an LTS series), which means that most likely meson would become a bootstrap dependency anyway.

[williamh]

I haven't forgotten about this issue, and I still plan on transitioning to Meson. It has been put on the back burner for a little while because of other issues, but it will be happening farely soon.

[williamh]

Part of the transition will be to keep the makefiles around for a few releases to gather feedback on whether there are issues with the meson build that need to be fixed.

[liugang]

I think meson is better than pure Makefile, CMake, Autotools, It simple to write, easy to read, and maintain.

I see all gnome project is porting to mesonbuild, and systmed, gstream are using mesonbuild as default buildsystem now.

ninja has prebuilt binary on their release site, and it only depend libc.

meson only depend on python3.4+, and all distribution is going to use python3 instead of python2 as default python recently.

meson using python as backend to generate ninja files, if you like, you can using any language to parse meson script, e.g. C/C++, go, rust, etc.

meson support cross compile very easy than CMake, just write a file, tell it where is gcc. you don't need set CC=xxxx, CPP=xxxx, and so on.

[stalkerg]

I also changed my mind, and now I also vote for Meson. The CMake can be better for C++ and some tricky cases but for OpenRC meson will be greater.

[stalkerg]

ahh, it's already merged! Fine!