Proposal: Support local build & install via third party build systems

[mikezackles]

My understanding is that third-party dependencies typically either come from the system or their builds must be ported over to meson. But this policy can become problematic when there isn't a good way to do system installs. For example, when cross-compiling, dealing with larger packages that would typically be built and installed on the host system becomes much more painful.

Porting large, complex packages to meson to be used as subprojects can present an undue burden because it means creating and maintaining a potentially complex unsupported third-party build, which is a time consuming source of potential bugs. Universal meson ports supported by the community are a nice idea, but in my experience, attempts at one-size-fits-all packaging are much more brittle than a package that is streamlined for the use case, especially when it comes to cross-compilation.

It is sometimes preferable to download, build, and install large packages locally via their officially supported mechanisms so that meson can see them as "system packages". Something like conan is an option here, but the complexity it introduces seems prohibitive for smaller projects that just want to bundle up a few dependencies. (I know I've personally tried it and abandoned it more than a few times.) There are other projects like jhbuild, etc., but my opinion is that using them isn't particularly ergonomic.

I ended up writing my own "packaging system" for these types of packages, which I use alongside subprojects for dependencies that are easier to wrap/port. This approach works OK for me, but my little script is just a shim, and I'd be thrilled if the build system could do this for me properly, so that I just need to write simple wrapper packaging for each dependency. For example, xmake appears to take this approach with its packaging.

I know this is potentially a pretty big feature, but it's an itch I've had for a long time that meson hasn't been quite able to scratch, so I wanted to at least mention it. Is this something that would be considered?

[dcbaker]

We have support for building cmake projects as meson dependencies through the cmake module which runs cmake in a "build-plan" mode and then generates meson commands. there's the external_project module, which calls configure+make style projects (and waf?) as a standalone executable. I have WIP patches for transparently converting cargo dependencies in the same way that the CMake works.

Apart from projects using other less popular build systems, what is missing?

[mikezackles]

Ha I kind of forgot that I'd seen those in the docs at some point -- thanks for reminding me! Nice to hear about cargo as well. I guess it's just been too long since I refreshed myself on newer (to me) meson features.

I was envisioning a sort of unified third-party packaging support, so at a glance these feel a little more structured than I had envisioned, but they do seem to cover a bunch of cases. (I can still see something like boost still being problematic, but that definitely falls under less popular build systems.)

I'll see what kind of mileage I can get out of the modules. Sorry for cluttering up the tracker :)

[mikezackles]

Apart from projects using other less popular build systems, what is missing?

One thing I noticed: Unless I'm reading the docs wrong, there is no way to specify dependencies between projects built using either the cmake module or the external_project module, is there? This seems like a significant limitation.

After thinking some more, I do still think that it would be really nice to have a simplistic sort of system package manager emulation for scenarios like cross-compilation where the normal system package escape hatch is not there. On a regular system, there are no build system restrictions around what can be packaged, and it would be nice to enjoy the same freedoms when cross compiling. Maybe the modules approach will evolve into this.

[xclaesse]

Meson is not a package manager, there are other options for that, like yocto.

That said, with external_project module, I think you can do pretty much everything as long as your projects users pkg-config to find their dependencies.

[eli-schwartz]

There's an open PR to teach cmake subprojects to recognize *-uninstalled.pc from meson subprojects.

[eli-schwartz]

think that it would be really nice to have a simplistic sort of system package manager emulation for scenarios like cross-compilation where the normal system package escape hatch is not there.

Sure it's there. As @xclaesse pointed out, Yocto is a thing. But more generally, if such functionality was needed and missing, and assuming the Meson developers decided that it's a need we personally need to solve... It would still be best solved by a second tool which wraps Meson or cmake or whatever. It would be a "developed by the same people that brought you Meson" kind of situation.

There are blatant limitations to the use of subprojects, which manifest extremely quickly any time someone tries to use one that isn't a native meson subproject. When it comes to the cmake subproject support, Meson has never promised anything other than "we believe the simple cases should work" (note: not "will").

[mikezackles]

there are other options for that, like yocto

Yuck, but this is a completely valid point of view. In my experience yocto was more painful than rolling something myself, but that could likely have just been me being dumb.

assuming the Meson developers decided that it's a need we personally need to solve

Totally valid. I'm absolutely not trying to come along and demand that you build me a package manager. I just wanted to point out that for someone like me, given meson's quality of implementation, it would be amazing if such a thing existed and either was a part of meson or integrated seamlessly. I know it's a mostly orthogonal problem from the dev's perspective, but from this user's perspective, an integrated solution, or even an additional, tightly integrated project as @eli-schwartz suggests, would be really really nice.

with external_project module, I think you can do pretty much everything

Interesting, I was reading this like it was only geared for autotools-type projects with a special case for waf, but I guess I need to look more closely. It makes sense to me that projects could "see" each other via pkg-config, but how is the build order determined between interdependent projects?

Meson has never promised anything other than "we believe the simple cases should work" (note: not "will")

That makes sense to me. cmake-specific support wasn't really what I was after.

Anyway, maybe this goes without saying, but might as well anyway: Thanks for all the awesome work thus far.

[eli-schwartz]

It makes sense to me that projects could "see" each other via pkg-config, but how is the build order determined between interdependent projects?

Generally, they can't. You can force them to, by manually crafting your build files to call the projects in the correct order. For meson subprojects specifically, Meson knows how to divert a pkg-config lookup into a subproject, calling that subproject as needed in the right order, but it can't do that for other build systems and must instead depend on project pkg-config files being created in time for other buildsystem lookups. Also, Meson creates pkg-config files at configure pointing to the build tree (the ones called -uninstalled.pc) but most other build systems do no such thing... so dependencies between two cmake or autotools subprojects may in fact be hopeless. I haven't explored that area myself.

[mikezackles]

so dependencies between two cmake or autotools subprojects may in fact be hopeless. I haven't explored that area myself.

If it helps, what I was envisioning is the ability to write the python equivalent of a PKGBUILD, local to my project, and have the third-party project's normal install step run and output to a shared directory inside the build. I know there are other projects that do this, but iirc last time I evaluated this they all felt like they were trying to orchestrate a moon landing (except maybe something like jhbuild). What I don't care about is the ability to share "packages" with other developers, orchestrate binary downloads, external package repositories, etc., because I think one-size-fits-all packages are really brittle for this use case and it makes more sense to just manually specify what to do. I just want to write a handful of PKGBUILD-like scripts, specify the dependencies, let them build and install, and have meson pick those up as system installs. (I know it's probably not quite that simple, but that's what I want, intuitively.)

[mikezackles]

As a real world example, I extracted this. (Please excuse my code.)

[dcbaker]

That looks remarkably like a full fledged package manager. Arch pkgbuilds and gentoo ebuilds look a lot like that

[mikezackles]

look a lot like that

I would guess that the package definitions in a hypothetical meson implementation would use python, but yeah -- I essentially wanted pkgbuilds. And yes, it would be a package manager, stripped down to the bare minimum needed to support packages written as a part of the project.

My thought was that this would remove or reduce the need to juggle external tooling across multiple platforms, making meson more of a one-stop shop. E.g., I typically cross-compile for Android on Arch and for iOS on macOS, and I could see building with msvc down the line, so if I'm using one or more external package managers, I need them installed/configured everywhere I'm building, which to me makes my crappy script start sounding more and more appealing.

This can definitely be accomplished with other tools, but in my mind doing so is not particularly ergonomic. That's not necessarily meson's problem, but I was pretty envious when I saw this workflow in xmake, and I thought it might be worth floating/suggesting here. But I can completely understand if the consensus is that any sort of package management does not belong in meson. And maybe it's just a really dumb idea in general :)

[jpakkane]

One of the design principles of Meson is that we only add features that are reliable and robust. The problem with supporting arbitrary third party projects and build systems is that it is a source of infinite support work. Random makefile projects (and others) can work and break in a multitude of dazzling ways and trying to keep on top of them is a nightmare.

It has been my personal experience (obviously other people may have different opinions and that is totally fine) that porting weirdo third party dependencies to Meson is less work than ducttaping them together and fixing unexpected breakages from here to eternity. This is, sadly, a lot more work up front but as the old saying goes there are no free lunches.

[mikezackles]

trying to keep on top of them is a nightmare

I believe this, but I do sincerely wish for some sort of middle ground. (Though I can understand the fear of a slippery slope.)

porting weirdo third party dependencies to Meson is less work

I was militant about this when I started with Meson, but over the years I have become much less sure. I understand the (good) reasoning behind it, but in some ways this line of thinking feels a bit siloed, akin to "rewrite it in Rust". Yes, there are real advantages to porting dependencies to Meson, but I also think that this can reach a point of diminishing returns, especially for deps that are essentially read-only.

Take the above example. These are interdependent projects, and some of them are crypto libraries. Do I trust myself with porting those? Maybe? I definitely don't feel confident about it. What if I update and something silently breaks behind my back? Should I treat someone else's unsanctioned third-party port as a black box? That feels even worse.

I'm sure there are tons of projects shipping broken builds out there (esp. those random makefile projects), but plenty of non-Meson projects are perfectly fine. And the pragmatist in me says that if the builds for these projects are good enough for distro packagers, they are probably good enough for me. Ultimately porting amounts to reaching around the public interface presented by a project and mucking with the internals. And if I'm thorough, it means that every time I update I need to comb through the dependency and make sure my port is in sync. IMO, sometimes that is worth it, and sometimes it just isn't.

obviously other people may have different opinions and that is totally fine

Agreed!