Dependency checks should automatically offer to install the necessary -dev packages

[nirbheek]

Similar to what some distros have for missing commands:

$ jython
bash: jython: command not found...
Install package 'jython' to provide command 'jython'? [N/y] 

This will really improve the 'build a project for the first time' experience, which is pretty shit for all build systems in the world right now, as detailed in https://github.com/mesonbuild/meson/issues/985#issuecomment-267258425.

Must also include an option for disabling this when building in CI, for instance.

[sarum9in]

Note that this is distro-specific. For example on ArchLinux we don't have dev packages at all. There are different ways to find missing files, maybe we should aim for more general approach to look for a package with a file, such as libfoo.pc, via available package managers (e.g. dpkg, alpm, rpm?)...

[nirbheek]

maybe we should aim for more general approach to look for a package with a file, such as libfoo.pc, via available package managers

The feature I mentioned above is implemented with PackageKit, which provides a distro-agnostic wrapper for fetching and installing the package that provides that executable. It's called pk-command-not-found, and it's provided by the package PackageKit-command-not-found on Fedora.

So we would use the same thing.

[nirbheek]

In case someone wants to work on this, two things are needed:

1. A way to know which dependencies we should prompt for, and which not. https://github.com/mesonbuild/meson/pull/3376 should help with that.
2. A Python implementation of pk-command-not-found.c that searches for pkg-config files instead of executables. PackageKit has Python bindings through gobject-introspection: from gi.repository import PackageKitGlib

[ferdnyc]

I'll just note that we attempted to use PackageKit to do distro-agnostic dependency installs in GSConnect. (Which is a GNOME Shell extension written in pure JavaScript+Python, distributed via the extensions,gnome.org catalog. It can make use of, but obviously can't be distributed with, various libraries/executables/etc. typically installed using distro package management.) Our attempted support for automated installs caused us no end of problems — to the point where we eventually just abandoned the idea entirely. The issues we hit seem like they'd apply at least equally (if not more) in the context of dependency management for Meson.

Off the top of my head, the major ones would be:

1. This quote from the PackageKit FAQ:

   PackageKit does not do dependency resolution. This problem has already been solved by the backend systems and we don't really want to re-invent the wheel.

2. The fact that pk-command-not-found cheats, in ways which make it very hard to modify for other types of searches.

   1. PackageKit can only search for very concrete things: package names, full pathnames to files within a package's contents, etc. pk-command-not-found takes advantage of the fact that it knows it's searching for commands, and just hardcodes two standard PATH directories as prefixes (unless you override it in the local configuration): pk-command-not-found.c@611:614

          /* fallback */
          if (config->locations == NULL) {
              g_warning ("not found SearchLocations, using fallback");
              config->locations = g_strsplit ("/usr/bin;/usr/sbin", ";", -1);

      Then it searches for matches at each of the configured paths.

   2. But .pc files don't live in standard, cross-distro directories. On one distro they might be in /usr/lib64/pkgconfig, on another it'll be /usr/lib/x86_64-linux-gnu/pkgconfig. If the package is 32-bit, that's a different path on both of those systems. If the package is arch-independent, the .pc file might end up in /usr/share/pkgconfig, or it might not.

   3. Because of point 1, PackageKit can't help determine the correct package to install, when there are multiple packages that can provide the same file. So it returns them all, which is exactly the opposite of what you want from a dependency-resolution standpoint. That's not really a problem for /usr/bin/ and friends, since the PATH has a clear, well-defined resolution order and typically only one package owns any given /usr/bin/name file — generic names are managed with symlink aliases or tools like alternatives. But it can be a real headache when trying to resolve things like library dependencies, where there might be multiple versions available for multiple architectures.

3. PackageKit is ultimately a user-oriented, interactive frontend to the distro-specific package management systems. Whether it's being used from Gnome Software or the pk-command-not-found helper, the focus is on installing a single package (and possibly its dependencies), at the user's explicit request. Facilities for initiating bulk package-management actions on behalf of the user, like installing a set of package dependencies for a build, are basically nonexistent — by intentional (lack of) design. So if you need to install 7 separate dependencies, you may end up with 7 discrete PackageKit transactions — each of which will then prompt the user for their password, if the backend is configured that way. Oh, yeah, and that prompt will be a confusing message that gives no indication that it's asking for authorization to perform an install action on behalf of Meson, because PackageKit doesn't have any API for that.

[ferdnyc]

Fedora's rpmbuild, OTOH — which does exactly what it says on the tin — can leverage the dnf package-manager to install the full set of dependencies required to build a package from a given RPM .spec file. It's able to do that because it already has the list of dependencies in the exact format dnf uses — the .spec file contains a list of BuildRequires: that it can just hand over to dnf in bulk to be resolved, after which it'll offer to install what's missing.

But like @sarum9in, I'm pessimistic about there being any way to genericize that in a distro-agnostic way. PackageKit for sure can't do it. In fact, the most damning sign is that efforts in those directions seem to have largely been abandoned — these days, rather than trying to unify or interface with the distro package managers, the focus seems to have shifted to containerized solutions (Flatpak, Snap, Docker, etc.) that isolate packages from the underlying distro as completely as possible.

[eli-schwartz]

Fedora's rpmbuild, OTOH — which does exactly what it says on the tin — can leverage the dnf package-manager to install the full set of dependencies required to build a package from a given RPM .spec file.

That's exactly the same thing that every other package manager's build system does, whether it be makepkg -s, emerge, dpkg's mk-build-deps, xbps-src, or what have you.

The format doesn't really matter, though, if two distributions use the same format but have a different set of packages available.

[eli-schwartz]

So right from the very start:

Similar to what some distros have for missing commands:

But, this is simply https://www.gnu.org/software/bash/manual/html_node/Command-Search-and-Execution.html item 3, the command_not_found_handle() hook, which may be defined anywhere at all.

Arch Linux has one of these provided by the pkgfile package, which can be sourced to provide the same user experience, and also incorporates this option:

  -b, --binaries          return only files contained in a bin dir

I suppose it would be possible for meson to do such custom lookups using a strict distribution mapping, and assuming the necessary programs are installed on the distro (they aren't always, by default, and archlinux pkgfile uses a special database, much larger than the pacman one, that contains file manifests too). It would very often not work out of the box, though.

[ferdnyc]

That's exactly the same thing that every other package manager's build system does, whether it be makepkg -s, emerge, dpkg's mk-build-deps, xbps-src, or what have you.

Oh, agreed. I was using it as an example, of how non-distro-agnostic tools can leverage their knowledge of the local package ecosystem to provide features that PackageKit, by very nature of its being distro-agnostic, will never really be able to match.

[ntonnaett]

Are there equivalents to dnf install "pkgconfig(dependency)" in other package managers that could be used?

[CoderThomasB]

A basic thing meson could do is detect your distro uses DNF or other package managers and suggest that you install the package using a command. Like the following:

ERROR: Dependency "sofia-sip-ua-glib" not found, tried pkgconfig and cmake.
You can try runing 'sudo dnf install "pkgconfig(sofia-sip-ua-glib)"' to install this dependency.

What do people here think about this?

[eli-schwartz]

detect your distro uses DNF or other package managers

How do you suggest to reliably do that? It's trivial to install many different package managers on the same system, for example to use those package managers to create chroots of the OS that uses it (see tools like debootstrap, pacstrap, osbuild, mkosi). So it's not enough to check which package manager is installed, you also have to check which package manager claims ownership over the system.

[CoderThomasB]

you also have to check which package manager claims ownership over the system.

While that is entirely true, I think because what I am suggesting is just a supplement to the error message, leaving the user to copy and paste the command on their own. It is probably fine if meson suggests using DNF even if the user doesn't want to use it, because then the user would just ignore the suggestion and use their preferred package manager. And it would help people like me who now only need to copy and paste a command rather than typing something.

[eli-schwartz]

even if the user doesn't want to use it, because then the user would just ignore the suggestion

Actually they will get very confused and possibly report a bug saying that meson told them to do something which didn't work.

And even on real dnf-based distros it will still return factually incorrect results if the dependency in question doesn't have a pkg-config file at all, only a cmake one -- meson does support cmake dependencies. It could also falsely trigger for config-tool or built-in dependencies that don't have either pkg-config or cmake available... so overall, I think the only way to do this in practice is to actually check the package manager to see if it reports a file as existing, which was the original suggestion.

This quite recent suggestion to use dnf-specific magic pkg-config aliases won't work, I'm afraid. It doesn't actually solve the challenging part of the problem, which is determining whether offering a suggestion is going to be constructive.

[eli-schwartz]

Robustly implementing the proposed feature will need to:

• figure out the correct platform package manager, maybe via trial and error
• use the package manager facilities to do a (regex-based) file lookup
• on successful lookups, report the useful command for installing the dependency

This could be done for:

• pkg-config files
• cmake files (beware: cmake supports a variety of names)
• find_program lookups
• dependencies with config-tool lookups
• cc.find_library patterns
• cc.has_header