New library metasystem

[H5117]

Hi! In the process of porting an application to Meson I met two design issues of pkg-config, described here and here. Firstly, pkg-config does not know what type of a library it deals with so it can't return correct flags for linking without adjusting of either metafiles or CLI parameters. Secondly, pkg-config in some situations requires presence of metafiles which in theory it does not need.

As aforementioned issues can't be fixed without breaking compatibility, I suggest to design a new library metasystem. My thoughts on new design are following.

A metafile description

A metafile with some name and extension '.lm' corresponds to a library with the same name. This allows to determine more easily the type of a library (shared, static, both) by checking existence of corresponding files. A metafile has the following format:

# This file contains metadata describing the library 'foo'
# All fields are optional (empty file is a valid file)

[Library]
Version=1.0.0
Description=A library for ...
URL=https://example.com/

[Paths]
PathLibrary=/full/path/to/the/library
PathHeaders=/full/path/to/the/headers1
PathHeaders=/full/path/to/the/headers2

[Dependencies]
RequiredHeaders=bar1/bar2/bar3 # refers to the .lm files
RequiredLibraries=bar1/bar2 # refers to the .lm files or libraries in standard location
Versions=bar1 >= 1.0.0/bar2 >= 2.0.0

[C/C++]
Definition=definition1
Definition=definition2=1024
CompilationArgument=-pthread
LinkingArgument=-pthread

[Variables]
UserVar1=true
UserVar2=1

The format imply different dependency chains for compiling and linking. This allows to express critical, visible and invisible dependencies (link) without overlinking or excess CFLAGS or headers.

How it is supposed to work

1. Meson retrieves standard search directories for libraries from a compiler (g++ -print-search-dirs).
2. A user declares a dependency for a project: some_dep = dependency('foo', type: 'compile-time'|'link-time'|'both').
3. For the type: 'compile-time' Meson searches for a file 'foo.lm' in a path provided in a parameter or an environment variable. If the metafile is found Meson calculates compiling flags by combining Paths.PathHeaders, C/C++.Definition and C/C++.CompilationArgument from each metafile in the dependency chain. If any metafile can't be found Meson raises an error (metafiles is the only way to locate headers).
4. For the type: 'link-time' Meson searches for a file 'foo.lm' in a path provided in a parameter or an environment variable. If the metafile is found Meson calculates linking flags by combining Paths.PathLibrary, the name of the library (it is equal to the name of the metafile) and C/C++.LinkingArgument from each metafile in the dependency chain. Dependency chain is calculated taking into account the type of a library some metafile belongs to. If any metafile can't be found Meson assumes that provided name directly refers to the library in the standard location (and checks it at configure time, and gives a warning if this is a static library).
5. For the type: 'both' (default) Meson does both points 3 and 4.

Comments are welcome.

My questions are:

1. Is a Meson team interested in a new library metasystem?
2. Seems with explicit declaration of types of dependencies Meson can generate suggested metafiles automatically in most cases (if you don't use preprocessor definitions in headers). Am I wrong?

[jpakkane]

Is a Meson team interested in a new library metasystem?

Basically yes, but note that such a thing is a big task and should really be its own project. It should also not be a Meson-only thing but should also support things like CMake et al. You should also note that this file format should handle Visual Studio's different runtime types natively as that is one of the big reasons why they do not want to support Pkg-Config.

[mensinda]

There is already some work on the CMake side on this, however, the progress seems to have stalled since they want to evaluate pkg-config first (with currently unknown progress).

Some resources on the CMake side for this:

• https://mwoehlke.github.io/cps/schema.html
• https://gitlab.kitware.com/cmake/cmake/-/issues/20106

CPS files (once the spec is finished) would be very interesting and probably relatively easy to produce and consume.

cc @mathstuf

[dcbaker]

I'd also thought about a replacement for pkg-config. If thought about using json as the format, I'd be willing to sign up in the meson side of there's interest from cmake to work on something together

[deepbluev7]

DDS also has a similar format, but it is a bit to simplistic to reuse at the moment, imo: https://github.com/vector-of-bool/dds

[dcbaker]

I've gone and read the CPS schema. It seems fairly reasonable, and likely to cover most of the problems that pkg-config has. Since there seems to be some interest on the cmake side to actually implementing it, I've gone ahead and opened a bunch of issues. If anyone else is interested the github is here: https://github.com/mwoehlke/cps/

[dcbaker]

Also, since it's JSON we could avoid the "call an external tool" problem :)

[mcatanzaro]

CPS is receiving some attention now due to this talk at cppcon. Looks like the CMake ecosystem will be moving this direction.

[marktsuchida]

Another CppCon talk on CPS (by Conan cofounder) is now public and goes into details. Also a new project URL: https://github.com/cps-org/cps

[dcbaker]

I'm going to leave this here: https://github.com/cps-org/cps/issues/30

I'd love for CPS to take off and be a thing, but at the moment I'm concerned that it's still basically just serving the needs of CMake, and I don't feel like my legitimate concerns as someone who needs to implement CPS support are being taken seriously.

[eli-schwartz]

I'd love for CPS to take off and be a thing, but at the moment I'm concerned that it's still basically just serving the needs of CMake, and I don't feel like my legitimate concerns as someone who needs to implement CPS support are being taken seriously.

This was very predictable, yes. That's exactly the kind of development approach I would expect from cmake users specifically.

It's how we got into the "turing-complete foo-config.cmake" situation to begin with.

[mwoehlke-kitware]

To be clear, folks at Kitware are generally confident that pkg-config isn't the answer. The pkg-config community, understandably, begs to differ, but to my knowledge no one has made any serious effort at using pkg-config to solving the problems with pkg-config. So that's something of a NIH problem.

I realize CPS is on "temporary hiatus". We're hoping that will cease to be the case Any Day Now™. Meanwhile, are there specific concerns you feel need to be addressed? (Please understand that I have mentally filed several of them as "yes, we need to do something about this", but I haven't had time to do so.)

[mcatanzaro]

This was very predictable, yes. That's exactly the kind of development approach I would expect from cmake users specifically.

This is not a helpful comment.

[eli-schwartz]

To be clear, folks at Kitware are generally confident that pkg-config isn't the answer. The pkg-config community, understandably, begs to differ, but to my knowledge no one has made any serious effort at using pkg-config to solving the problems with pkg-config.

As far as I can ascertain from an outside perspective, folks at Kitware spent about five seconds trying to use pkg-config, discovered that it used Unix flags on Unix and Microsoft flags on Windows, and refused to touch it ever again.

So much so that cmake has aggressively attempted to stop users from using pkg-config even on Linux, even to detect dependencies that are distributed with pkg-config files only. There is no effort to meet existing userbases halfway.

If you view pkg-config as a key/value line-delimited database with a couple of magic fields, which is exactly what it is, it turns out you can get pretty far. As far as I can tell, there's no data which can be sideloaded into a cmake variable, that can't also be sideloaded into a pkg-config variable.

In terms of actual compile interfaces (Cflags/Libs, in pkg-config parlance), cmake people have told me a bunch of times that using native platform flags for this is bad "because then we have to translate it from Unix to $CURRENT_PLATFORM". I always wonder in return "okay so why not make your best effort at actually doing that, for the sake of the existing wide adoption", but I've yet to get an answer.

(To put this into perspective, meson has gone to incredible lengths to make our best effort at translating cmake config files into meson. For the sake of the existing wide adoption of cmake.)

Perhaps it would be easier to assess why this isn't a viable forward-looking solution, if cmake offered it and documented what's wrong with it, and people could try it out for themselves and observe the failures. Meson already does something like this though, because like cmake we prefer to not deal with raw compiler flags in situations where we don't have to. Thus far our efforts at taming pkg-config seem to have overall worked?

If nothing else, you want this as a backwards compatibility mode for people who aren't yet using CPS. while preferring CPS wherever possible. Unless the cmake answer is "if projects aren't using CPS we should punish their users by making their users write labyrinthine custom hacks.

So that's something of a NIH problem.

I'm sorry, is this supposed to be cmake criticizing others for NIH? You do realize that pkg-config, being a generic interoperability tool, is exactly the opposite of NIH? The people who argue in favor of using pkg-config aren't the authors of pkg-config, they are the diverse and happy userbase representing numerous communities.

CMake, the ancient and venerable bastion of NIH, is attempting to break into the world of interoperability standards, which is admirable. We may very well get a fantastic modern replacement for pkg-config out of it. But it would be somewhat helpful to know whether it's an interoperability standard or just a cmake-specific tool which happens to not be turing-complete.

Precisely how CPS handles the concerns of non-cmake users will play a large part in the answer.

[eli-schwartz]

And in particular

To be clear, folks at Kitware are generally confident that pkg-config isn't the answer.

https://gitlab.kitware.com/cmake/cmake/-/issues/20106#note_671029

Says:

For reference, work on CPS has stalled because we don't think it can gain acceptance until we first make a serious attempt at using the pkg-config .pc format for install(EXPORT) and export() implementations. Such an attempt would either succeed (great!) or provide proof that the format is insufficient. In the latter case we'd then need to work with pkg-config folks to address the limitations. Only if that doesn't work out would a new format like CPS be truly justified.

As far as I can tell from the public record, that attempt was never made?

Even if it proved the format today is insufficient, and no effort was made to work with pkg-config to address the limitations, a major part of that argument was that proving it won't work is, in itself, valuable for the sake of convincing the users of a dozen different independent build systems that cmake is justified in claiming everyone needs to move to CPS.

[dcbaker]

@mwoehlke-kitware, https://github.com/cps-org/cps/issues/2 comes to mind as a pretty important issue that I haven't seen movement on.

I have a partial implementation of a pkg-config style tool for cps written in C++ (my goal is to having something drop in compatible for pkg-config so that someone could, in theory, do PKG_CONFIG=cps-config <legacy build system>, and possibly more capable than that.

[mwoehlke-kitware]

cps-org/cps#2 is the sort of change I don't want to unilaterally make, even though I'm inclined to agree with it. "A community model of changes" doesn't just mean "not blocked by an unresponsive maintainer", the community needs to exist and be active in discussions. I'm all for improvement in that respect!

FWIW, I felt your pain when I did the CMake demonstration implementation... but even if CPS does become case-strict, I don't know that I'll rip it out. BLIWYA...

[dcbaker]

@mwoehlke-kitware, I'm going to move that dsicusion to the CPS issue

[bretbrownjr]

Hi all! Nice thread!

I'm just chiming in here to welcome anyone who's interested to participate in working on these kinds of issues. The ecosystem really needs specs that can work across build systems and reasonably port to new contexts (like new package managers or OSs).

The best way to get looped in would be to join the Ecosystem Evolution mailing list. It's a Google Group, but only because it seems to be the easiest way to set up a shared mailing list at the moment. We have monthly catchups with an open agenda, minutes from previous meetings, and so on.

As to technical substance, @mathstuf wrote a paper on why some of us are pessimistic about extending pkg-config to meet our needs. See P2800. I'm happy to elaborate if anyone wants to dig into that reasoning. But we're feeling defining libraries as such is important and pkg-config is a format for defining partial compile and link commands. These things are related, but it's error prone and non-portable to take some flags and back into a definition of a library. That being said, it's important to me to provide a nice bridge between existing pkg-config users and future technology defining libraries more directly. My organization will have an adoption curve that is far too steep if we cannot figure out something ergonomic there.

[dcbaker]

@bretbrownjr I did send a request to join the group, and I'd be happy to show up to the meeting at the end of the month (I don't currently have any conflicts for that). I've also been working on a pkg-config style tool that could be used as a drop in replacement for pkg-config, but using CPS files, which might be of interest, though it's still in very early development.