Version Requirements

[bretbrownjr]

Overview

Up front, I'll say that I'm for adding features for version checking.

That being said, I'm concerned that we need to be clear about what version compatibility checks provide and what they do not provide. I'm also thinking we should perhaps set intricate version checking as a relatively low priority for now.

Framing

To frame the discussion, it's important to distinguish between two kinds of versions:

• source version: the version of the project source code as published; this will typically have a 1-1 mapping to some commit identifier in the upstream version control system

• binary version: the identifier of the package as installed; there will be a 1-N mapping from a given commit in the upstream version control system to this binary

Note that sometimes binary versions can be compatible with other binary versions. This is common to see in libraries that provide stable ABIs. These are often libraries like language runtimes installed in system directories.

Confusingly, the source version is a pretty good determinant of API changes for a given library. That is, what changed in relevant header files for C and C++. But it's a pretty bad determinant of overall backwards compatibility for a given release.

Also note that binary versions are sensitive to and can be invalidated if downstream users choose incompatible:

• Preprocessor definitions
• Language versions
• Versions of transitive dependencies

That is, it's very uncommon for a dependency to bump a major version because one of its dependencies also bumped a major version. But logically in many respects, major version bumps should require this. To pick a silly example, if boost deleted boost::string_ref (it won't), any libraries that use string_ref in their interfaces would have immediate API breaks as well. Any libraries that use string_ref in their implementations might additionally have ABI breaks.

So Why Check Versions?

I do think checking versions is a very helpful "shift left" activity for end users, though. If there is a real break, I would rather see "you need a new version of the jabberwocky library" than "undefined reference to jabberwocky::internal::EventListener::snickersnack".

Aside: I'm not sure "cannot find the jabberwocky library" on its own is a whole lot better, for the record, especially if the package manager reports that jabberwocky-devel is in fact installed. But I'll assume we can work on excellent error messages once we decide what we're supporting.

If a library maintainer is breaking something, it would be excellent to allow that to be annotated. Same if a package maintainer is breaking something. And it's frequent enough that the library maintainer accidentally breaks something (see above) and the package maintainer makes some annotations to help everyone else cope with reality.

So my current thinking is versions are annotations by project maintainers to provide good diagnostics. They are not going to provide correctness. Other tools, especially tools that manage and build dependencies, are responsible for that step. And those tools will be opaque to CPS since CPS files are an output of that dependency management and build process, not an input.

On Version Ordering

So I'm OK with versions. Especially build identifiers of some sort make a lot of sense. For very few projects like header-only projects with no preprocessor tricks and no dependencies, semantic versioning can be fairly high quality as well, though that's a very niche use case. Very stable libraries with intentionally stable ABIs would also benefit. There are a lot of usage of those, but fairly few of them in number, however.

When Is This Issue Done?

I'm thinking we should mostly deprioritize version support in getting cps-config to a minimum-viable-product except to the extent that we have practical adoption hurdles for sufficiently interesting ecosystems. If there's a champion semver that wants to do nothing but work on this, I won't be especially upset, but I think the priority needs to start at more fundamental activities like modelling graphs well, perhaps adding some pkg-config interop, and things like that. I also expect that any sufficiently advanced needs by end users can be served by being able to pin specific dependencies to specific CPS files.

So I think this issue is done if we get consensus about the priority of version ordering features in an initial release. My current thinking is versioning is a "nice-to-have" but it is not especially urgent.

We'll probably need a "forget all the version info" mode anyway, at least for troubleshooting workflows for package maintainers. We can start out developing that and explicit CPS file selection as the only supported initial modes.

[bretbrownjr]

I'll suggest the Conan 2.0 documentation on versioning, revisions, and lockfiles required reading for anyone that thinks all this is just about sorting strings:

https://docs.conan.io/2/tutorial/versioning.html

It's not a feature. It's a whole family of features. And, again, they're really features best left to the dependency management system. From my understanding, the packaging systems maintainers we've discussed this with have expressed that it's not reasonable for CPS or cps-config to be all that intelligent about version resolution and conflict resolution. I want to say @drodri and @ras0219-msft have both expressed this concern in different ways.

For what it's worth, the organization I currently work at does not use version specifiers in our dependency declarations in our pkg-config metadata. We add versions to the pkg-config files, but that's mostly because it's there to fill in. It's mildly useful for certain kinds of troubleshooting activities, but people tend to query the package manager when wondering what's installed and at what version.

[memsharded]

Hi @bretbrownjr

I prefer to use @memsharded for mentions about Conan in Github.

That is correct, and I'd like to go further: that documentation is only the tip of the iceberg. There is way more that what is documented there, and that we keep learning from users. Maybe some non exhaustive list of things that would not be reflected in that docs section:

• Very few projects in the C and C++ ecosystem really follow semver, even if the semver specification has been there for a long time. So I wouldn't expect some guidelines from the CPS specification to be quickly adopted.
• Many projects don't even release or version at all, including popular projects, typically from google or other important players.
• The semver specification that works more or less ok for other languages, is extremely limited, not to say clearly insuficcient to express C and C++ linkage model, in which binaries embed code from headers, templates from dependencies, this is very unique to C and C++.
• The customization level of dependencies in C and C++ is at least one order of magnitude larger than other languages. While you can depend on one dependency in NPM, pip, Maven, and absolutely all users will get exactly the same dependency binary, in C and C++ ,libraries and packages have tons of toggles that enable and disable features, precission, speed, support for ssl and other protocols, bindings in C for C++ libraries or C++ wrappers of C libraries... And users often what to go beyond what the standard libraries offer and add extra flags, code patches, etc.

So while I think versioning is an important problem that would be great to address, I think it can be decoupled from the basic/core CPS functionality, that in my opinion is the specification of a library/package for a build system to be consumed (an the generation of the CPS basic information from build system), which can be made completely independent of versions. In my opinion the basic CPS flow would be:

• One user has project A in his folder A, with build system A. They don't even need to define a version at all. They can build that project and the build will generate a pkga.cps file
• The user has another project B in another folder B with build system B. They need to define 2 things:
  • Tell the build system B: "when you want to use A, you can find its cps in /path/to/my/pkga.cps"
  • I want to use pkga: pkga = load_cps("pkga")
  • Please link my B executable with the pkga target/library/package abstraction

This flow:

• Don't even need to consider versions at all
• Don't have to consider binary configuration resolution or compatible
• Is very "doable". This is close to pkg-config .pc files, but modern, portable, and not using just flags for everything but better abstraction. We know what is needed for this, contrary to other parts like trying to define standard versioning
• The value delivered by this would be huge, this alone already addresses many of the most painful current issues.

[dcbaker]

Very few projects in the C and C++ ecosystem really follow semver, even if the semver specification has been there for a long time. So I wouldn't expect some guidelines from the CPS specification to be quickly adopted.

I'll admit that i exist in a niche environment, but doing low level Linux development everyone more or less does follow semver, or at least close enough. This is the environment that pkg-config was born in, and this is an environment I need to have to support CPS. I agree that there is more than "just sort versions" but "just sort versions" is required to get parody with pkg-config.

In other words, it's a requirement for Meson, and for the project hat I work on as a day job that I can: do a CPS search, find multiple cps, and then filter them against criteria of versions and/or features to determine a suitable candidate. Maybe that means that I have to be the owner for that feature, and that's fine.

for cps-config, I think it's absolutely essential that we provide pkg-config compatible version searching, even with the known limitations thereof. That's just a basic requirement for drop-in compatibility.

The larger question though seems to be on aimed at the CPS spec, because frankly, how a package identifies itself is not a question that cps-config can or should answer, that's a question of how the CPS file itself is generated, and seems to be really a question of "how do we represent a dependency?", and that seems to be the general question that https://github.com/cps-org/cps/issues/43 poses too.

[memsharded]

for cps-config, I think it's absolutely essential that we provide pkg-config compatible version searching, even with the known limitations thereof. That's just a basic requirement for drop-in compatibility.

But that is a functionality of the tool, isn't it? As long as the CPS define some version = ... defined, tools can use that information to operate and decide what they want to do. So this doesn't affect the CPS specification?

A different story is trying to add version information of dependencies within the CPS itself. That can be way more challenging. Adding the version of a dependency could be the dependency that was used to build that specific package, but that really doesn't mean that it cannot use another version of the dependency, there will be many cases with binary compatibility. This is the thing that can be challenging, defining version-ranges, possibly a-posteriori is complicated, even impossible if the cps file is inside a pre-compiled binary that you have already stored and hashed.

The larger question though seems to be on aimed at the CPS spec, because frankly, how a package identifies itself is not a question that cps-config can or should answer, that's a question of how the CPS file itself is generated, and seems to be really a question of "how do we represent a dependency?"

I agree that there is a naming challenge here. We can have the openssl.cps file that needs to represent that it has a dependency to zlib.cps that is somewhere else. The bare minimum that openssl.cps needs to specify is depends: ["zlib"], assuming that the system will be able to find the zlib.cps file somewhere, even if there are no versions involved.

I don't oppose to have extra version information like depends: [{name: zlib, versions: [>=1.0 <2]}], I am just saying that this will be challenging to get it right, to have the right definitions of compatible dependencies. So it will also be more challenging to agree on something to propose for standardizing. While agreeing that we want to have a "includedirs" specification is something that we can do, so I am pushing for a quite "barebones" CPS with the well-known bits, that can grow in the future as necessary.

[dcbaker]

Okay, I think we're on the same page then. I agree that figuring out how to model all of the different ways in which a library (whether C, C++, Fortran, or even Rust) can be configured is going to be difficult.