[question] How to properly "evolve" lockfiles?

What is your question?

Hi All

I recently started looking into conan2 lockfiles and I'm not sure one is supposed to evolve them over time. particularly

Add dependencies
Remove dependencies
Change the pinned version of a given dependency

I have a dummy project where I just have a bunch of requirements using version ranges.

    def requirements(self):
        if self.settings.arch == 'x86_64':
            self.requires('gtest/1.14.0')
        self.requires('boost/[>=1.84 <2.0]')
        self.requires('zstd/[>=1.4 <1.5.6]')
        self.requires('apriltag/[>=3 <4]')
        self.requires('libpng/[>=1.6 <2]')
        self.requires('zlib/[>=1.3.0 <2]')
        self.requires('eigen/[>=3.3.9 <4]')
        self.requires('protobuf/[>=5.27 <6]')

When I lock the project with conan lock create . I get a "complete" lockfile where every package is fully locked (I get a version, a recipe revision and two more numbers %xxxx.yyy which I'm not sure what they mean).

{
    "version": "0.5",
    "requires": [
        "zstd/1.5.5#1f239731dc45147c7fc2f54bfbde73df%1715599909.17",
        "zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024%1705999193.776",
        "protobuf/5.27.0#ccce9aa25886556c6d66c77b2be4d806%1721234376.138",
        "libpng/1.6.43#c219d8f01983bac10c404fc613605eef%1708791038.007",
        "gtest/1.14.0#25e2a474b4d1aecf5ff4f0555dcdf72c%1715706694.24",
        "eigen/3.4.0#2e192482a8acff96fe34766adca2b24c%1715707231.422",
        "bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4%1715709059.861",
        "boost/1.85.0#6ceb5022e53c08e5f6c7bb57ac2f158a%1719853306.554",
        "apriltag/3.1.4#d5ff37d1a9bdb4bda13ed10258bc04bb%1681321594.567",
        "abseil/20240116.2#996c9b7c09f1f561bdf2e2f3c889a8cb%1720072848.278"
    ],
    "build_requires": [
        "b2/5.2.1#91bc73931a0acb655947a81569ed8b80%1719853898.01"
    ],
    "python_requires": [],
    "config_requires": []
}

So far so good. But lets say that I want update just zlib, the docs say I should do conan lock add --requires zlib/1.3.1

But this produces an "incomplete" lockfile, zlib doesn't have a RREV nor the other two numbers after the %.

    "requires": [
        "zstd/1.5.5#1f239731dc45147c7fc2f54bfbde73df%1715599909.17",
        "zlib/1.3.1",
        "zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024%1705999193.776",
        <...>

And in order to complete it I need to conan install . --lockfile-out=conan.lock --lockfile-clean Which takes me back to a sane lock

    "requires": [
        "zstd/1.5.5#1f239731dc45147c7fc2f54bfbde73df%1715599909.17",
        "zlib/1.3.1#f52e03ae3d251dec704634230cd806a2%1715709024.483",
        "protobuf/5.27.0#ccce9aa25886556c6d66c77b2be4d806%1721234376.138",
        <...>

Why is this extra step needed? Furthermore this only works to update dependencies. If I repeat these steps to use an older zlib, it doesn't work:

conan lock add --requires zlib/1.2.13

produces

    "requires": [
        "zstd/1.5.5#1f239731dc45147c7fc2f54bfbde73df%1715599909.17",
        "zlib/1.3.1#f52e03ae3d251dec704634230cd806a2%1715709024.483",
        "zlib/1.2.13",
        "protobuf/5.27.0#ccce9aa25886556c6d66c77b2be4d806%1721234376.138",

But when i do conan install to complete the lockfile (conan install . --lockfile-out=conan.lock --lockfile-clean) I get 1.3.1 again

{
    "version": "0.5",
    "requires": [
        "zstd/1.5.5#1f239731dc45147c7fc2f54bfbde73df%1715599909.17",
        "zlib/1.3.1#f52e03ae3d251dec704634230cd806a2%1715709024.483",
        "protobuf/5.27.0#ccce9aa25886556c6d66c77b2be4d806%1721234376.138",

Similarly, I tried to remove boost from my project so I took it out of my conanfile.py and ran conan lock remove --requires boost/1.85.0 To my surprise all of boost dependencies were left behind on the lockfile. They did got cleaned up after conan install . --lockfile-out=conan.lock --lockfile-clean

{
    "version": "0.5",
    "requires": [
        "zstd/1.5.5#1f239731dc45147c7fc2f54bfbde73df%1715599909.17",
        "zlib/1.3.1#f52e03ae3d251dec704634230cd806a2%1715709024.483",
        "protobuf/5.27.0#ccce9aa25886556c6d66c77b2be4d806%1721234376.138",
        "libpng/1.6.43#c219d8f01983bac10c404fc613605eef%1708791038.007",
        "gtest/1.14.0#25e2a474b4d1aecf5ff4f0555dcdf72c%1715706694.24",
        "eigen/3.4.0#2e192482a8acff96fe34766adca2b24c%1715707231.422",
        "apriltag/3.1.4#d5ff37d1a9bdb4bda13ed10258bc04bb%1681321594.567",
        "abseil/20240116.2#996c9b7c09f1f561bdf2e2f3c889a8cb%1720072848.278"
    ],
    "build_requires": [],
    "python_requires": [],
    "config_requires": []
}

Am I doing something wrong? Or this is the expected workflow? (each conan lock operation should be followed by a conan install --lockfile-out=conan.lock --lockfile-clean)

I'm used to working with poetry lockfiles, so my brain might be wired to do something else. I think that poetry's lockfile management is exceptional. Just in case you're not familiar, these are some example cmds:

poetry add numpy: adds numpy to the pyproject and it plus all of it transitive dependencies to the lockfile.
poetry update numpy: update just poetry (and some of it transitive dependencies, if needed)
poetry remove numpy: removes poetry from pyproject and it plus all if its transitive dependencies from the lockfile
poetry add numpy== locks in a specific version. I think they way poetry does it is very intuitive. Maybe it doens't work on the c++ world.

Have you read the CONTRIBUTING guide?

[ ] I've read the CONTRIBUTING guide

I also tried with conan lock update --requires=zlib/1.3.1m which produces exactly the same behavior. The lockfile ends up incomplete until I "install" it.

Hi @ericriff

Thanks for your question.

In general, yes, this is expected and as designed. I think you might be interested in https://github.com/conan-io/conan/issues/16811 and specially in https://github.com/conan-io/conan/issues/16811#issuecomment-2282906412

The key is that if you want to update a lockfile, it is because there is another thing you want to test. If there is another thing to test, you surely already know its full reference, so you can use it in conan lock operations, providing the full recipe-revision, and then making that extra step unnecessary. Modifying and updating a lockfile with incomplete information is fine, but then Conan will need to complete it when possible. This can be done with different operations, like your conan install, but conan lock create or conan graph info can also complete the lockfile being faster, as it will not need to install/download/build binaries.

Please let me know if this helps.

So the way I'm thinking about Conan lockfiles is: we maintain a conanfile with version ranges and we lock the dep tree using the lockfile. Once our project is mature enough, the conanfile will rarely change and most conan changes will only impact the lockfile. The two scenarios I can think of to update the lockfile is

We realize there is a bug on libA/1.1.1 so we bump it to whatever latest version fits our ranges (lets say the major version doesn't change)
We want a new version of whatever lib to get a newer feature. Again if only a minor or patch version changes, only the lockfile changes.

So changes there are not just for testing. I see the lockfile as the source of true for the actual packages we will be using.

The thing is

If I know the full reference, why use conan lock <> cmds if they're just txt operations on the json? The docs say I shouldn't manually modify them but conan lock <> doesn't see to do anything but that (add might be doing some sorting IIRC).
The extra step seems to be always necessary after conan lock remove <> if we're dropping a dependency.

What's not clear to me is why the conan lock cmds are ok with producing incomplete lockfiles. It seems error prone to me. I work at a rather large company and folks are not familiar with Conan. It is just an implementation detail that "just works" for them. So, the way I see it, I would have to create a script that wrap these conan calls to tweak the lockfile. Otherwise an unaware dev might

Remove a package and leave all of its transitive dependencies on the lockfile
Do conan lock update <> and commit an incomplete lockfile.

Would it be possible to include some sort of stricter behavior on the conan lock commands so they always produce a valid, complete lockfile?

So changes there are not just for testing. I see the lockfile as the source of true for the actual packages we will be using.

Agree with this, lockfile is a source of true for reproducing dependencies.

If I know the full reference, why use conan lock <> cmds if they're just txt operations on the json? The docs say I shouldn't manually modify them but conan lock <> doesn't see to do anything but that (add might be doing some sorting IIRC).

It does modify the lockfile contents, but it does an important operation: it sorts the versions and revisions in chronological order. This sorting is critical for the correct functioning of lockfiles, and this is why it is discouraged to modify the lockfile manually, as it would easily result in inconsistencies.

What's not clear to me is why the conan lock cmds are ok with producing incomplete lockfiles. It seems error prone to me. I work at a rather large company and folks are not familiar with Conan. It is just an implementation detail that "just works" for them.

Because there will be very different users with very different needs. Evolving a lockfile with incomplete information, having partial lockfiles, etc is possible.

As a note about comparing with other technologies lockfiles: how many other lockfiles allow to lock different versions or even different revisions of the same package in the same lockfile? This is completely unexpected and unsupported in many technologies but some C++ users need to express dependency graphs with different versions of their dependencies.

Would it be possible to include some sort of stricter behavior on the conan lock commands so they always produce a valid, complete lockfile?

To be able to implement this, it would be necessary to add the full suite of arguments (--profile, --settings, --options, --build, --update, --remote, ....) to the conan lock add/remove/update command, because in order to have a complete lockfile the dependency graph needs to be resolved. This sounds too much for the conan lock add/remove/update.

The thing is that this wouldn't be necessary in most cases. I think that users shouldn't be doing conan lock add/remove/update in most cases. This would belong mostly to CI, not to devs.

Regarding the "extra" necessary conan install, why would it be an issue? lets say that a developer can compute a complete lockfile with a conan lock add operation. Is that the end? Most likely not, but they will use that lockfile to resolve their graph, build and test their apps, etc. So at the end a conan install will happen anyway?

What are the user stories?

Story 1: A user is creating a new version/revision of some dependency, and then they want to test their app with the new thing. No need to do conan lock add, just pass the lockfile in-and-out the first conan create command of the new version/revision, and it will be automatically added (and complete) Story 2: User wants to test a new version/revision of some dependency, but they didn't create it. They might have the full version+revision or only the version, they decide to do a conan lock update ..., then they will do a conan install to build and test their application.

As a note about comparing with other technologies lockfiles: how many other lockfiles allow to lock different versions or even different revisions of the same package in the same lockfile? This is completely unexpected and unsupported in many technologies but some C++ users need to express dependency graphs with different versions of their dependencies.

I was reading at this and I must say, I'm surprised Conan supports this. I think it is confusing. With other tools when people ask me, which version do we use here for this? I say just look at the lockfile. With conan2 there could be more than one entry for the same dependency.

Unless I find some really nit thing about these lockfiles with multiple versions of the same package, I think I'll personally continue using the v1 style with different locks for different configs.

Maybe I just "don't see it" just yet. Lockfiles are a hairy topic.

To be able to implement this, it would be necessary to add the full suite of arguments (--profile, --settings, --options, --build, --update, --remote, ....) to the conan lock add/remove/update command, because in order to have a complete lockfile the dependency graph needs to be resolved. This sounds too much for the conan lock add/remove/update.

If it is a pain to implement, that's ok.

BTW I'm just trying to find a good, straightforward to the question "Hey, how do I bump / downgrade the version of libX". I think that this is a key feature that's not properly documented here https://docs.conan.io/2/tutorial/versioning/lockfiles.html Particularly because conan lockfiles seem to take a completely different approach at locking dep trees, compared to other tools (at least the ones i've used).

As an example it says

To be clear: manually adding with conan lock add is not necessarily a recommended flow But then it doesn't mention what the recommended workflow actually is.

One more thing, could you please elaborate a bit more on this?

Story 1: A user is creating a new version/revision of some dependency, and then they want to test their app with the new thing. No need to do conan lock add, just pass the lockfile in-and-out the first conan create command of the new version/revision, and it will be automatically added (and complete)

I don't understand the part about conan create with lockfile in-and-out to update a given package. Thanks.

I was reading at this and I must say, I'm surprised Conan supports this. I think it is confusing. With other tools when people ask me, which version do we use here for this? I say just look at the lockfile. With conan2 there could be more than one entry for the same dependency.

Unless I find some really nit thing about these lockfiles with multiple versions of the same package, I think I'll personally continue using the v1 style with different locks for different configs.

It is not different configs having different versions. The same dependency graph can depend on different versions of the same package. This is not a Conan 2 new supported case, this was already supported in Conan 1 and it was the reason that lockfiles were way more complex in Conan 1, because they had to represent the full graph in the lockfile too, and Conan 1 lockfiles could also have more than 1 different version of the same package in the same lockfile.

It is not confusing, it is a real world scenario that Conan must support, as users need it. We of course prefer and recommend to only have 1 version of the same package in the dependency graph, but there are users that simply cannot do it because they have other constraints. As a recent example, we introduced the capacity of openssl depending on a previous version of itself to be able to provide FIPS compliance. this is how openssl itself says it should be done. This case will need the lockfiles to support multiple versions of the same package in the same lockfile, for example.

I don't understand the part about conan create with lockfile in-and-out to update a given package. Thanks.

This is the example in https://github.com/conan-io/conan/issues/16811#issuecomment-2282906412

Using something like conan create . --lockfile=app.lock --lockfile-out=change.lock (lockfile in app.lock, lockfile-out change.lock. The final change.lock will contain something like:

app/1.0#revapp1
lib_a/2.3#revliba2
lib_a/2.3#revliba1

No need for a explicit conan lock add/update/remove operation, the same flow of creating the new version can update the lockfile that can be later used to test app with this specific change, which is the original intention of the modification of the lockfile, isn't it?

Hi,

just came across this issue as I'm also working with lockfiles and have one particular question which is also in the first post from @ericriff. Hope its fine to add it here:

The initial lockfile contains timestamps as shown in the examples above (everything after the %). Is it fine that all modifications done via the conan lock command are missing those timestamps? I guess so, just wanted to ask why this is the case if they are required for ordering.

I still don't get it. Is there a place where I can read about lockfiles with multiple versions of the same package? I didn't find anything on the docs.

Furthermore, I'm not sure what conan create . --lockfile=app.lock --lockfile-out=change.lock does. Based on your example, if my app has this conanfile

    self.requires('lib_a/[<=2.3 < 2.5]')

My lockfile would look something like

lib_a/2.3#revliba1

Why will that conan create cmd compute a different lockfile than the one you passed with --lockfile? And how do you control which package is update? In this case there is only one, lib_a, but that's not a realistic scenario.

If my conanfile looks like this

    def requirements(self):
        self.requires('boost/[>=1.84 <2.0]', transitive_headers=True)
        self.requires('zstd/[>=1.4 <1.5.6]')
        self.requires('eigen/[>=3.3.9 <4]', transitive_headers=True)
        self.requires('protobuf/[>=5.27 <6]')
        self.requires('lib_a/[<=2.3 < 2.5]')

How do I make it update only lib_a? Does your conan cmd assumes the rest of the packages on the lockfile were already pointing to latest?

The initial lockfile contains timestamps as shown in the examples above (everything after the %). Is it fine that all modifications done via the conan lock command are missing those timestamps? I guess so, just wanted to ask why this is the case if they are required for ordering.

It is true that the conan lock add command only works for adding new versions or by adding the full revision including timestamps, if the lockfile already contains another revision for the same pkg-version. You can try to do this, and then conan lock add will raise an error like

            if existing and existing.revision is not None:
                raise ConanException(f"Cannot add {ref} to lockfile, already exists")``

The full revision with timestamp can be displayed for example with the conan list --format=compact format.

But when i do conan install to complete the lockfile (conan install . --lockfile-out=conan.lock --lockfile-clean) I get 1.3.1 again

I was seeing this, too, but was able to resolve it by doing a conan lock remove of the other version before running conan install.

The initial lockfile contains timestamps as shown in the examples above (everything after the %). Is it fine that all modifications done via the conan lock command are missing those timestamps? I guess so, just wanted to ask why this is the case if they are required for ordering.

It is true that the conan lock add command only works for adding new versions or by adding the full revision including timestamps, if the lockfile already contains another revision for the same pkg-version. You can try to do this, and then conan lock add will raise an error like
            if existing and existing.revision is not None:
                raise ConanException(f"Cannot add {ref} to lockfile, already exists")``
The full revision with timestamp can be displayed for example with the conan list --format=compact format.

Is there any shortcoming if I don't add the timestamp, just the version and rref of the project? I'm only using one entry per project in my lockfile if that is important. Not e.g. multiple versions of the same as described above.

Is there a special reason why I need to use conan list to get the timestamp? Inside the JSON graph coming out of conan create I can see that the timestamps are not set:

"rrev_timestamp": null,
"prev_timestamp": null,

I still don't get it. Is there a place where I can read about lockfiles with multiple versions of the same package? I didn't find anything on the docs.

In the docs https://docs.conan.io/2/tutorial/versioning/lockfiles.html#evolving-lockfiles, the code is already showing how a lockfile can represent more than 1 version for the same package.

But this is not something exclusive from the lockfile, it is a property of the dependency graph.

Consider this:

conan lock create --requires="opencv/[*]"

The created conan.lock will contain:

"build_requires": [
        ...
        "strawberryperl/5.32.1.1#8f83d05a60363a422f9033e52d106b47%1721821035.998",
        "strawberryperl/5.30.0.1#d125df083747d815c66e9ee621f3909f%1721821035.934",
       ...
        "meson/1.2.2#04bdfb85d665c82b08a3510aee3ffd19%1721821034.057",
        "meson/1.2.1#f641f02771e4660c772354736da0b9c6%1721821034.01",
        ...
    ],

Why will that conan create cmd compute a different lockfile than the one you passed with --lockfile? And how do you control which package is update? In this case there is only one, lib_a, but that's not a realistic scenario.

conan create creates a new entity, maybe a new version, or a new revision. While all the dependencies of that conan create will be locked, taken from the provided lockfile, the conan create can add the new created version/revision to that lockfile if using --lockfile-out=new.lock

Full case:

app1.0 -> liba1.0 -> libb1.0
app1.0 contains a requires = "liba/[>=1.0 <2]".
liba1.0 contains a requires = "libb/[>=1.0 <2]"

app1.0 creates a app1.lock lockfile:

app/1.0#revapp1
liba/1.0#revliba1
libb/1.0#revlibb1

Now someone creates a new liba/1.1, with the intention to integrate it in app1.0. Running conan create . --lockfile=app1.lock --lockfile-out=change.lock will result in a lockfile with:

app/1.0#revapp1
liba/1.1#revliba2
liba/1.0#revliba1
libb/1.0#revlibb1

Note the 2 versions of liba in the same lockfile. Now this lockfile can be apply to build/check app1, something for example like:

conan install --requires=app/1.0#revapp1 --lockfile=change.lock --build=missing --lockfile-out=app2.lock --lockfile-clean

It is important how the 2 different versions of liba in the same lockfile play their role:

If liba/1.1 is in the version range (this example) of the requires of app, then the liba/1.1 will be used. app/1.0 might need to build a new binary with a new package_id to account for the new dependency version. The --lockfile-clean will remove the now unused liba/1.0 version.
If liba/1.1 would be outside of the version range (hypothetical, not this example, if the range was [>=1 <1.1]) of app, then app would still resolve to the previously locked liba/1.0#revliba1 (even if there were other versions that would be in the range like liba/1.0.1. Then, this will be a no-op, nothing will build. The --lockfile-clean will remove the liba/1.1 version.

This is the approach that can produce fully deterministic parallel builds for concurrent changes to liba by different developers.

conan-io / conan

[question] How to properly "evolve" lockfiles? #16819

What is your question?

Have you read the CONTRIBUTING guide?