search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.11k stars 960 forks source link

[question] Building and uploading missing packages changes the recipe revision timestamp #16612

Open alexsmedin opened 2 months ago

alexsmedin commented 2 months ago

What is your question?

Not sure if this is a bug or intentional for some reason.

A conan config change (core.package_id:default_python_mode in global.conf) caused our conan installs to search for different package ids, which were then missing because the recipe revisions that were locked in lockfiles did not have packages for the new config. To address this, we used -b missing in CI and wrote json output to a graph, then ran conan list on the graph and finally an upload on the list, causing the missing packages to be built and uploaded to Artifactory.

So far, everything worked well and projects that locked these revisions could find and install the new packages. However, when updating lockfiles with -u to get the latest revision, we noticed that the rebuilt revisions are now being picked up as the latest. Checking in Artifactory confirms that the recipe revision .timestamp files have changed for the revisions that we uploaded new packages for.

This is a problem because it means that the revision that is actually supposed to be the latest either needs to be rebuilt and reuploaded or the timestamp manually edited to make it the latest revision again. And since the latest revision is already built with the new config before rebuilding the older revision upload of the latest revision is skipped, making it hard to restore the latest revision (though I guess force uploading may help if that also changes the timestamp). Wouldn't it make more sense if only the .timestamp files in the added packages were new and not the one for the recipe revision, since the recipe hasn't changed?

We are using conan 2.4.1 and Artifactory 7.55.10.

Is there some reason I'm not aware of why the recipe timestamp needs to change or is this a bug? Would it make sense with a feature request to add a flag to the conan upload command to upload packages without altering the recipe timestamp in order to simply add additional packages without changing the order of revs?

Have you read the CONTRIBUTING guide?

memsharded commented 2 months ago

Hi @alexsmedin

Thanks for your question.

I have been trying to reproduce, but so far it seems the behavior is correct, and the ordering of revisions in the server side is stable.

This is what I have tried:

So the revisions order in the server side seems stable as expected, uploading new binaries for older revisions do not alter the order of the revisions

alexsmedin commented 2 months ago

Thank you for the quick reply! If that is the case and the expected behavior is that the revision order persists when adding packages, I'll try to reproduce it manually and share some instructions for doing so if I'm able to. Just wanted to know what was expected before investigating it further!

alexsmedin commented 2 months ago

We figured out what went wrong and the problem was not where we expected it, the upload of additional packages does indeed not change the timestamps.

There was a pipeline that used a lockfile and ran conan upload with --force after building packages. Every time that ran, all the revisions in the lockfile became the latest of its requirements. I guess that is to be expected when reuploading, so the error was on our side and we'll remove that flag. However, there could perhaps also be a warning in the conan upload documentation that the force flag changes the revision order. Closing the issue, thanks again for looking into it.

memsharded commented 1 month ago

I am not so sure either that the --force changes that order too. We have a test in the test suite, I have done https://github.com/conan-io/conan/pull/16621, to test it against Artifactory, in case it was behaving differently. The test is passing correctly.

The test is re-exporting an older revision, so it gets a new timestamp, and uploading it with --force. apparently it doesn't change the revision order either.

alexsmedin commented 1 month ago

Hm, maybe it was the combination of adding new packages with --force? I am able to reproduce it with very similar steps to what you described before.

The first conan create made a myapp/0.1#ebe3dfe396fc585b7e4e3a93994cdcd1, and after editing a cpp file the second run created myapp/0.1#c186e0c3d24ece0f27cf34d323b6e990. Also uploaded them both to artifactory after creating, and running list on the local cache and the remote gives the following output.

> conan list "myapp/*#*"
Found 1 pkg/version recipes matching myapp/* in local cache
Local Cache
  myapp
    myapp/0.1
      revisions
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:00:24 UTC)
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:01:36 UTC)
> conan list "myapp/*#*" -r artifactory-local
Found 1 pkg/version recipes matching myapp/* in artifactory-local
artifactory-local
  myapp
    myapp/0.1
      revisions
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:01:22 UTC)
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:02:07 UTC)

After reverting to the original src content and running conan create . -s build_type=Debug, local cache looks like this

> conan list "myapp/*#*"
Found 1 pkg/version recipes matching myapp/* in local cache
Local Cache
  myapp
    myapp/0.1
      revisions
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:01:36 UTC)
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:04:53 UTC)

I then ran conan upload myapp/0.1#ebe3dfe396fc585b7e4e3a93994cdcd1 -r artifactory-local --force (note the force flag) and listed the revisions in the remote again. As you can see the ebe... revision is now the latest.

> conan list "myapp/*#*" -r artifactory-local
Found 1 pkg/version recipes matching myapp/* in artifactory-local
artifactory-local
  myapp
    myapp/0.1
      revisions
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:02:07 UTC)
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:07:13 UTC)
memsharded commented 1 month ago

Thanks for the detail.

UPDATE: ~I have been able to reproduce,~ I have added a test that fails when used against Artifactory in my PR https://github.com/conan-io/conan/pull/16621

~We need to carefully check where would be the discrepancy, if it is a failure in the testing server or in Artifactory, it is not fully clear now what was the expected behavior.~

The test was failing against my local Artifactory because I was reusing the repo and not cleaning it. After cleaning it, the test, that I think it captures your described scenario, is passing in both cases.

memsharded commented 1 month ago

I have updated my comment above, the test seems to be passing against Artifactory too, I was testing it wrong.

memsharded commented 1 month ago

Maybe the Artifactory version is related? I am using 7.77.5, and you are using Artifactory 7.55.10. For some reason I overlooked the version, I think I read the 55 and confused it for 77.5. That version is like 15 months old, I don't recall right now a explicit bugfix for this, but some other changes were done in between, there might be a chance that this got fixed since then.

alexsmedin commented 1 month ago

It is an old self-hosted artifactory indeed. If it's on the server side I can try to get it updated (which it should be anyways) and then try to reproduce it again, but it may take a while to do so.

memsharded commented 1 month ago

You might try to reproduce locally, with an ArtifactoryCE, just to test if there is something else on your project or client side that could be affecting, and at least we can rule out the root cause being in one side or the other.

alexsmedin commented 1 month ago

Ah, right, forgot that CE includes conan support. Started an Artifactory CE container with 7.77.5 (same as you were using), created a conan repo and tried the same thing.

This is the output after creating and uploading two revisions:

> conan list myapp/*#* -r artifactoryce
Found 1 pkg/version recipes matching myapp/* in artifactoryce
artifactoryce
  myapp
    myapp/0.1
      revisions
        7ef91bf548f9b8eed39122c23ed967a5 (2024-07-09 07:01:31 UTC)
        485afec4f8bcfc8ff27d284f445ba11a (2024-07-09 07:02:05 UTC)

And this is after building and force uploading a debug build of 7ef91...:

> conan list myapp/*#* -r artifactoryce
Found 1 pkg/version recipes matching myapp/* in artifactoryce
artifactoryce
  myapp
    myapp/0.1
      revisions
        485afec4f8bcfc8ff27d284f445ba11a (2024-07-09 07:02:05 UTC)
        7ef91bf548f9b8eed39122c23ed967a5 (2024-07-09 07:03:26 UTC)

As you can see I am able to reproduce it with conan 2.4.1 and artifactory 7.77.5.

alexsmedin commented 1 month ago

Was running it in Windows, tried doing the same thing on Ubuntu (wsl) as well while I had a local artifactory up anyways and got the same result.

Before force upload:

$ conan list myapp/0.1#* -r artifactoryce
artifactoryce
  myapp
    myapp/0.1
      revisions
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-09 07:17:35 UTC)
        8051e99400d22ad27e20b832b387e446 (2024-07-09 07:18:41 UTC)

After force uploading additional packages to ebe3d...:

$ conan list myapp/0.1#* -r artifactoryce
artifactoryce
  myapp
    myapp/0.1
      revisions
        8051e99400d22ad27e20b832b387e446 (2024-07-09 07:18:41 UTC)
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-09 07:21:56 UTC)
memsharded commented 1 month ago

I am still trying to reproduce this, I have written this script:

import os

def run(cmd):
    os.system(cmd)

repo_name = "art"
run("conan remove * -c")
run(f"conan remove * -c -r={repo_name}")
run("conan new cmake_lib -d name=myapp -d version=0.1 -f")

# First revision
run("conan create . -tf=")
run(f"conan upload * -r={repo_name} -c")
run(f"conan list *#* && conan list *#* -r={repo_name}")

# mod, second revision
content = open("conanfile.py", "r").read()
new_content = content + "\n# test"
open("conanfile.py", "w").write(new_content)
run("conan create . -tf=")
run(f"conan upload * -r={repo_name} -c")
run(f"conan list *#* && conan list *#* -r={repo_name}")

# Use first revision now, different binary
run("conan new cmake_lib -d name=myapp -d version=0.1 -f")
run("conan create . -s build_type=Debug -tf=")
# Now force it
run(f"conan upload * -r={repo_name} -c --force")
run(f"conan list *#* && conan list *#* -r={repo_name}")

Running this script, I can still see that the revisions are stable, and the last --force does not change it.

  myapp
    myapp/0.1
      revisions
        9b89d84542ef3b0ce48f958ee45f8f98 (2024-07-12 14:41:11 UTC)
        0d217871286073d9ccd6d6d060de28da (2024-07-12 14:41:19 UTC)

Both timestamps are also not changed and matching the first time they were uploaded.

Maybe you can try to run it from your side and see if you see any difference? Just to rule out other possible minor details that we would be missing?

memsharded commented 2 weeks ago

Hi @alexsmedin

Any update here? I have the test https://github.com/conan-io/conan/pull/16621 that might close this ticket, it would be great if you can use the script above to test it.

memsharded commented 1 week ago

Hi @alexsmedin

https://github.com/conan-io/conan/pull/16621 has been merged, I am removing this from 2.7 milestone, if you can please have a look to the script above and try it that would be great.

alexsmedin commented 5 days ago

Hi, I'm sorry for the long delay, I have been away on a long vacation and haven't had time to look at this.

Tested it now and got some interesting results.

When I run the script, the order persists and it looks like this in the remote (9b89 is the original revision before the conanfile was edited) :

  myapp
    myapp/0.1
      revisions
        9b89d84542ef3b0ce48f958ee45f8f98 (2024-08-28 11:04:04 UTC)
        729e10f954dc023f3e76532cd489137b (2024-08-28 11:04:11 UTC)

If I copy-paste the commands and run them one by one in a terminal, the revision order is changed (got a different revision hash because I manually edited the conanfile for the second run, but 9b89 is still supposed to be the oldest).

  myapp
    myapp/0.1
      revisions
        e51c8e0c13880bd62272a2bcc239d542 (2024-08-28 11:00:18 UTC)
        9b89d84542ef3b0ce48f958ee45f8f98 (2024-08-28 11:01:08 UTC)

Tried both running it manually and through the python script a couple times and was able to reproduce this.

memsharded commented 5 days ago

Thanks for the feedback @alexsmedin, no problem about the delays, hope the vacation was good :)

Tried both running it manually and through the script a couple times and was able to reproduce this.

Sorry, I am not fully sure of the result. First you said that running the script works fine and the order persists. Are you able to reproduce the issue just by running the script? Does it happen at every script run, or is it random?

Are you still using the local Artifactory CE container with 7.77.5?

alexsmedin commented 5 days ago

First you said that running the script works fine and the order persists.

Yes, the python script works and the order persists every time I run it. I can't reproduce the issue when running it through python.

The issue is only reproduced when I run the same conan commands as in the script but manually one by one in a bash terminal (manually running conan remove * -c and so on) even though the result should be the same.

I was using a different artifactory version, but now I've started the same container (using the image releases-docker.jfrog.io/jfrog/artifactory-cpp-ce:7.77.5 and tried again using that remote with the same result.

I don't understand why I would get a different result when running the same sequence of commands from a script compared to when running it in bash...

memsharded commented 5 days ago

The issue is only reproduced when I run the same conan commands as in the script but manually one by one in a bash terminal (manually running conan remove * -c and so on) even though the result should be the same.

Yes, this is definitely shocking! 😕 I have no idea why the result would be different when running manually... Is it possible that you have different versions of Conan installed in your machine? And running the Python script will run one version, and running from terminal will prioritize other version (like a system installed one with the installers)?

alexsmedin commented 5 days ago

Found something interesting.

Split out the last two commands into their own script that only does the force upload and lists revisions, removing them from the original script.

run(f"conan upload * -r={repo_name} -c --force")
run(f"conan list *#* && conan list *#* -r={repo_name}")

Ran the first part (everything except the force upload) and it looks like this:

local-container
  myapp
    myapp/0.1
      revisions
        9b89d84542ef3b0ce48f958ee45f8f98 (2024-08-28 11:56:10 UTC)
        729e10f954dc023f3e76532cd489137b (2024-08-28 11:56:12 UTC)

Then the second part (force upload):

local-container
  myapp
    myapp/0.1
      revisions
        9b89d84542ef3b0ce48f958ee45f8f98 (2024-08-28 11:56:10 UTC)
        729e10f954dc023f3e76532cd489137b (2024-08-28 11:56:12 UTC)

But, here's the interesting part, I ran part two (force upload) again, and now I get this:

local-container
  myapp
    myapp/0.1
      revisions
        729e10f954dc023f3e76532cd489137b (2024-08-28 11:56:12 UTC)
        9b89d84542ef3b0ce48f958ee45f8f98 (2024-08-28 11:57:16 UTC)
memsharded commented 5 days ago

This keeps shocking me.

The good news is that I have finally managed to reproduce. Not sure what happens, it is quite random, sometimes it happens sometimes not. It is definitely not immediate, this is why scripts might not detected, it might be related to the server side indexing when some force upload is done. Definitely not deterministic on my side, I need to run the second script a few times to finally reproduce it, usually it works well and keeps the order.

The bad news is that this kind of non deterministic behavior in the server side will be challenging to debug, but at least we got something now. Thanks very much for all your feedback and testing.