Closed ericriff closed 8 months ago
I was about to open an issue as well; I just had to go save our Artifactory server from running out its datastore (some of our packages are big). We also run nightlies and the RREV/PREV combo gives us an explosion of old packages.
What I'd like to have is a conan purge
command, which would have the ability to:
conan search
Example use cases:
$ conan purge -r conan-local "*@datalogics/nightly" --recipes
$ conan purge -r conan-local "*@datalogics/nightly" --packages
$ conan purge -r conan-local "bigpackage/1.0.0@datalogics/nightly" --packages
$ conan purge -r conan-local "bigpackage/1.0.0@datalogics/nightly#<some RREV>" --packages
$ conan purge -r conan-local "*" --recipes # really clean up everything
That's exactly what I'm looking for.
This might take a while to consider, as we will probably be discussing to incorporate this into the command line for Conan 2.0. In the meantime, you can get the revisions information with:
$ conan search zlib/1.2.11@ --revision -r=conan-center
f3b17fb16cd1b7fbc969400a2ba1bc08 (2020-04-16 10:21:36 UTC)
ddccdddea098293f5202c5e8eb29967b (2020-03-13 17:07:40 UTC)
...
bf8cd01f8b018a3c0871562ae5251bff (2019-09-10 07:54:59 UTC)
c42d82150723a4bb1baca4757fb939ed (2019-08-30 13:32:07 UTC)
And for packages:
conan search zlib/1.2.11#f3b17fb16cd1b7fbc969400a2ba1bc08:ff82a1e70ba8430648a79986385b20a3648f8c19 -r=conan-center --revisions
Revisions for 'None' at remote 'conan-center':
69e8d72ae2418f7626e21529b3d00bd3 (2020-04-16 10:21:45 UTC)
The --json=file.json
might also help to automate this together with conan remove
calls. So it seems that a couple of python scripts for the admins of the Artifactory repository should be doable without big problems, so you can start doing that maintenance already.
Just in case, possibly a good practice is to use more than one Artifactory repository. We are recommending it for CI, for example, where most of the builds are uploaded to a temporary "tmp" Artifactory repo. The purge policy there could be automated for example with an Artifactory plugin that removes artifacts older than a given time, automatically.
Also some notes for ideas for implementations and our future selves:
ETA: Thanks for taking this under consideration.
that removes artifacts older than a given time, automatically
That sounds great, but actually wouldn't work for us. If the nightlies are failing for a few days, we need to be able to have our devs get packages (we have things that are way too big to build in the cache). If we set the timeout to a week or two, we run out of space. Also, I've been getting the strong vibe that Conan package management should be done from the client.
In other words, we need the ability to delete things because they were superseded, not just because they're old.
It might be a bit dangerous to hardcode purge policies based only on just the "latest" revision available. That might break lots of build-infos or lockfiles using previous revisions.
conan purge
would be deliberate. If it doesn't fit your workflow...don't do it. conan remove
is dangerous too; if it doesn't fit your workflow, don't do it.
I could do a lot with a more comprehensive search or a search that could cascade. And more qualifiers on purging would be ok as well, number of revs to keep, purge by date but keep N revs, etc.
It might be that the best approach is to do a lockfile or buildinfo purge behavior.
What would that entail? So far, we don't really use lockfiles...too many different platforms and configurations of things. I suppose our use of Conan is not like most open source projects.
Closing as solved in v2. Patterns for conan remove
(And most other commands!) accept the latest/!latest
keywords, so that:
conan remove */*#!latest:*
would remove every binary of each revision but the latest for each version of each library
Hello, I have a question regarding the management of package revisions on my Artifactory server. The way we're using conan consists on a Jenkins pipeline that builds a bunch of recipes and uploads them to artifactory. We have revisions activated because we're using the recipe revisions as locks for the dependencies, which works great for us. The issue that I'm facing is that my Artifactory server is getting polluted with old packages, because sometimes the package revision change, resulting on a new package being uploaded. Is there an tool to remove old package revisions? Like the
--outdated
flag onconan search
? Or a way to tellconan upload
to override the package if theRREV
is the same?Thanks!