Closed ArcticSnowman closed 4 years ago
Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Description
While setting up and using the garbage collection, I notice that while you can set a inclusion tag filter, there is no way to easily exclude a set of tags/images.
I think we need a way to exclude images or tags from the GC process.
Hi @ArcticSnowman Idid not say anything yet, because I had nothing to say. Today, I think you might be interested in reading this : https://github.com/SUSE/Portus/issues/2281#issuecomment-592591559
I support your request, agree with it, and want to add :
PORTUS_DELETE_EXCLUSION_FILTER="phab-*/*hashicorp*:1.0.*"
PORTUS_DELETE_EXCLUSION_FILTER_FILE=/path/to/file/gc-exclusion.filter
notary
, so service netname / netID / service ID + namespace + image name + tag) : one image ID (GUN) per line in a file which path is specified with PORTUS_DELETE_EXCLUSION_FILTER_FILE=/path/to/file/gc-exclusion.filter
PORTUS_DELETE_EXCLUSION_FILTER_FILE
may have , per line either a full image ID (GUN), considered as a special case of regular expression, or any regular expression that geenrates any sub dictionnary of the "GUN
language", like "phab-*/*hashicorp*:1.0.*"
. Though I'm a pacifist. @ArcticSnowman If I may, and If it is okay that I ask, related to https://github.com/SUSE/Portus/issues/807#issuecomment-511461401 :
And btw, the content trust / The Update Framework issue is thrilling.
@Jean-Baptiste-Lasselle - When we first built out our internal docker registry, 3+ years ago, Portus was the best out there at the time.
We did look at habor about a year ago, mainly to look at the inclusion of Clair. We did not feel it was any better that Portus at the time and it pulled in lots of new tech that, in an enterprise environment, we did not want to support. And then we found support had been added to Portus.
I made reference to Harbor in #807, mainly to show that other registry tools have the support.
@Jean-Baptiste-Lasselle - When we first built out our internal docker registry, 3+ years ago, Portus was the best out there at the time.
We did look at habor about a year ago, mainly to look at the inclusion of Clair. We did not feel it was any better that Portus at the time and it pulled in lots of new tech that, in an enterprise environment, we did not want to support. And then we found support had been added to Portus.
I made reference to Harbor in #807, mainly to show that other registry tools have the support.
HI @ArcticSnowman thank you so much for your answer, and sorry I did not answer sooner : I think what we are discussing here, is probably the most exciting challenge with Portus
, and probably most important in the container world, though it looks like we are lost in the comments of a forgotten project. I will answer you on #807 regarding notary
. About Clair scanner, I could set it up,and get actual vulnerabilities identified, displayed in Portus, with CVE-MITRE actual references. See https://github.com/SUSE/Portus/issues/2241#issuecomment-593370789
I am also working on releasing a complete fully tested automation with docker compose, to finally provide people a secured, coherent working first setup of a private docker registry / Portus.
Will happen in the next weeks, It's just that I got involved into a project aside, So I work a little bit everyday, but it takes time.
tty on #807
Hi again @ArcticSnowman , After a good night of sleep, I today have to add something about this issue ( #807 is another subject) :
you need a way to exclude tags from Garbage collection
I join to you for this feature request : because it is logical, it makes sense,
and moreover, what you describe is a feature that extends an existing one, which you probably will be interested in : the keep_latest
feature.
you will find a complete description and study of the keep_latest
feature in this issue : https://github.com/SUSE/Portus/issues/2241
and in that issue, I want to point especially this comment and answer to it by @robgiovanardi :
background
.keep_latest
, all you have to do, is to delete those tags from the Database (S3 storage), and then restart portus and its background, with SYNC
. because what SYNC will do, is progressively update what is displayed in portus WebUI, until it is in sync with the riegsitry : so until all deleted images no longer appear in Portus WebUI. You will read here that @diranged did that to delete more than 80 terabytes of docker images. The only problem he had, is to configure properly the SYNC feature.keep_latest
feature is enough.The most interesting thing that happened, is that thanks to @diranged noticing that, we proved that no opensuse
docker image contains a version of portus
including that keep_latest
feature. Except one image , the opensus/portus:2.5
on dockerhub :
keep_latest
feature.But what you probably want is to try keep_latest
, so here you go :
Debian
, underneath your Portus, I kind of repaired the official opensuse/portus:2.5
image : https://github.com/pokusio/opensuzie-oci-library/releases/tag/0.0.2 . Worth noting about this image, is that portus is installed inside of it, using opensuse package manager, zypper
: It's like they apt-get instal portus
inside the container, to install portus
inside of it. Important Update for you : Ok, it seems like either the OpenSuSE Team either did notice what we proved with our community friends, and now I am wondering if it's war they want, because : zypper
, same as apt-get
or apk
for ex.)docker-containers
, and guess what I just today discovered ? Well they updated their opensuse package, and now it is all different, my docker images crashes badly, and it's all a messportus
.Yep, they really pieced me off. At least, they could communicate, I have posted numerous help comments to numerous users, and solved issues, and now they update a package without any communication/message ? Explanation is quite clear, they really don't give a s***t about us end users, and that's why people have months after months, let this project to die. But I won't. We will have our portus
, entirely free for the world, whether they like or not. Too late, mates.
And btw, Thank you so much : it is because of your message, that I today went "oh okay, I'll just launch the last successful release of my recipe to say hello to portus
".
*
@ArcticSnowman I just want to share the big friendly laugh I had at home, when I discovered that you work tax collection, it is very comforting to know they they rely on us, and maybe they 'll have to remember that someday :rofl:
Hi @Jean-Baptiste-Lasselle - We have tried the keep_latest
, however we found a problem with it. The way our pipeline creates image we generate a fair number of tags for every checked in change. Some of those changes might never get all the way to production because of testing failures. This can easily push the current production release tag off the end of the keep_latest
.
I feel that having an exclude_tags
regexp would compliment that keep_latest
. Allow us to preserve specifically names tags while allowing the GC to remove all the 'extras' that we generated during development and testing.
Hi @Jean-Baptiste-Lasselle - We have tried the
keep_latest
, however we found a problem with it. The way our pipeline creates image we generate a fair number of tags for every checked in change. Some of those changes might never get all the way to production because of testing failures. This can easily push the current production release tag off the end of thekeep_latest
.I feel that having an
exclude_tags
regexp would compliment thatkeep_latest
. Allow us to preserve specifically names tags while allowing the GC to remove all the 'extras' that we generated during development and testing.
Hi @ArcticSnowman , how very interesting case indeed. Just delivering you my thoughts as I write, I actually never considered the case, and how about you use several docker registry, one for integration tests, one for staging, one for production etc... ? Apart from pure curiosity, I am asking because there might be very interesting reason, like business or regulations constraints which prevent you from using that pattern.
Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Description
While setting up and using the garbage collection, I notice that while you can set a inclusion tag filter, there is no way to easily exclude a set of tags/images.
I think we need a way to exclude images or tags from the GC process.