[feature] Portus Garbage Collection (tag removal) should support an exclusion filter

[ArcticSnowman]

Description

While setting up and using the garbage collection, I notice that while you can set a inclusion tag filter, there is no way to easily exclude a set of tags/images.

I think we need a way to exclude images or tags from the GC process.

[stale[bot]]

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Jean-Baptiste-Lasselle]

Description

While setting up and using the garbage collection, I notice that while you can set a inclusion tag filter, there is no way to easily exclude a set of tags/images.

I think we need a way to exclude images or tags from the GC process.

Hi @ArcticSnowman Idid not say anything yet, because I had nothing to say. Today, I think you might be interested in reading this : https://github.com/SUSE/Portus/issues/2281#issuecomment-592591559

I support your request, agree with it, and want to add :

• that this exclusion filter, should enable the user to define the set of exclusion using :
  • a single regular expressions : directly specified as value of a config option, like PORTUS_DELETE_EXCLUSION_FILTER="phab-*/*hashicorp*:1.0.*"
  • a list of regular expressions : one regular expression per line in a file which path is specified with PORTUS_DELETE_EXCLUSION_FILTER_FILE=/path/to/file/gc-exclusion.filter
  • explicit arbitrary list of image IDs (GUN like for notary, so service netname / netID / service ID + namespace + image name + tag) : one image ID (GUN) per line in a file which path is specified with PORTUS_DELETE_EXCLUSION_FILTER_FILE=/path/to/file/gc-exclusion.filter
  • So all in all PORTUS_DELETE_EXCLUSION_FILTER_FILE may have , per line either a full image ID (GUN), considered as a special case of regular expression, or any regular expression that geenrates any sub dictionnary of the "GUN language", like "phab-*/*hashicorp*:1.0.*". Though I'm a pacifist.

[Jean-Baptiste-Lasselle]

@ArcticSnowman If I may, and If it is okay that I ask, related to https://github.com/SUSE/Portus/issues/807#issuecomment-511461401 :

• I have my reasons, for not focusing only on harbor , starting with I think it is very important that harbor is not left alone on the market, to rule us all
• But can I ask your reasons for being willing to use Portus, instead of harbor, though it is obvious you know about harbor ?

And btw, the content trust / The Update Framework issue is thrilling.

[ArcticSnowman]

@Jean-Baptiste-Lasselle - When we first built out our internal docker registry, 3+ years ago, Portus was the best out there at the time.

We did look at habor about a year ago, mainly to look at the inclusion of Clair. We did not feel it was any better that Portus at the time and it pulled in lots of new tech that, in an enterprise environment, we did not want to support. And then we found support had been added to Portus.

I made reference to Harbor in #807, mainly to show that other registry tools have the support.

[Jean-Baptiste-Lasselle]

@Jean-Baptiste-Lasselle - When we first built out our internal docker registry, 3+ years ago, Portus was the best out there at the time.

We did look at habor about a year ago, mainly to look at the inclusion of Clair. We did not feel it was any better that Portus at the time and it pulled in lots of new tech that, in an enterprise environment, we did not want to support. And then we found support had been added to Portus.

I made reference to Harbor in #807, mainly to show that other registry tools have the support.

HI @ArcticSnowman thank you so much for your answer, and sorry I did not answer sooner : I think what we are discussing here, is probably the most exciting challenge with Portus, and probably most important in the container world, though it looks like we are lost in the comments of a forgotten project. I will answer you on #807 regarding notary. About Clair scanner, I could set it up,and get actual vulnerabilities identified, displayed in Portus, with CVE-MITRE actual references. See https://github.com/SUSE/Portus/issues/2241#issuecomment-593370789

I am also working on releasing a complete fully tested automation with docker compose, to finally provide people a secured, coherent working first setup of a private docker registry / Portus.

Will happen in the next weeks, It's just that I got involved into a project aside, So I work a little bit everyday, but it takes time.

tty on #807

[Jean-Baptiste-Lasselle]

Hi again @ArcticSnowman , After a good night of sleep, I today have to add something about this issue ( #807 is another subject) :

• you need a way to exclude tags from Garbage collection

• I join to you for this feature request : because it is logical, it makes sense,

• and moreover, what you describe is a feature that extends an existing one, which you probably will be interested in : the keep_latest feature.

• you will find a complete description and study of the keep_latest feature in this issue : https://github.com/SUSE/Portus/issues/2241

• and in that issue, I want to point especially this comment and answer to it by @robgiovanardi :

  • SYNC feature, is closely related to garbage collection, and both are carried out by background.
  • Say your registry stores its images into a minio database for example (or any S3 storage) : to delete any set of tags, without using keep_latest, all you have to do, is to delete those tags from the Database (S3 storage), and then restart portus and its background, with SYNC. because what SYNC will do, is progressively update what is displayed in portus WebUI, until it is in sync with the riegsitry : so until all deleted images no longer appear in Portus WebUI. You will read here that @diranged did that to delete more than 80 terabytes of docker images. The only problem he had, is to configure properly the SYNC feature.
  • and for all other use cases, keep_latest feature is enough.

• The most interesting thing that happened, is that thanks to @diranged noticing that, we proved that no opensuse docker image contains a version of portus including that keep_latest feature. Except one image , the opensus/portus:2.5 on dockerhub :

  • the problem is that this image is really a mess, so I had to completely rebuild it, to finally be able to use the keep_latest feature.
  • There are even funnier details about that little story, and a complete analysis proving how and why that happened in the opensuse team, here : https://github.com/SUSE/Portus/issues/2241#issuecomment-586972528

• But what you probably want is to try keep_latest, so here you go :

  • With a debian based complete rebuild from Portus Source code Docker image I recently released : https://github.com/pokusio/portus-build-from-source/releases/0.0.1
  • And if you care about having opensuse instead of Debian, underneath your Portus, I kind of repaired the official opensuse/portus:2.5 image : https://github.com/pokusio/opensuzie-oci-library/releases/tag/0.0.2 . Worth noting about this image, is that portus is installed inside of it, using opensuse package manager, zypper : It's like they apt-get instal portus inside the container, to install portus inside of it. Important Update for you : Ok, it seems like either the OpenSuSE Team either did notice what we proved with our community friends, and now I am wondering if it's war they want, because :
  • In the fixing opensuse image process, we could determine thanks to @Josua-SR , that what they do at SUSE, is that they " install everything with package manager (zypper, same as apt-getor apk for ex.)
  • And they don't just install portus that way, but the whole Ruby stack. And I recently found a bug in their installation, a pretty messy one.
  • I reported the bug : https://github.com/openSUSE/docker-containers/issues/114#issuecomment-596765602 and @Josua-SR was as surprised as me, when he realized I got my hands on a point : https://github.com/SUSE/Portus/issues/2279#issuecomment-593705544
  • No one ever answered my opened issue on docker-containers, and guess what I just today discovered ? Well they updated their opensuse package, and now it is all different, my docker images crashes badly, and it's all a mess
  • So it seems that people in the suse team eihter :
  • They don't understand that people using a fixed version number of a docker container, expect it to be identical, at two different dates.
  • Or they don't give a s***t about portus user (most probable explanation I believe).
  • All in all, I have now more arguments to explain why I made a build from Portus source code, and that's why I strongly recommend to not use https://github.com/pokusio/opensuzie-oci-library/releases/tag/0.0.2 or any opensuse based Portus image but instead use https://github.com/pokusio/portus-build-from-source/releases/0.0.1 which does not depend on any OpenSUSE package. And which you can use without fearing me someday updating something and breaking your portus.

Yep, they really pieced me off. At least, they could communicate, I have posted numerous help comments to numerous users, and solved issues, and now they update a package without any communication/message ? Explanation is quite clear, they really don't give a s***t about us end users, and that's why people have months after months, let this project to die. But I won't. We will have our portus, entirely free for the world, whether they like or not. Too late, mates.

And btw, Thank you so much : it is because of your message, that I today went "oh okay, I'll just launch the last successful release of my recipe to say hello to portus".

* 

[Jean-Baptiste-Lasselle]

@ArcticSnowman I just want to share the big friendly laugh I had at home, when I discovered that you work tax collection, it is very comforting to know they they rely on us, and maybe they 'll have to remember that someday :rofl:

[ArcticSnowman]

Hi @Jean-Baptiste-Lasselle - We have tried the keep_latest, however we found a problem with it. The way our pipeline creates image we generate a fair number of tags for every checked in change. Some of those changes might never get all the way to production because of testing failures. This can easily push the current production release tag off the end of the keep_latest.

I feel that having an exclude_tags regexp would compliment that keep_latest. Allow us to preserve specifically names tags while allowing the GC to remove all the 'extras' that we generated during development and testing.

[Jean-Baptiste-Lasselle]

Hi @Jean-Baptiste-Lasselle - We have tried the keep_latest, however we found a problem with it. The way our pipeline creates image we generate a fair number of tags for every checked in change. Some of those changes might never get all the way to production because of testing failures. This can easily push the current production release tag off the end of the keep_latest.

I feel that having an exclude_tags regexp would compliment that keep_latest. Allow us to preserve specifically names tags while allowing the GC to remove all the 'extras' that we generated during development and testing.

Hi @ArcticSnowman , how very interesting case indeed. Just delivering you my thoughts as I write, I actually never considered the case, and how about you use several docker registry, one for integration tests, one for staging, one for production etc... ? Apart from pure curiosity, I am asking because there might be very interesting reason, like business or regulations constraints which prevent you from using that pattern.

[stale[bot]]

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.