search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.05k stars 4.75k forks source link

Add "List all downloaded images for the proxy-cache" and "whether project is working in proxy-cache mode" to REST API #21053

Open dmitriiblium opened 2 weeks ago

dmitriiblium commented 2 weeks ago

Hi, we have a Rancher production cluster, created using Harbor, since it has to be done in the air-gapped environment. Neuvector in the cluster has to check all the images before they are being used in production, to ban images that has known vulnerabilities from being used by the end user. Also we are using Harbor in the proxy-cache mode in order to save space, but still have updated images being added to the registry. But since Harbor doesn't have API calls to check whether image was already downloaded, or whether project is working in proxy-cache mode, Neuvector start checking all the images and download them. It makes it download terabytes of postgres images from all the possible tags. We have an open issue with SUSE and Neuvector to solve this issue, but they cannot solve it from their side, since there are no API calls from Harbor side. Could you please make these calls?

Vad1mo commented 2 weeks ago

Can you elaborate what API you would expect? If you can see it in the UI there is an API ;-D

dmitriiblium commented 2 weeks ago

Good morning, I would expect 2 API calls:

  1. Allowing Neuvector to know if project is proxy-cache or not.
  2. Allowing to know if image is already downloaded in proxy-cache or not. I have wrote about the issues Neuvector having, when it downloads terabytes of images and checking them, because Harbor in proxy-cache says that it have these images
dmitriiblium commented 2 weeks ago

@Vad1mo, are you saying that you have already implemented some of them? SUSE support said that they couldn't find nothing like that

wy65701436 commented 1 week ago
  1. Regarding question 1, you need to check the response from the Project Info API (https://{yourharbor}/api/v2.0/projects/{project_id}) to determine if the registry_id in the returned body has a non-zero value.
  2. Could you please provide us with a detailed end-to-end (e2e) user case?
dmitriiblium commented 1 week ago

Thank you, number 1 works. @wy65701436, Can you give a link on how to do that? I think it could be just another call for the proxy cache project, that will allow to get a list of images already downloaded in the proxy cache, or for each image have a flag if it is already downloaded. I was mentioning before, since Neuvector doesn't know what was downloaded to the proxy cache and what wasn't yet downloaded, it downloads terabytes of old images, then spend weeks to find all of their vulnerabilities. While we have only ~100 images in all the proxy-cache projects, it will take 5 minutes to download and check them all. Neuvector is a tool that helps us with security on the kubernetes cluster, for example it can stop user from creating pods with the image that was not checked, or checked and have too much vulnerabilities. It has to scan every ~10 minutes and it shouldn't go over terabytes of images, otherwise it will jsut stop working.

dmitriiblium commented 1 week ago

Hi @Vad1mo, the issue is still open. Can you please elaborate on what info is needed from our side, or from Neuvector/SUSE side?

dmitriiblium commented 1 week ago

@Vad1mo @stonezdj @reasonerjt @wy65701436 I see it as an issue with connection between Harbor and Neuvector/SUSE, and Rancher users are just a customers of you both. So, Harbor can solve this issue, creating API calls from their side or SUSE/Neuvector will develop something, replacing Harbor for their customers. I think they are already doing it. So, essentially Harbor have motivation to work with SUSE/Neuvector together and set up communication channel with them, not through the customers. Otherwise it will be replaced. We need a final product and next week I will start looking at the alternatives