Open toastbrotch opened 2 years ago
missing visibility:
there is no global/project dashboard about vulnerabilities findings?
_No, the vulnerabilities findings are associated with one signle artifact. What kind of user scenario if there is an dashborad to show all findings of an project?_
its not possible to create/schedule/mail/export/... any vulnerabilities report?
_Monitor this proposal: https://github.com/goharbor/community/pull/174_
there is no webhook/mail/... that is triggered when new vulnerabilities (of a specified level) are found?
_Harbor can notify CVE results summary by webhook on scan complete, then can call Harbor API to get the details. What do you mean by `new vulnerabilities`? Compare with which version? and what kind of user scenairo if we provide it?_
there is no integration into kubernetes/openshift that would add the missing visibility there (like the quay security operator tries to do (but fails))?
_Can you clarify the query by giving more details?_
broken security concept of "Prevent vulnerable images from running" :
images that are not scanned (whatever reason "Unsupported" has) are not blocked from pulling like those that were scanned. if i just want "secure" images (no vulnerabilities above a certain level), an unscanned image is simply not secure and should not be able to pull!
when used as proxy-cache i get the image before its scanned and possibly not allowed to pull.
_These are by designed, cc @xaleeks_
there is no global/project dashboard about vulnerabilities findings?
No, the vulnerabilities findings are associated with one signle artifact. What kind of user scenario if there is an dashborad to show all findings of an project?
its not possible to create/schedule/mail/export/... any vulnerabilities report?
Monitor this proposal: https://github.com/goharbor/community/pull/174
ok thanx. i've missed this while searching.
there is no webhook/mail/... that is triggered when new vulnerabilities (of a specified level) are found?
Harbor can notify CVE results summary by webhook on scan complete, then can call Harbor API to get the details. What do you mean by
new vulnerabilities
? Compare with which version? and what kind of user scenairo if we provide it?
i mean: if new CVE arise it works like this: trivy db-update, scheduled re-scan, new finding in existing image found. and now there should be some actions triggered so that responsible people get informed and can go in to research/resolve this issue, as this image pontentialy runs somewhere.
there is no integration into kubernetes/openshift that would add the missing visibility there (like the quay security operator tries to do (but fails))?
Can you clarify the query by giving more details?
this is just another layer of visibility that you see image-scan results where it runs. you might work there more often than in the registry. i just tried to find out how i get informed that something is insecure, and all the possiblities i know are not existing or not helping me. see demo of this here https://www.youtube.com/watch?v=5iiKMpBszZ0 . i opened also https://github.com/quay/container-security-operator/issues/60 .
after all i think there are 2 possiblities if you can not get actively informed: passive informed with dashboard in harbor or dashboard where you run your images... but best would be of course if i get actively informed.
broken security concept of "Prevent vulnerable images from running" :
These are by designed, cc @xaleeks
@wy65701436 you mean its designed insecure?
Please disable the "prevent vulnerable" for the proxy-cache project, otherwise you can't pull images from the project.
Please disable the "prevent vulnerable" for the proxy-cache project, otherwise you can't pull images from the project.
@heww : i can. and i get the image the first time before it is even scanned (and i get it if scan shows "unsupported" (whatever the reason is) even afterwards) which is the issues i report...
Prevent vulnerable images from running
in the context of Harbor literally means "Prevent the images that are scanned and reported CVEs from pulling and running". So for those images that are not scanned is ok to be pulled by definition and by design.
@zyyw : security features that mean something which is not obvious are not really a good choice!
a not scanned image is not free from vulnerabilities! we just don't know. in fact it must be handled like an image that has vulnerabilities, as this might be actually the case. everything else is simply not secure design!
this is critical, as "Prevent vulnerable images from running" acts as a security safeguard. but in current state it does not prevent you from getting images with vulnerabilities. the opposite is the case: this option misleads you in believing all images are "secure"! and as described you get insecure images. so this is very dangerous and bad.
i propose to fix "Prevent vulnerable images from running" to be a real security feature:
Agree with @toastbrotch . For example, the scenario is to build an external harbor to first control the image that can be used by the internal harbor. At this time, the external harbor needs to be able to complete the vulnerability scan first, in the meanwhile, prevent the images which are not fully scanned from pulling by internal one which set external as proxy cache.
I also agree with @toastbrotch - this is an issue for our organisation also, and I find this design decision highly problematic.
We had a case just recently where our jobservice container had failed, and so new images were not getting scanned. In this case we would expect the system to "fail safe". As it was, we had vulnerable images being allowed, and even if the jobservice had been running correctly there is still a window where vulnerable images can be allowed though until they are scanned. You would never use an AntiVirus product that scanned unknown executables after they have already run.
Harbor should definitely be taking the cautious approach, treat unscanned images as 'potentially vulnerable' and disallow pulls on unscanned images if "prevent vulnerable images from running" is enabled. I can't imagine any scenarios where the current behaviour would be desired or expected.
@zyyw, @wy65701436, @stonezdj, @heww : i don't understand why those important security-related topics here do not lead to any action at all? as far as i see all that happened was talking around it....
i have the feeling this issue and it consequences is not fully understand by you/the maintainers. which brings me to the question if there are even more bad security-related decisions made in harbor? and therefore i think everone should re-consider if using harbor as a security-defense-measure is a good choice... i would turn off security scanning at all and instead use stackrocks/ACS directly on my clusters. or in my case: i don't even migrate to harbor and stay with my dumb registry.
For the policy on Unsupported artifact, we have to make it consistent with chart files or other unsupported format but OCI compatible files.
The current scanner don't support scan chart files -- no matter trivy, clair or others -- if change the behavior to block the unsupported artifact, the chart files cannot be pulled anymore.
Hi, @toastbrotch, let me clarify the reply from @zyyw, his reply about the Prevent vulnerable images from running
is wrong.
Prevent vulnerable images from running
will block the image to be pulled when it's supported by the scanner but it isn't be scanned.
Because there is no scanner that supports scanning the helm chart OCI artifact, harbor allows the unsupported OCI artifact can be pulled even there is a prevent vulnerable policy in the project.
For the proxy cache project with preventing vulnerable policy, the harbor will save the artifact in the harbor side only users pulled the whole artifact through the proxy cache project. The first time users pull the artifact from the proxy cache project, the whole artifact is not in the harbor storage, the scanner can't scan the artifact at all. The policy can't affect the artifact which is not in the harbor. After the artifact is saved in the harbor, users can't pull it when it's not scanned or the severity is higher or equal with the severity in the policy.
@heww yes i know, as i found this in my tests and thats why i opened this issue, as this is a insecure fallback. and thats why this policy is no valid security defense, and therefore inherently insecure and bad designed!
as written many times "Prevent vulnerable images from running" is bad designed and falls back to insecure, which is the worst case for a feature that should act as a security safeguard. so in the end if you are worried about the security of your workload, do not use "Prevent vulnerable images from running" at all as it does not prevent you from running insecure images. its misleading and broken. don't use it! use a proper safeguard.
Your interpretation means there arte two choices:
-- Roger Klorese (They/Them or He/Him) Product Line Manager @. @.%0b>Office: 425.709.5248 Mobile: 425.444.5493
[VMware]http://www.vmware.com/
From: mr.shintla @.> Date: Saturday, March 26, 2022 at 2:10 PM To: goharbor/harbor @.> Cc: Subscribed @.***> Subject: Re: [goharbor/harbor] missing vulnerability visibiltity and broken "Prevent vulnerable images from running" which is a security issue (Issue #16218)
⚠ External Email
@hewwhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fheww&data=04%7C01%7Crklorese%40vmware.com%7C9544cd423be14f81c51508da0f6d1e27%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839258595871712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rYwGWu9Vijiu4u%2BzJ0NkPe09Zj7IMzZxzll0TebeRpI%3D&reserved=0 yes i know, as i found this in my tests and thats why i opened this issue, as this is a insecure fallback. and thats why this policy is no valid security defense, and therefore inherently insecure and bad designed!
as written many times "Prevent vulnerable images from running" is bad designed and falls back to insecure, which is the worst case for a feature that should act as a security safeguard. so in the end if you are worried about the security of your workload, do not use "Prevent vulnerable images from running" at all as it does not prevent you from running insecure images. its misleading and broken. don't use it! use a proper safeguard.
— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoharbor%2Fharbor%2Fissues%2F16218%23issuecomment-1079775511&data=04%7C01%7Crklorese%40vmware.com%7C9544cd423be14f81c51508da0f6d1e27%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839258595871712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2cyQRGg6KefSHbU3oda06TudIMPolcX2NSoSnYSdvlU%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZWVL4MOKDAY76ZGQ4QY5DVB54N5ANCNFSM5L3AGGWQ&data=04%7C01%7Crklorese%40vmware.com%7C9544cd423be14f81c51508da0f6d1e27%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839258595871712%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qvwfG%2FgvFl5pXeeUVW5dRoUgMvMFF6I55ULWt71qIX0%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.***>
⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.
"Prevent vulnerable images from running" -> is about images... so this has nothing todo with helm charts. i dont get your point. the option "Prevent vulnerable images from running" should only block images as it is called. in particular: unscanned images
There are two different cases here: non-image OCIO artifacts and cached images. I was addressing the former. As for the latter, it would be necessary to deliver to the cache, then scan, then service the pull to do what you are describing. If the image is delivered before it is in Harbor, it cannot be scanned. Should that first pull deliver the image to Harbor, then scan, then fail the request if it fails the scan? It somewhat defeats the reason people use caches…
-- Roger Klorese (They/Them or He/Him) Product Line Manager @. @.%0b>Office: 425.709.5248 Mobile: 425.444.5493
[VMware]http://www.vmware.com/
From: mr.shintla @.> Date: Saturday, March 26, 2022 at 2:44 PM To: goharbor/harbor @.> Cc: Roger Klorese @.>, Comment @.> Subject: Re: [goharbor/harbor] missing vulnerability visibiltity and broken "Prevent vulnerable images from running" which is a security issue (Issue #16218)
⚠ External Email
"Prevent vulnerable images from running" -> is about images... so this has nothing todo with helm charts. i dont get your point. the option "Prevent vulnerable images from running" should only block images as it is called. in particular: unscanned images
— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoharbor%2Fharbor%2Fissues%2F16218%23issuecomment-1079780179&data=04%7C01%7Crklorese%40vmware.com%7C34319dc1a1fa4f9a243408da0f71baf0%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839278398456246%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=o9IXGBmelQ2dxSTRdK5nQwJKgke7Mx%2BndSa%2BeLiwf6Q%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZWVL5XO5IDGO5MJ5ODKTLVB6AJZANCNFSM5L3AGGWQ&data=04%7C01%7Crklorese%40vmware.com%7C34319dc1a1fa4f9a243408da0f71baf0%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839278398456246%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6ppj%2FsytsO8oSiJl0WTYM8kIJrxQOlM9jq73ALqatqg%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>
⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.
I think the problem here is conflating caching and scanning of images.
The expectation from end users is that both will be satisfied completely, however the current implementation falls short.
Our current workflow is going to mandate that we don't use Harbor for both use cases as it currently can't satisfy them, and instead use it only as an internal repository with a pleasant UI. We will instead implement in-line trivy scanning prior to reaching Harbor, with regular checking of the status of cached images.
Regarding the chart issue - it may be relevant to the code-base, but isn't for those (like us) currently using Harbor only for images. Perhaps it would be helpful to find out how people are using Harbor and base future direction on that?
We had hoped Harbor might provide both a caching/repository and a secure vulnerability scanning platform, but currently it does not. If this is desired and designed in behaviour it might be best to consider removing scanning from the product altogether - as it stands it doesn't provide what most people seem to expect or want. As a security feature it is sadly a lot less than useful.
Our particular use case is actually slightly refined from the above - we'd like to be scanning upstream images that we replicate or cache, but we would also like to be scanning our own derived images for any fresh vulnerability caused by our builds. We also need to ensure that no vulnerable image is pulled. We can only rely on Harbor to provide the registry service for this, so we have had to implement our own in-line scanning, unless and until this issue gets resolved. This seems very sad, as Harbor so very nearly meets our requirements.
A vulnerability-scanned cache, along with a vulnerability-scanned registry which prevents vulnerable images from being pulled would be an enormous boon to a lot of people. I respect your decision (although I disagree strongly with it) if you choose not to implement Harbor that way. I would suggest however that if that is the case that you stop wasting effort on scanning development entirely, as the end result is counterintuitive and leads to a worse security landscape for us all.
Best regards.
I definitely agree that if scanning is set and blocking of insecure images is as well, images should not be delivered without a scan being completed. This obviously seems to require a re-examination of the design.
Sent from my iPhone
On Mar 26, 2022, at 4:26 PM, Danny Smith @.***> wrote:
⚠ External Email
I think the problem here is conflating caching and scanning of images.
The expectation from end users is that both will be satisfied completely, however the current implementation falls short.
Our current workflow is going to mandate that we don't use Harbor for both use cases as it currently can't satisfy them, and instead use it only as an internal repository with a pleasant UI. We will instead implement in-line trivy scanning prior to reaching Harbor, with regular checking of the status of cached images.
Regarding the chart issue - it may be relevant to the code-base, but isn't for those (like us) currently using Harbor only for images. Perhaps it would be helpful to find out how people are using Harbor and base future direction on that?
We had hoped Harbor might provide both a caching/repository and a secure vulnerability scanning platform, but currently it does not. If this is desired and designed in behaviour it might be best to consider removing scanning from the product altogether - as it stands it doesn't provide what most people seem to expect or want. As a security feature it is sadly a lot less than useful.
Our particular use case is actually slightly refined from the above - we'd like to be scanning upstream images that we replicate or cache, but we would also like to be scanning our own derived images for any fresh vulnerability caused by our builds. We also need to ensure that no vulnerable image is pulled. We can only rely on Harbor to provide the registry service for this, so we have had to implement our own in-line scanning, unless and until this issue gets resolved. This seems very sad, as Harbor so very nearly meets our requirements.
A vulnerability-scanned cache, along with a vulnerability-scanned registry which prevents vulnerable images from being pulled would be an enormous boon to a lot of people. I respect your decision (although I disagree strongly with it) if you choose not to implement Harbor that way. I would suggest however that if that is the case that you stop wasting effort on scanning development entirely, as the end result is counterintuitive and leads to a worse security landscape for us all.
Best regards.
— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoharbor%2Fharbor%2Fissues%2F16218%23issuecomment-1079793504&data=04%7C01%7Crklorese%40vmware.com%7C362d9712817f480c8fc708da0f800fea%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839339948290015%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=en3minKu8xAFz07oFZkG17%2BFM3hQXsphXAND0W8BdoQ%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZWVL2EU65QSBTNYMJZ6JTVB6MKPANCNFSM5L3AGGWQ&data=04%7C01%7Crklorese%40vmware.com%7C362d9712817f480c8fc708da0f800fea%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637839339948290015%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=XYY0I05XM63zhjxVwSuxaq2KEUhrrevoecCrMts7BYU%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>
⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.
I completely agree that something needs to change here. The current way that Harbor works is insecure. Consider this:
Say that I set up a project as a pull-through cache for Docker Hub. Then I pull an image through Harbor from Docker Hub. At this point, the image has not been scanned and should be considered to be potentially vulnerable. It should block my pull until the image has been scanned.
Instead of being blocked, the first time I try to pull an image through a pull-through cache, it gets delivered to me. So, even if I'm pulling an image with many critical vulnerabilities, it still gets delivered.
The vulnerable image does get blocked on subsequent pulls but it should never be allowed to be pulled before it has been scanned. This is insecure and will make me reconsider using Harbor unless a plan is made to fix it.
Completely agree with this issue... an image not present in harbor (and therefore not scanned) should not be able to be pulled... I hope a solution will emerge a day... :(
This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.
the ignorance of this security relevant issue and the lack of understanding of basic security concepts does not shed any good light on this project.
meanwhile i've abandoned the plan to use harbor at all.
This is still an issue. Though it is no longer an issue for our organisation as we've had to resort to implementing our own scanning and alerting pipeline outside of Harbor.
the ignorance of this security relevant issue and the lack of understanding of basic security concepts does not shed any good light on this project.
meanwhile i've abandoned the plan to use harbor at all.
First off, comments like this one are very impolite and, after all, pointless. Especially from enterprise users who haven't contributed a single line of code, yet use the project and make money out of it. I hope that one day CNCF will moderate and block such activity on GitHub.
Regarding the lack of understanding of basic security concepts, I was the one who designed it along with many other maintainers and security vendors. When we introduced so called pluggable vulnerability scanners our priority was to deliver it incrementally, support seamless migration from the previous releases (without blocking docker pulls unexpectedly), and collect users feedback to see if we want to add more features related to vulnerability management.
I hope this clarifies the current design and implementation and reminds a bit about standards of communication.
Finally, I do agree with a different or configurable security policy that blocks images that were not scanned and I encourage everyone, especially @toastbrotch, to submit a design proposal and implement it instead of criticizing people that you have never met in your life.
@danielpacak glad you stepped in, and we finally have some qualified answer to this topic. and thank you for finally acknowledging that there is a security issue with current implementation of "Prevent vulnerable images from running".
i do not enter the discussion about the rest of your answer. but i also hope that CNCF does monitor such tickets and enters the discussion way earlier with understanding of security related issues that are not recognized from its devs. as its not ok when a "product" is promoted as secuirty safeguard, which it does not fulfill and it takes over a year to get finaly someone acknowledging the reported security issue.
and finally this security problem with "Prevent vulnerable images from running" and proxy-cache in current implementation should be raised in the docs (sorry if i missed it, but just now searched again).
I just ran across this issue also and I have to say I agree with @toastbrotch , if proxy caches are being used and deployment security is enabled, vulnerable and/or unscanned images should not be pullable. This defeats the use of deployment security, as the insecure new image will have been deployed regardless of the deployment security setting.
In my opinion, ideally it should work like this (when deployment security is enabled):
I understand that case 1 would mean a longer wait on the client side as the client has to wait for the scan to complete before the pull completes but this could be made configurable with something like a proxyCacheWaitForScan
setting. Setting that to true
would make it work as described.
Could that be a solution to the problem?
@OrlinVasilev as discussed.
This issue has now been open for 2,5 years and there is still no reply in regards to proposed solutions for the problem. My last in-person discussion with @OrlinVasilev and @Vad1mo was on the 20th of March (hence my previous comment which was requested by @OrlinVasilev ).
What is your feedback on my proposed solution?
Hi i was trying to get myself a picture about security scan handling and processes around those and so i've set up a harbor 2.4.1. (config pretty standard --with-trivy ) and i hope i did not miss anything...
as posted in slack i think the processes in harbor to handle security vulnerability findings (in particular "Prevent vulnerable images from running.") and the general visibility is missing or even broken:
broken security concept of "Prevent vulnerable images from running" :
so the mechanism to prevent insecure images to go into running state is broken/insecure/pointless.
missing visibility:
so no visibility at all. it's not possible to give harbor access to the security department as they are not able to use it like this. all the integrated security scannings are pointless as the results can only be found by clicking through everything.
i know i've covered lots of topics here, but i just want to open discussion about those, that are very important to my organisation and i think in general, as the whole security mechanisms seem a little pointless if implemented like this. i run quay (which is even more broken and incomplete) and would love to use harbor (cncf graduated and oss), but with those findings above its impossible to justify a migration.
cheers.ivo