Pods have violation-related annotations even though they aren't marked as "completed" in the perceptor model?

[jlin963]

Setup: Minikube 1.6, install w/ protoform This was off a brand new cluster (started a new minikube instance, ran the install script for kube1.6, let it start scanning).

Metrics (these numbers haven't changed for the last few hours): Note that out of the 6 total images, it says that only 2 are "scan status complete".

In order to see which pods and images were marked complete, I ran wget http://perceptor:3001/metrics and then curl-ed the model and scan_results (I attached them as txt files but you can copy-paste the JSON output within them into a JSON viewer):

full_model.txt model_scan_results.txt

The two pods that are showing as completed in the model are storage-provisioner and prometheus. However, I'm also seeing violation and policy violation status annotations on some of the other pods (perceptor-scanner, pod-perceiver). Why are these two pods annotated even though they aren't marked as "scan status complete" in the perceptor model?

More info:

describe_pod_perceiver.txt

Screenshots of pods in Kubernetes Dashboard:

And screenshot of scan results in Hub (all 7 images have scan results):

[jayunit100]

@jlin963 its completely possible that , given that your scans have been running for several hours, maybe your metrics don't have the old scan data

[rrati]

@jlin963 The perceiver is dumb. It only annotates information that the perceptor tells it about, so if the pods are being annotated with image data then it is because the perceptor has data from a scan to send to the perceiver. It could also be that the perceptor has some information, ie scan client, and it is sending that to perceiver. Either way, I'm not sure this is a perceiver issue.

[jlin963]

@rrati I wasn't sure whether to open this under perceivers or perceptor, but I was told to open it under perceivers, so that's what I did. I'll try doing this again from scratch (nuking my Hub and minikube instance) and see if I see this again.

[jlin963]

@rrati @mattfenwick Just some stuff I saw today:

I used a different, empty Hub and nuked my minikube instance before starting a new one and installing though protoform. When I went to the Dashboard:

-When I first looked at each of the pods, they hadn't finished being created. At that time, none of them had any annotations (as expected).

-I then refreshed my browser tab for the image-perceiver pod and saw this:

The other pods had been annotated as well with pod-level annotations (pod.overall-status: NOT_IN_VIOLATION, pod.policy-violations: 0, etc.). I was constantly refreshing so I'm fairly sure it didn't have time to finish any scans when I took the above screenshot.

-After the hub scan for all images associated with a particular pod had finished, the annotations were updated again (as expected). Take the image-perceiver pod:

So...yeah, basically I'm confused about why the pods had been annotated with pod-level annotations (pod.overall-status: NOT_IN_VIOLATION, pod.policy-violations: 0, etc.) already before any scans had even been completed.

[rrati]

@jlin963 That is happening because the perceptor is providing the pod in the scan results that the perceiver receives. The perceiver annotates anything in the scan results that it receives from the perceptor.

[jlin963]

But the scans haven't completed yet. Shouldn't it wait to add those annotations until after the scans complete? Otherwise a user might see, say, pod.policy-violations: 0 and pod.vulnerabilities: 0 and mistakenly believe that that pod is fine (no vulnerabilities or violations) when in fact there are vulnerabilities and/or violations, but the scans just haven't finished and updated the annotations yet. That's what I thought when I first saw this before I went and checked the Hub I was using.

@rrati Sorry if I'm completely missing something here. It's just what it seems like to me.

[rrati]

The perceiver doesn't know anything about what is going on with the perceptor/scans. It basically just does what the perceptor tells it to do. I agree that if at least 1 scan hasn't completed that the pod shouldn't be annotated, and in fact I would probably say that a pod shouldn't be annotated until all images are scanned, but that isn't something the perceiver can do on its own. It relies on the perceptor to tell it what to annotate, so if the perceptor is giving the perceiver data for a pod that doesn't have all the images scanned the perceiver has no real way to tell.

There's 2 ways to address this: 1) The perceptor doesn't make scan results for a pod available until all images in the pod are scanned. This continues the simplicity in the perceiver at the cost of adding complexity and a stronger pod knowledge in the perceptor. We have talked about removing pod knowledge from the perceptor all together, which would make this path contrary to that path. 2) Have the perceiver attempt to figure out if all images in a pod have been scanned. This adds complexity to the perceiver and would rely on the perceptor only making scan results available for images that have full/completed scans.

@mattfenwick Thoughts? I'm kind of favoring option 2 if perceptor can meet the image scan result requirement.

[jlin963]

https://github.com/blackducksoftware/perceptor/issues/121

Opening in perceptor and closing this per Matt's instruction after more investigation; looks like it's probably a perceptor bug, not perceivers.