search

aquasecurity / starboard

Moved to https://github.com/aquasecurity/trivy-operator
https://aquasecurity.github.io/starboard/
Apache License 2.0
1.35k stars 197 forks source link

Support private container registries for Pods which do not specifying image pull secrets #120

Open avdhoot opened 4 years ago

avdhoot commented 4 years ago

What steps did you take and what happened:

Followed install instructions. Tried to run scan using below on command.

$ kubectl starboard  find vulnerabilities deployment/XXX-qa-web -n XXXX-qa -v 3
I0814 12:52:38.530373   28112 scanner.go:56] Getting Pod template for workload: {Deployment XXXX-qa-web XXXX-qa}
I0814 12:52:40.842053   28112 scanner.go:71] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0814 12:52:41.183767   28112 runner.go:79] Running task and waiting forever
I0814 12:52:41.183840   28112 runnable_job.go:47] Creating runnable job: starboard/b75ba5e8-82c9-4915-ad35-4b35c37987ab
I0814 12:52:41.535929   28112 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:41.535978   28112 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.19.0-alpha.3/tools/cache/reflector.go:156
I0814 12:52:50.479003   28112 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0814 12:52:50.479115   28112 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0814 12:52:52.784556   28112 manager.go:177] Container default terminated with Error: 2020-08-14T07:22:49.629Z FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (us.gcr.io/XXXX-1/XXXX:116579-23d73da-release-2019-10): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://us.gcr.io/v2/token?scope=repository%3xxxl-1%2FXXXX%3Apull&service=us.gcr.io: UNKNOWN: Unable to parse json key.

error: running scan job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit

What did you expect to happen: Scan should completed without error

Anything else you would like to add: trivy support gcr. But I am not to able find way to pass custom ENV totrivy using starboard.

Environment:

avdhoot commented 4 years ago

@danielpacak In case you missed it ^^

danielpacak commented 4 years ago

Hi @avdhoot ! Thank you for reporting this issue. I'm going to follow up on that and link it to a parent story or epic about configuring scanners. In your case the root cause is that we currently cannot pass TRIVY_USERNAME or TRIVY_PASSWORD envs, but there're more config options that users might want to pass to a scanner.

danielpacak commented 4 years ago

@avdhoot Actually starboard can use image pull secret attached to the Pod template or the service account. In your case did you set the secret on the Pod template level or at the service account level?

avdhoot commented 4 years ago

Our cluster created using kops. We seeded it with registry secrets using below command. We do not have image_pull secret attached to pod. @danielpacak kops create secret dockerconfig -f ~/.docker/config.json

danielpacak commented 4 years ago

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:

You neither use image pull Secret set at Pod template not Service Account. You're using a feature1 of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

avdhoot commented 4 years ago

Thanks @danielpacak you are understanding is correct.

danielpacak commented 4 years ago

@avdhoot I renamed the title of this issue to reflect the specific use case of kops and authorizing Kubernetes nodes to pull images from private registry and repositories.

As already mentioned Starboard only supports recommended way of handling private registries with image pull secrets. We need to think more whether / how to support such cases as you described. /cc @lizrice

n3rd253 commented 3 years ago

@danielpacak

I am trying to make use of the above discussed functionality within AKS. I am not finding that the operator pulls the imagePullSecret of the deployment. In my case I am exposing the secret as imagePullSecret on the deployment level.

Secret generation command as detailed here: ACR Auth Docs

kubectl create secret docker-registry acrcred --namespace platform --docker-server=aagacr.azurecr.io --docker-username=$(AagacrUser) --docker-password=$(AagacrKey)

ACR Cred Secret:

apiVersion: v1 data: .dockerconfigjson: eyJhdXRocyI6eyJhYWdh[...]HSm1PR016WkRkak1UY3dPa2t5T0M1K09XRkZYMHhLZFM1UUxsY3ROMkptT1RZd1J6ZDJZa2QrVW5GM1RYRT0ifX19 kind: Secret metadata: creationTimestamp: "2020-11-09T18:02:46Z" managedFields: apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:.dockerconfigjson: {} f:type: {} manager: kubectl-create operation: Update time: "2020-11-09T18:02:46Z" name: acrcred namespace: platform resourceVersion: "14763349" selfLink: /api/v1/namespaces/platform/secrets/acrcred uid: 886f8fca-4ae6-4a9a-a6b0-133a75d606af type: kubernetes.io/dockerconfigjson`

Deployment Spec:

spec: containers: envFrom: configMapRef: name: pnrhelperservice-configmap image: aagacr.azurecr.io/platform/pnrhelperservice:20201109.3 imagePullPolicy: Always livenessProbe: failureThreshold: 3 httpGet: path: /api/v1/ping port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: platform-pnrhelperservice ports: containerPort: 8080 name: pnrhelpersvc protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /api/v1/ping port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 200m memory: 64Mi requests: cpu: 100m memory: 32Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: ALL privileged: false runAsNonRoot: true runAsUser: 10000 startupProbe: failureThreshold: 30 httpGet: path: /api/v1/ping port: 8080 scheme: HTTP initialDelaySeconds: 15 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: mountPath: /tmp name: tmp mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: svc-platform-pnrhelperservice-token-zjpdx readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: name: acrcred nodeName: aks-apppool-15092464-vmss00000y preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 10000 runAsNonRoot: true runAsUser: 10000 serviceAccount: svc-platform-pnrhelperservice serviceAccountName: svc-platform-pnrhelperservice terminationGracePeriodSeconds: 30 tolerations: effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists volumes: emptyDir: {} name: tmp name: tls-platform-pnrhelperservice secret: defaultMode: 420 secretName: tls-platform-pnrhelperservice name: acrcred secret: defaultMode: 420 secretName: acrcred name: svc-platform-pnrhelperservice-token-zjpdx secret: defaultMode: 420 secretName: svc-platform-pnrhelperservice-token-zjpdx

Operator Logs:

Loading... {"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]} {"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]} {"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"} {"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"} {"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="} {"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="} {"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"} {"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1} {"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"} {"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1} {"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"} {"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]} {"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]} {"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"} {"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"} {"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="} {"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="} {"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"} {"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1} {"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"} {"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1} {"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}

Looking forward to your response, as it would save my day.

danielpacak commented 3 years ago

@danielpacak

I am trying to make use of the above discussed functionality within AKS. I am not finding that the operator pulls the imagePullSecret of the deployment. In my case I am exposing the secret as imagePullSecret on the deployment level.

Secret generation command as detailed here: ACR Auth Docs

kubectl create secret docker-registry acrcred --namespace platform --docker-server=aagacr.azurecr.io --docker-username=$(AagacrUser) --docker-password=$(AagacrKey)

ACR Cred Secret:

apiVersion: v1 data: .dockerconfigjson: eyJhdXRocyI6eyJhYWdh[...]HSm1PR016WkRkak1UY3dPa2t5T0M1K09XRkZYMHhLZFM1UUxsY3ROMkptT1RZd1J6ZDJZa2QrVW5GM1RYRT0ifX19 kind: Secret metadata: creationTimestamp: "2020-11-09T18:02:46Z" managedFields: apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:.dockerconfigjson: {} f:type: {} manager: kubectl-create operation: Update time: "2020-11-09T18:02:46Z" name: acrcred namespace: platform resourceVersion: "14763349" selfLink: /api/v1/namespaces/platform/secrets/acrcred uid: 886f8fca-4ae6-4a9a-a6b0-133a75d606af type: kubernetes.io/dockerconfigjson`

Deployment Spec:

spec: containers: envFrom: configMapRef: name: pnrhelperservice-configmap image: aagacr.azurecr.io/platform/pnrhelperservice:20201109.3 imagePullPolicy: Always livenessProbe: failureThreshold: 3 httpGet: path: /api/v1/ping port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: platform-pnrhelperservice ports: containerPort: 8080 name: pnrhelpersvc protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /api/v1/ping port: 8080 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: 200m memory: 64Mi requests: cpu: 100m memory: 32Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: ALL privileged: false runAsNonRoot: true runAsUser: 10000 startupProbe: failureThreshold: 30 httpGet: path: /api/v1/ping port: 8080 scheme: HTTP initialDelaySeconds: 15 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: mountPath: /tmp name: tmp mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: svc-platform-pnrhelperservice-token-zjpdx readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: name: acrcred nodeName: aks-apppool-15092464-vmss00000y preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 10000 runAsNonRoot: true runAsUser: 10000 serviceAccount: svc-platform-pnrhelperservice serviceAccountName: svc-platform-pnrhelperservice terminationGracePeriodSeconds: 30 tolerations: effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists volumes: emptyDir: {} name: tmp name: tls-platform-pnrhelperservice secret: defaultMode: 420 secretName: tls-platform-pnrhelperservice name: acrcred secret: defaultMode: 420 secretName: acrcred name: svc-platform-pnrhelperservice-token-zjpdx secret: defaultMode: 420 secretName: svc-platform-pnrhelperservice-token-zjpdx

Operator Logs:

Loading... {"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]} {"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]} {"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"} {"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"} {"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="} {"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="} {"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"} {"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1} {"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"} {"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1} {"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(_Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"} {"level":"info","ts":1604949460.8825052,"logger":"main","msg":"Resolving install mode","install mode":"SingleNamespace","operator namespace":"starboard-operator","target namespaces":["platform"]} {"level":"info","ts":1604949460.8826096,"logger":"main","msg":"Constructing multi-namespaced cache","namespaces":["platform","starboard-operator"]} {"level":"info","ts":1604949461.888167,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1604949461.888932,"logger":"main","msg":"Using Trivy as vulnerability scanner","image":"aquasec/trivy:0.11.0"} {"level":"info","ts":1604949461.8890698,"logger":"main","msg":"Starting controllers manager"} {"level":"info","ts":1604949461.8893642,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1604949461.8895195,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","source":"kind source: /, Kind="} {"level":"info","ts":1604949461.8897321,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","source":"kind source: /, Kind="} {"level":"info","ts":1604949462.0902774,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod"} {"level":"info","ts":1604949462.0903811,"logger":"controller","msg":"Starting workers","reconcilerGroup":"","reconcilerKind":"Pod","controller":"pod","worker count":1} {"level":"info","ts":1604949462.0904198,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job"} {"level":"info","ts":1604949462.090432,"logger":"controller","msg":"Starting workers","reconcilerGroup":"batch","reconcilerKind":"Job","controller":"job","worker count":1} {"level":"error","ts":1604949471.7839265,"logger":"controller.job","msg":"Scan job container","job":"starboard-operator/2b1b65b9-e131-4e13-9601-68fa1eb3cab6","container":"platform-pnrhelperservice","status.reason":"Error","status.message":"2020-11-09T19:17:51.225Z\t\u001b[31mFATAL\u001b[0m\tunable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t unable to inspect the image (aagacr.azurecr.io/platform/pnrhelperservice:20201109.3): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t_ GET https://aagacr.azurecr.io/oauth2/token?scope=repository%3Aplatform%2Fpnrhelperservice%3Apull&service=aagacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\n\n\n","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).processFailedScanJob\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:163\ngithub.com/aquasecurity/starboard/pkg/operator/controller/job.(JobController).Reconcile\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/job/job_controller.go:69\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}

Looking forward to your response, as it would save my day.

Hi @n3rd253 Currently we only support private registries with Starboard CLI. I hasn't been implemented for the operator yet, but we're working on that

mathieudebrito commented 3 years ago

Hi guys ! Just discovered this awesome project and I was wondering why Starboard could not fetch the images from my private repo :p Note : the imagePullSecrets is given at the deployment level : 

kind: Deployment
apiVersion: apps/v1
metadata:
  name: my-deployment
spec:
  template:
    spec:
      containers:
        - name: my-container
          image: 'private-repo/image-name:version'
      imagePullSecrets:
        - name: regcred

Note 2 : Even if I try it using the CLI, it does not work : 

kubectl starboard scan vulnerabilityreports deployment/my-deployment
->
E0208 12:30:00.426504   59282 runnable_job.go:164] Container my-deployment terminated with Error: 2021-02-08T11:29:59.619Z      FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (<private-repo/my-deployment:version>): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        * GET https://private-repo/blabla : unsupported status code 401

error: running scan job: warning event received: Job has reached the specified backoff limit (BackoffLimitExceeded)

Is there any news / ETA on this or is this something you don't consider to add on a short time ?

Thank you very much anyway, it's a great job !

danielpacak commented 3 years ago

Hi @mathieudebrito Starboard CLI and Starboard Operator v0.9 should read image pull Secret from pod template as explained here. If that's not the case it may be regression. I'll double check on our end.

Could you provide more details regarding the private registry used in your setup? Is it Docker Hub or a managed registry such as ECR or GCR? Which authentication method are you using? Basic vs auth token?

mathieudebrito commented 3 years ago

Hi @danielpacak Thank you for your quick answer !

If it helps, the secret is created as follow :

kubectl create secret docker-registry regcred --docker-server=$REGISTRY_URL --docker-username=$REGISTRY_USER --docker-password=$REGISTRY_PASSWORD --dry-run -o yaml | kubectl apply -f -

Thank you again !

danielpacak commented 3 years ago

Hey @mathieudebrito I can reproduce this error so it seems to be regression or a corner case that's not handled properly by Starboard (both operator and CLI as they share this functions). I'll track it down and try to fix it in the upcoming bug fix release v0.9.1.

Note: For some reason we cannot map registry server to container that refers to private image around these lines https://github.com/aquasecurity/starboard/blob/main/pkg/kube/secrets.go#L49

danielpacak commented 3 years ago

I found what was causing this problem. Essentially you can create image pull secret which is actually concatenation of registry server (rg.pl-waw.scw.cloud) and repository name (private). If that was the case Starboard could not parse registry server properly and match it with registry server taken from image reference. BTW, docker login strips down the repository part when it updates ~/.docker/config.json file

This was causing problem in v0.9.0

$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud/private \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -

This would work in v0.9.0

$ kubectl create secret docker-registry regcred \
  --docker-server=rg.pl-waw.scw.cloud \
  --docker-username=nologin \
  --docker-password=$SCW_SECRET_TOKEN --dry-run -o yaml | kubectl apply -f -
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - image: rg.pl-waw.scw.cloud/private/nginx:1.16
        imagePullPolicy: Always
        name: nginx
EOF

This particular problem is fixed in #379. However the original issue was created to support private registries without specifying image pull secrets on a pod. For example, by configuring nodes to authenticate with private registry.

mathieudebrito commented 3 years ago

@danielpacak awesome thank you ! Indeed, I understand the issue is not the same, I should have created a new one, sorry for the inconvenience :(

Edit : Just tested the 0.9.1, it fixes well the issue I was having. I can now see the vulnerability reports, thank you so much for you so quick fix !

maati55 commented 3 years ago

Hi, any chance to add supporting the private registry to the Helm, please? I'm getting this error: unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (****.azurecr.io/app:v1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://****.azurecr.io/oauth2/token?scope=repository%3Aapp%3Apull&service=****.azurecr.io: UNAUTHORIZED: authentication required

danielpacak commented 3 years ago

Hi, any chance to add supporting the private registry to the Helm, please? I'm getting this error: unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:\n\t* unable to inspect the image (****.azurecr.io/app:v1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* GET https://****.azurecr.io/oauth2/token?scope=repository%3Aapp%3Apull&service=****.azurecr.io: UNAUTHORIZED: authentication required

Please create a separate issue for that @maati55 . This issue has nothing to do with Helm charts. We'll never close it if we report this way.

ryanbrainard commented 2 years ago

Thank you for providing all the details @avdhoot So just to rephrase and confirm my understanding:

You neither use image pull Secret set at Pod template not Service Account. You're using a feature1 of kops to store the config.json in /root/.docker/config.json on all nodes (including masters) so that both Kubernetes and system containers may use registries defined in it.

@avdhoot I renamed the title of this issue to reflect the specific use case of kops and authorizing Kubernetes nodes to pull images from private registry and repositories.

As already mentioned Starboard only supports recommended way of handling private registries with image pull secrets. We need to think more whether / how to support such cases as you described. /cc @lizrice

We are running into this same issue. Could it perhaps be as simple as adding a configuration to mount the Docker config file from the host as a volume onto the Starboard pods doing the scanning?

danielpacak commented 2 years ago

👋 @ryanbrainard I think we have already mentioned somewhere that mounting container runtime sockets is suboptimal for these reasons:

If you want to contribute to designing such functionality we'd appreciate reviewing design docs under the docs/design directory that are related to private and managed registries. You can also submit your own design proposal by following the template from existing docs.

VF-mbrauer commented 2 years ago

@avdhoot, @maati55, @mathieudebrito, @ryanbrainard, you may have a look at the PR I created: https://github.com/aquasecurity/starboard/pull/1103