timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match

btwseeu78 commented 1 year ago

Enviornment Details

chartVersion : v0.12.1 trivytag : v0.38.3 GKE Version: 1.25.6-gke.1000 mode: standalone

after updating to latest version we are getting errors on trivy scanjob

{"level":"error","ts":"2023-03-24T09:54:10Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-57d5bdbb9d","container":"webapp-promtheus-metrics","status.reason":"Error","status.message":"2023-03-24T09:54:07.646Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}

configs


data:
  configAuditReports.scanner: Trivy
  node.collector.imageRef: 'ghcr.io/aquasecurity/node-collector:0.0.5'
  report.recordFailedChecksOnly: 'true'
  scanJob.compressLogs: 'true'
  scanJob.podTemplateContainerSecurityContext: >-
    {"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"readOnlyRootFilesystem":true}
  vulnerabilityReports.scanner: Trivy```

```apiVersion: v1
data:
  trivy.additionalVulnerabilityReportFields: ''
  trivy.command: image
  trivy.dbRepository: ghcr.io/aquasecurity/trivy-db
  trivy.dbRepositoryInsecure: 'false'
  trivy.httpProxy: 'http://proxy.internal.svc.cluster.local:80'
  trivy.httpsProxy: 'http://proxy.internal.svc.cluster.local:80'
  trivy.imagePullSecret: artifact-registry
  trivy.mode: Standalone
  trivy.noProxy: >-
    172.20.0.0/16, *.alertmanager-operated, *.monitoring, *.cert-manager.svc,
    127.0.0.1, localhost, 169.254.169.254, metadata, metadata.google.internal,
    *.googleapis.com, *.alpha.applis.renault.fr
  trivy.repository: >-
    europe-docker.pkg.dev/irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/trivy
  trivy.resources.limits.cpu: 500m
  trivy.resources.limits.memory: 2Gi
  trivy.resources.requests.cpu: 100m
  trivy.resources.requests.memory: 900Mi
  trivy.severity: 'HIGH,CRITICAL'
  trivy.slow: 'true'
  trivy.supportedConfigAuditKinds: >-
    Workload,Service,Role,ClusterRole,NetworkPolicy,Ingress,LimitRange,ResourceQuota
  trivy.tag: 0.38.3
  trivy.timeout: 5m0s
  trivy.useBuiltinRegoPolicies: 'true'```

Envvars
--------

`env:
        - name: OPERATOR_NAMESPACE
          value: trivy-system
        - name: OPERATOR_TARGET_NAMESPACES
        - name: OPERATOR_EXCLUDE_NAMESPACES
          value: 'kube-system, trivy-system'
        - name: OPERATOR_TARGET_WORKLOADS
          value: >-
            pod,replicaset,replicationcontroller,statefulset,daemonset,cronjob,job
        - name: OPERATOR_SERVICE_ACCOUNT
          value: trivy-custom
        - name: OPERATOR_LOG_DEV_MODE
          value: 'false'
        - name: OPERATOR_SCAN_JOB_TTL
          value: 5m
        - name: OPERATOR_SCAN_JOB_TIMEOUT
          value: 5m
        - name: OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT
          value: '10'
        - name: OPERATOR_CONCURRENT_NODE_COLLECTOR_LIMIT
          value: '1'
        - name: OPERATOR_SCAN_JOB_RETRY_AFTER
          value: 30s
        - name: OPERATOR_BATCH_DELETE_LIMIT
          value: '10'
        - name: OPERATOR_BATCH_DELETE_DELAY
          value: 10s
        - name: OPERATOR_METRICS_BIND_ADDRESS
          value: ':8080'
        - name: OPERATOR_METRICS_FINDINGS_ENABLED
          value: 'true'
        - name: OPERATOR_METRICS_VULN_ID_ENABLED
          value: 'false'
        - name: OPERATOR_HEALTH_PROBE_BIND_ADDRESS
          value: ':9090'
        - name: OPERATOR_VULNERABILITY_SCANNER_ENABLED
          value: 'true'
        - name: OPERATOR_VULNERABILITY_SCANNER_SCAN_ONLY_CURRENT_REVISIONS
          value: 'true'
        - name: OPERATOR_SCANNER_REPORT_TTL
          value: 24h
        - name: OPERATOR_CONFIG_AUDIT_SCANNER_ENABLED
          value: 'true'
        - name: OPERATOR_RBAC_ASSESSMENT_SCANNER_ENABLED
          value: 'false'
        - name: OPERATOR_INFRA_ASSESSMENT_SCANNER_ENABLED
          value: 'false'
        - name: OPERATOR_CONFIG_AUDIT_SCANNER_SCAN_ONLY_CURRENT_REVISIONS
          value: 'true'
        - name: OPERATOR_EXPOSED_SECRET_SCANNER_ENABLED
          value: 'true'
        - name: OPERATOR_METRICS_EXPOSED_SECRET_INFO_ENABLED
          value: 'false'
        - name: OPERATOR_WEBHOOK_BROADCAST_URL
        - name: OPERATOR_WEBHOOK_BROADCAST_TIMEOUT
          value: 30s
        - name: OPERATOR_PRIVATE_REGISTRY_SCAN_SECRETS_NAMES
          value: '{}'
        - name: OPERATOR_ACCESS_GLOBAL_SECRETS_SERVICE_ACCOUNTS
          value: 'true'
        - name: OPERATOR_BUILT_IN_TRIVY_SERVER
          value: 'false'
        - name: TRIVY_SERVER_HEALTH_CHECK_CACHE_EXPIRATION
          value: 10h
        - name: OPERATOR_MERGE_RBAC_FINDING_WITH_CONFIG_AUDIT
          value: 'false'`

chen-keinan commented 1 year ago

@btwseeu78 could you please share the scan-job descriptor

btwseeu78 commented 1 year ago

@chen-keinan The job is not getting created, the logs are from operator pod.

chen-keinan commented 1 year ago

@chen-keinan The job is not getting created, the logs are from operator pod.

could you please put here the descriptor of the pods it tries to scan

chen-keinan commented 1 year ago

@btwseeu78 I want to see if pod has multi container on same registry

btwseeu78 commented 1 year ago

this is the container shows in error.


kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: cd9b308fba17d2bc2a13f238094ad9e4ba55a4d6564d69dea3544ed6f669980f
    cni.projectcalico.org/podIP: 172.20.188.24/32
    cni.projectcalico.org/podIPs: 172.20.188.24/32
    data-ingest.dynatrace.com/injected: "true"
    dynakube.dynatrace.com/injected: "true"
    oneagent.dynatrace.com/injected: "true"
  creationTimestamp: "2023-03-10T10:54:04Z"
  generateName: demo-webapp-promtheus-metrics-8879585b8-
  labels:
    app.kubernetes.io/instance: demo
    app.kubernetes.io/name: webapp-promtheus-metrics
    pod-template-hash: 8879585b8
  name: demo-webapp-promtheus-metrics-8879585b8-nhq58
  namespace: keda-demo
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: demo-webapp-promtheus-metrics-8879585b8
    uid: 13535427-4897-4b61-b73c-2841e58b3c3e
  resourceVersion: "973464126"
  uid: addabec0-1ce5-40cf-9416-78813d2526c4
spec:
  containers:
  - env:
    - name: DT_DEPLOYMENT_METADATA
      value: orchestration_tech=Operator-cloud_native_fullstack;script_version=v0.9.1;orchestrator_id=b200f391-97db-4a64-9d74-094e2d19fb19
    - name: LD_PRELOAD
      value: /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
    - name: DT_NETWORK_ZONE
      value: dev-tech-70740
    image: europe-docker.pkg.dev/irn-71889-adm-dev-ope-68/docker-gke-irn70740/webapp-promtheus-metrics:v0.1.0
    imagePullPolicy: Always
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: web
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: webapp-promtheus-metrics
    ports:
    - containerPort: 8080
      name: web
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: web
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 64Mi
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-p64xx
      readOnly: true
    - mountPath: /etc/ld.so.preload
      name: oneagent-share
      subPath: ld.so.preload
    - mountPath: /opt/dynatrace/oneagent-paas
      name: oneagent-bin
    - mountPath: /var/lib/dynatrace/oneagent/agent/config/container.conf
      name: oneagent-share
      subPath: container_webapp-promtheus-metrics.conf
    - mountPath: /var/lib/dynatrace/enrichment
      name: data-ingest-enrichment
    - mountPath: /var/lib/dynatrace/enrichment/endpoint
      name: data-ingest-endpoint
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: artifact-registry
  initContainers:
  - args:
    - init
    env:
    - name: CONTAINERS_COUNT
      value: "1"
    - name: FAILURE_POLICY
      value: silent
    - name: K8S_PODNAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: K8S_PODUID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: K8S_BASEPODNAME
      value: demo-webapp-promtheus-metrics-8879585b8
    - name: K8S_CLUSTER_ID
      value: b200f391-97db-4a64-9d74-094e2d19fb19
    - name: K8S_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: FLAVOR
    - name: TECHNOLOGIES
      value: all
    - name: INSTALLPATH
      value: /opt/dynatrace/oneagent-paas
    - name: INSTALLER_URL
    - name: VERSION
    - name: MODE
      value: provisioned
    - name: ONEAGENT_INJECTED
      value: "true"
    - name: CONTAINER_1_NAME
      value: webapp-promtheus-metrics
    - name: CONTAINER_1_IMAGE
      value: europe-docker.pkg.dev/irn-71889-adm-dev-ope-68/docker-gke-irn70740/webapp-promtheus-metrics:v0.1.0
    - name: DT_WORKLOAD_KIND
      value: Deployment
    - name: DT_WORKLOAD_NAME
      value: demo-webapp-promtheus-metrics
    - name: DATA_INGEST_INJECTED
      value: "true"
    image: europe-docker.pkg.dev/irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/dynatrace-operator:v0.9.1
    imagePullPolicy: IfNotPresent
    name: install-oneagent
    resources:
      limits:
        cpu: 300m
        memory: 1536Mi
      requests:
        cpu: 100m
        memory: 512Mi
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mnt/bin
      name: oneagent-bin
    - mountPath: /mnt/share
      name: oneagent-share
    - mountPath: /mnt/config
      name: injection-config
    - mountPath: /var/lib/dynatrace/enrichment
      name: data-ingest-enrichment
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-p64xx
      readOnly: true
  nodeName: gke-gke-irn-70740-dev-tec-devtec-n2-6be02883-mxv6
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  priorityClassName: standard
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-p64xx
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
  - name: injection-config
    secret:
      defaultMode: 420
      secretName: dynatrace-dynakube-config
  - csi:
      driver: csi.oneagent.dynatrace.com
      volumeAttributes:
        dynakube: dynakube-dev-tech-70740
        mode: app
    name: oneagent-bin
  - emptyDir: {}
    name: oneagent-share
  - name: data-ingest-endpoint
    secret:
      defaultMode: 420
      secretName: dynatrace-data-ingest-endpoint
  - emptyDir: {}
    name: data-ingest-enrichment
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:54:05Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:54:07Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:54:07Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-03-10T10:54:04Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://c31f135b1de27f6f40be24008fef1abf567bb62264418872f5627080cc861645
    image: europe-docker.pkg.dev/irn-71889-adm-dev-ope-68/docker-gke-irn70740/webapp-promtheus-metrics:v0.1.0
    imageID: europe-docker.pkg.dev/irn-71889-adm-dev-ope-68/docker-gke-irn70740/webapp-promtheus-metrics@sha256:5fd00b1a52cce0299923a2c408e1cbc2c97fea85cdd3b66a9ef00c6835fafad2
    lastState: {}
    name: webapp-promtheus-metrics
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-03-10T10:54:06Z"
  hostIP: 172.20.187.167
  initContainerStatuses:
  - containerID: containerd://f3b03a9e7d910226488128bf051167487794cf99215cbc06e64a3ec05467ea27
    image: europe-docker.pkg.dev/irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/dynatrace-operator:v0.9.1
    imageID: europe-docker.pkg.dev/irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/dynatrace-operator@sha256:ce621425125ba8fdcfa0f300c75e0167e9301a4654fcd1c14baa75f4d41151a3
    lastState: {}
    name: install-oneagent
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://f3b03a9e7d910226488128bf051167487794cf99215cbc06e64a3ec05467ea27
        exitCode: 0
        finishedAt: "2023-03-10T10:54:05Z"
        reason: Completed
        startedAt: "2023-03-10T10:54:05Z"
  phase: Running
  podIP: 172.20.188.24
  podIPs:
  - ip: 172.20.188.24
  qosClass: Burstable
  startTime: "2023-03-10T10:54:04Z"```

chen-keinan commented 1 year ago

@btwseeu78 all of the images in pod are using the same imagePullSecret ?

btwseeu78 commented 1 year ago

yes all of the images using the same ones ,its just and example its happening with all actually.

{"level":"error","ts":"2023-03-28T09:46:38Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-d6f948488","container":"prometheus-operator","status.reason":"Error","status.message":"2023-03-28T09:46:35.563Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
14
{"level":"error","ts":"2023-03-28T09:46:41Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-57d5bdbb9d","container":"webapp-promtheus-metrics","status.reason":"Error","status.message":"2023-03-28T09:46:38.660Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
13
{"level":"error","ts":"2023-03-28T09:46:43Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-578df7dd75","container":"certmanager-webhook","status.reason":"Error","status.message":"2023-03-28T09:46:40.559Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
12
{"level":"error","ts":"2023-03-28T09:46:55Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-67c58b69c5","container":"oauth2-proxy","status.reason":"Error","status.message":"2023-03-28T09:46:52.562Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
11
{"level":"error","ts":"2023-03-28T09:46:55Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-844cfbc4b9","container":"goldilocks","status.reason":"Error","status.message":"2023-03-28T09:46:52.751Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
10
{"level":"error","ts":"2023-03-28T09:46:59Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-75d4b89674","container":"grafana","status.reason":"Error","status.message":"2023-03-28T09:46:57.035Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
9
{"level":"error","ts":"2023-03-28T09:46:59Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-75d4b89674","container":"grafana-sc-dashboard","status.reason":"Error","status.message":"2023-03-28T09:46:56.737Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
8
{"level":"error","ts":"2023-03-28T09:46:59Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-75d4b89674","container":"grafana-sc-datasources","status.reason":"Error","status.message":"2023-03-28T09:46:56.852Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
7
{"level":"error","ts":"2023-03-28T09:47:02Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-59cf984865","container":"dex","status.reason":"Error","status.message":"2023-03-28T09:46:59.679Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
6
{"level":"error","ts":"2023-03-28T09:47:02Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-5c5b5b6f57","container":"exporter","status.reason":"Error","status.message":"2023-03-28T09:47:00.356Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
5
{"level":"error","ts":"2023-03-28T09:47:16Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-fb5464d9f","container":"dynakube-operator","status.reason":"Error","status.message":"2023-03-28T09:47:12.851Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
4
{"level":"error","ts":"2023-03-28T09:47:32Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-fb5464d9f","container":"dynakube-operator","status.reason":"Error","status.message":"2023-03-28T09:47:29.674Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
3
{"level":"error","ts":"2023-03-28T09:48:16Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-57d5bdbb9d","container":"webapp-promtheus-metrics","status.reason":"Error","status.message":"2023-03-28T09:48:13.962Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
2
{"level":"error","ts":"2023-03-28T09:48:59Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-fb5464d9f","container":"dynakube-operator","status.reason":"Error","status.message":"2023-03-28T09:48:56.874Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
1
{"level":"error","ts":"2023-03-28T09:49:13Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-fb5464d9f","container":"dynakube-operator","status.reason":"Error","status.message":"2023-03-28T09:49:10.788Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}

chen-keinan commented 1 year ago

@btwseeu78 thanks , I'll try to reproduce it , If I'll need more info I'll ping you

btwseeu78 commented 1 year ago

we use private repository for all of our components as its happening with most of them at least

btwseeu78 commented 1 year ago

scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     PodInitializing   0          9s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   1/1     Running           0          10s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     Error             0          11s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Init:1/2          0          9s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     Error             0          12s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     PodInitializing   0          10s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     Error             0          13s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     Error             0          14s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     Terminating       0          14s
scan-vulnerabilityreport-57d5bdbb9d-6khmv   0/1     Terminating       0          14s
scan-vulnerabilityreport-fb5464d9f-p6rlq    1/1     Running           0          11s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Error             0          13s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Error             0          14s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Error             0          15s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Error             0          15s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Terminating       0          16s
scan-vulnerabilityreport-fb5464d9f-p6rlq    0/1     Terminating       0          16s

chen-keinan commented 1 year ago

scan-vulnerabilityreport-fb5464d9f-p6rlq 0/1 Error 0 15s scan-vulnerabilityreport-fb5464d9f-p6rlq 0/1 Error 0 15s scan-vulnerabilityreport-fb5464d9f-p6rlq 0/1 Terminating 0 16s scan-vulnerabilityreport-fb5464d9f-p6rlq 0/1 Terminating 0 16s

I know why the error is happening ,I need to check why its hapening in your use case

chen-keinan commented 1 year ago

@btwseeu78 can you please share how the Secret is created ?

trivy-operator expect it to be created in form of (example) :

 kubectl create secret docker-registry <secret name> --docker-server=https://index.docker.io/v1/ --docker-username=<User> --docker-password=<Password> --docker-email=<email>

btwseeu78 commented 1 year ago

No, its not in this format, we use google artifact registry they in json_key format ,we dont have username password based access.If other apps can pull the image with the same dockerconfigjson, trivy should not be exception.

chen-keinan commented 1 year ago

No, its not in this format, we use google artifact registry they in json_key format ,we dont have username password based access.If other apps can pull the image with the same dockerconfigjson, trivy should not be exception.

@btwseeu78 the kubernetes.io/dockerconfigjson is good format, the data inside the secret has to include BasicAuth, essentially the operator should be able to extract credentials (user and password) out of it, as this is how trivy authenticate against private registry

run :

 kubectl get secret <secret name> -n namespace> -o json

can you confirm if data include BasicAuth ?

btwseeu78 commented 1 year ago

@chen-keinan its service account key one of the format for long live token used by google


cat KEY-FILE | docker login -u _json_key --password-stdin \
https://HOSTNAME
    ``
https://cloud.google.com/container-registry/docs/advanced-authentication#json-key

btwseeu78 commented 1 year ago

i cant share exact details but it has the fields, the only things would be the username would be - "_json_key", password is service account private key ,its the default format used by google, it also contains field called "auth" but if you want to know if they contains those fields my answer would be yes they do

chen-keinan commented 1 year ago

@btwseeu78 as describe above the operator extract user/password from secret and pass it to trivy-scan-job , so if it include a valid credential then it should be fine.

on another question, the pod you put here has two containers (where scan show this error) ,does scanning works for other pods/replicaSet you have with single container ? or scanning in not working at all for you .

btwseeu78 commented 1 year ago

@chen-keinan now this became very complicated for me ,so its not entirely down, it is working for public repos for example prometheus-sd-exporter this one uses public repo , but its not working for any private registry, we only use artifact registry so its not working for artifact registry.I might have noticed this sooner but it seems it not able use the key provided .

but again if thats the case then its another problem, the testing might be be localized for gke using json service account key as registry secret, seems its working for other images that does not uses Artifact registry. for example these images.

monitoring                 replicaset-stackdriver-exporter-f59cf5964-stackdriver-exporter   prometheuscommunity/stackdriver-exporter                             v0.12.0            Trivy     3h52m
nginx-ingress-controller   replicaset-585c6ff7b                                             defaultbackend-amd64                                                 1.5                Trivy     3h54m
nginx-ingress-controller   replicaset-69b6d87b55                                            ingress-nginx/controller                                                                Trivy     3h42m
nginx-ingress-controller   replicaset-88b8c65                                               defaultbackend-amd64                                                 1.5                Trivy     3h54m
nginx-ingress-controller   replicaset-b96649b59                                             ingress-nginx/control

btwseeu78 commented 1 year ago

so probably if that's the case we would love to have the feature to have the same serviceaccount json key to be used for authentication since its one of the standard methods for apps running in GKE.

i just checked on of cluster running older version seems same issue was there got unnoticed by everyone.

but here also if you check below images they got scanned rather recently from the same repo,we are using same keys it synced through vault so no change of getting modified.

                                 v2.30.2                                     Trivy     281d
monitoring                 replicaset-fb6974846                                              devops/prometheus-operator                                                                       v0.46.0                                     Trivy     398d
monitoring                 replicaset-grafana-stack-5866bc8b5b-grafana                       irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/grafana                                      8.5.10                                      Trivy     169d
monitoring                 replicaset-grafana-stack-5866bc8b5b-grafana-sc-dashboard          irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/k8s-sidecar                                  1.17.0                                      Trivy     169d
monitoring                 replicaset-grafana-stack-5866bc8b5b-grafana-sc-datasources        irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/k8s-sidecar                                  1.17.0                                      Trivy     169d
monitoring                 replicaset-grafana-stack-67546d6f45-grafana                       irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/grafana                                      9.2.0                                       Trivy     161d
monitoring                 replicaset-grafana-stack-67546d6f45-grafana-sc-dashboard          irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/k8s-sidecar                                  1.17.0                                      Trivy     161d
monitoring                 replicaset-grafana-stack-67546d6f45-grafana-sc-datasources        irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/k8s-sidecar                                  1.17.0                                      Trivy     161d
monitoring                 replicaset-grafana-stack-6c555cd64c-grafana                       irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/grafana                                      9.3.8                                       Trivy     154m
monitoring                 replicaset-grafana-stack-6c555cd64c-grafana-sc-dashboard          irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/k8s-sidecar                                  1.22.0                                      Trivy     154m
monitoring                 replicaset-grafana-stack-6c555cd64c-grafana-sc-datasources        irn-71889-adm-ope-ope-d2/docker-shared-gke-irn70740/k8s-sidecar

seems its not related to this version but i wonder in which version they got created on the first place.

chen-keinan commented 1 year ago

@btwseeu78 can you make a simple test , create a secret as I describe above, and deploy a pod with container from private repo which associated with that secret , I want to make sure that you can get containers fro. private repo scanned at all , once we figure out that maybe we can check what enhancement can be done to support your use case

btwseeu78 commented 1 year ago

My organization does not allow me to pull from any private repository apprently.i will require some time to replicate this with simlar configs.

just for info though: https://aquasecurity.github.io/trivy/v0.28.1/docs/advanced/private-registries/gcr/

we are using the same format key as mentioned in this document by aqua security.

chen-keinan commented 1 year ago

My organization does not allow me to pull from any private repository apprently.i will require some time to replicate this with simlar configs.

just for info though: https://aquasecurity.github.io/trivy/v0.28.1/docs/advanced/private-registries/gcr/

we are using the same format key as mentioned in this document by aqua security.

@btwseeu78 thanks for the clarification , currently trivy-operator do not support this type of secret key , but definitely it is possible to support it

btwseeu78 commented 1 year ago

And just to add:

{
    "auths": {
        "https://europe-docker.pkg.dev": {
            "username": "_json_key",
            "password": "{\n  \"type\": \"service_account\",\n  \"project_id\": \"\",\n  \"private_key_id\": \"\",\n  \"private_key\": \"\"\n}",
            "email": "",
            "auth": ""
        }
    }
}

this type of keys also not working i think should work maybe.i replaced the contents inside password key they contains some secret data.

btwseeu78 commented 1 year ago

@chen-keinan its ok with private repo with traditional username password login, i reproduced the error with google AR service account creds it can not use them as imagepullsecret. just for info the secret in used from google service account created from this,key.json is google service account key in json format and email is serviceaccount email.

kubectl -n=NAMESPACE_NAME create secret docker-registry SECRET_NAME --docker-server HOST_NAME --docker-username _json_key --docker-email ANY_VALID_EMAIL --docker-password="$(cat ~/key.json)"

chen-keinan commented 1 year ago

@btwseeu78 thanks for update. if you have time to you can add support (PR) for service account role. if not I'll pick it up later

github-actions[bot] commented 1 year ago

This issue is stale because it has been labeled with inactivity.

github-actions[bot] commented 1 year ago

This issue is stale because it has been labeled with inactivity.

outbreaker commented 1 year ago

Hello @chen-keinan, we have exacly the same error with json key from google, did you find a solution?

chen-keinan commented 1 year ago

@outbreaker there is a PR #1401 to support service account json type

github-actions[bot] commented 11 months ago

This issue is stale because it has been labeled with inactivity.

quixoten commented 11 months ago

This issue (i think) started effecting our gitlab security scans:

.scan-image-template:
  extends: container_scanning
  variables:
    CS_REGISTRY_PASSWORD: $GCLOUD_AUTH_JSON
    CS_REGISTRY_USER: _json_key

As best as I can tell, the above started failing for us around 2023-10-24T14:05:59.968Z. I can't find any record of anything changing on our side, but we are using gitlab's shared template, so something might have changed there. i'm also wondering if this upstream change could potentially be the culprit.

the value of the GCLOUD_AUTH_JSON variable looks like this:

{
  "type": "service_account",
  "project_id": "[redacted]",
  "private_key_id": "[redacted]",
  "private_key": "[redacted]",
  "client_email": "[redacted]",
  "client_id": "[redacted]",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "[redacted]",
  "universe_domain": "googleapis.com"
}

i tried a version with all of the extra white space removed as well, but the failure persists, i.e.,

FATAL flag error: registry flag error: the length of usernames and passwords must match

byronmccollum commented 11 months ago

This issue (i think) started effecting our gitlab security scans:

I can confirm, seeing the same with GitLab Container Scanner. The version of Trivy changed from 0.36.1 to 0.43.1 sometime recently, and we are using a service account key as well.

byronmccollum commented 11 months ago

I think the issue might be related to support for multiple credentials. I haven't looked at the code, but it expects a comma separated list of users, and a comma separated list of passwords. Unfortunately, the JSON "password" has commas in it, which explains the mismatch between the number of users and passwords provided.

https://github.com/aquasecurity/trivy/pull/3906/files#diff-a812993c8d91c3eacbf5538b3f3cffaa8a1fe76c48f493caf7882484113d648fR8

It can handle multiple sets of credentials as well:

$ export TRIVY_USERNAME=USERNAME1,USERNAME2
$ export TRIVY_PASSWORD=PASSWORD1,PASSWORD2
$ trivy image YOUR_PRIVATE_IMAGE

In the example above, Trivy attempts to use two pairs of credentials:

- USERNAME1/PASSWORD1
- USERNAME2/PASSWORD2

Please note that the number of usernames and passwords must be the same.

byronmccollum commented 11 months ago

@quixoten -- Found a fix. Instead of specifying CS_REGISTRY_PASSWORD and CS_REGISTRY_USER separately, Trivy supports setting GOOGLE_APPLICATION_CREDENTIALS pointed to your service account key file.

This seems to be working well for me.

btwseeu78 commented 11 months ago

Yes, that's with trivia parameters it only needs google application crews to json file. and for this issue I will close it since the code changes related to k8s operator is already pushed and merged.

aquasecurity / trivy-operator