search

aquasecurity / trivy-operator

Kubernetes-native security toolkit
https://aquasecurity.github.io/trivy-operator/latest
Apache License 2.0
1.25k stars 208 forks source link

clusterComplaince and infraAssessement report activated by default #1078

Closed VF-mbrauer closed 1 year ago

VF-mbrauer commented 1 year ago

What steps did you take and what happened:

Even after disabling the clusterComplaince and infraAssessement with

# clusterComplianceEnabled the flag to enable cluster compliance report generation
  clusterComplianceEnabled: false
# infraAssessmentScannerEnabled the flag to enable infra assessment scanner
  infraAssessmentScannerEnabled: false

the logs still showing that there is an error with that:

{"level":"error","ts":"2023-03-20T12:39:41Z","logger":"reconciler.clustercompliancereport","msg":"failed to generate compliance report","compliance report":"/nsa","error":"no matches for kind \"ClusterInfraAssessmentReport\" in version \"aquasecurity.github.io/v1alpha1\"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).generateComplianceReport.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:68\nk8s.io/client-go/util/retry.OnError.func1\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:222\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:228\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.1/pkg/util/wait/wait.go:423\nk8s.io/client-go/util/retry.OnError\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/util/retry/util.go:50\nk8s.io/client-go/util/retry.RetryOnConflict\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/util/retry/util.go:104\ngithub.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).generateComplianceReport\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:50\ngithub.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).reconcileComplianceReport.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:44\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}

What did you expect to happen:

Anything else you would like to add:

Environment:

chen-keinan commented 1 year ago

@VF-mbrauer I'm unable to reproduce this issue , could you please share more info on config or chain of events before and after issue occur

sj-williams commented 1 year ago

Also seeing this issue following a trivy-operator helm upgrade from 0.10.2 to 0.12.1. Kubernetes : 1.22.17

{"level":"error","ts":"2023-03-23T16:42:54Z","logger":"reconciler.clustercompliancereport","msg":"failed to generate compliance report","compliance report":"/pss-baseline","error":"no matches for kind \"ClusterInfraAssessmentReport\" in version \"aquasecurity.github.io/v1alpha1\"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).generateComplianceReport.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:68\nk8s.io/client-go/util/retry.OnError.func1\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:222\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:228\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.2/pkg/util/wait/wait.go:423\nk8s.io/client-go/util/retry.OnError\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/util/retry/util.go:50\nk8s.io/client-go/util/retry.RetryOnConflict\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/util/retry/util.go:104\ngithub.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).generateComplianceReport\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:50\ngithub.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).reconcileComplianceReport.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:44\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}

helm get values

USER-SUPPLIED VALUES:
operator:
  builtInTrivyServer: true
  clusterComplianceEnabled: false
  infraAssessmentScannerEnabled: false
 ....

Before upgrade, I had not specified clusterComplianceEnabled at all (it still is not actually referenced in the base chart values.yaml ) and had not been seeing this error. Adding it after errors surfaced has not had any effect.

elchenberg commented 6 months ago

I see this issue with ClusterComplianceReports.

# kubectl get clustercompliancereports.aquasecurity.github.io 
NAME             AGE
nsa              13h
pss-baseline     13h
pss-restricted   13h

The cis ClusterComplianceReport is missing because I deleted it to check if it gets re-created.

Trivy Operator version: ghcr.io/aquasecurity/trivy-operator:0.19.4

Trivy Operator Helm chart version: 0.21.4

Environment Variables:

# trivy-operator container
      - env:
        - name: OPERATOR_NAMESPACE
          value: trivy
        - name: OPERATOR_TARGET_NAMESPACES
        - name: OPERATOR_EXCLUDE_NAMESPACES
        - name: OPERATOR_TARGET_WORKLOADS
        - name: OPERATOR_SERVICE_ACCOUNT
          value: trivy-operator
        envFrom:
        - configMapRef:
            name: trivy-operator-config

# trivy-operator-config configmap
  CONTROLLER_CACHE_SYNC_TIMEOUT: 5m
  OPERATOR_ACCESS_GLOBAL_SECRETS_SERVICE_ACCOUNTS: "true"
  OPERATOR_BATCH_DELETE_DELAY: 10s
  OPERATOR_BATCH_DELETE_LIMIT: "10"
  OPERATOR_BUILT_IN_TRIVY_SERVER: "false"
  OPERATOR_CACHE_REPORT_TTL: 120h
  OPERATOR_CLUSTER_COMPLIANCE_ENABLED: "false"
  OPERATOR_CLUSTER_SBOM_CACHE_ENABLED: "false"
  OPERATOR_CONCURRENT_NODE_COLLECTOR_LIMIT: "1"
  OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT: "5"
  OPERATOR_CONFIG_AUDIT_SCANNER_ENABLED: "false"
  OPERATOR_CONFIG_AUDIT_SCANNER_SCAN_ONLY_CURRENT_REVISIONS: "true"
  OPERATOR_EXPOSED_SECRET_SCANNER_ENABLED: "true"
  OPERATOR_HEALTH_PROBE_BIND_ADDRESS: :9090
  OPERATOR_INFRA_ASSESSMENT_SCANNER_ENABLED: "false"
  OPERATOR_LEADER_ELECTION_ENABLED: "true"
  OPERATOR_LEADER_ELECTION_ID: trivyoperator-lock
  OPERATOR_LOG_DEV_MODE: "false"
  OPERATOR_MERGE_RBAC_FINDING_WITH_CONFIG_AUDIT: "false"
  OPERATOR_METRICS_BIND_ADDRESS: :8080
  OPERATOR_METRICS_CLUSTER_COMPLIANCE_INFO_ENABLED: "false"
  OPERATOR_METRICS_CONFIG_AUDIT_INFO_ENABLED: "false"
  OPERATOR_METRICS_EXPOSED_SECRET_INFO_ENABLED: "false"
  OPERATOR_METRICS_FINDINGS_ENABLED: "true"
  OPERATOR_METRICS_IMAGE_INFO_ENABLED: "false"
  OPERATOR_METRICS_INFRA_ASSESSMENT_INFO_ENABLED: "false"
  OPERATOR_METRICS_RBAC_ASSESSMENT_INFO_ENABLED: "false"
  OPERATOR_METRICS_VULN_ID_ENABLED: "true"
  OPERATOR_PRIVATE_REGISTRY_SCAN_SECRETS_NAMES: '{"trivy":"trivy-operator-registry-credentials"}'
  OPERATOR_RBAC_ASSESSMENT_SCANNER_ENABLED: "false"
  OPERATOR_SBOM_GENERATION_ENABLED: "false"
  OPERATOR_SCAN_JOB_RETRY_AFTER: 5m
  OPERATOR_SCAN_JOB_TIMEOUT: 15m
  OPERATOR_SCAN_JOB_TTL: 1h
  OPERATOR_SCANNER_REPORT_TTL: 72h
  OPERATOR_SEND_DELETED_REPORTS: "false"
  OPERATOR_VULNERABILITY_SCANNER_ENABLED: "true"
  OPERATOR_VULNERABILITY_SCANNER_SCAN_ONLY_CURRENT_REVISIONS: "true"
  OPERATOR_WEBHOOK_BROADCAST_TIMEOUT: 30s
  OPERATOR_WEBHOOK_BROADCAST_URL: ""
  TRIVY_SERVER_HEALTH_CHECK_CACHE_EXPIRATION: 10h
elchenberg commented 6 months ago

I guess the reason why I have ClusterComplianceReports in my cluster is because the files in deploy/helm/templates/specs get deployed.

Wouldn't it make sense to only deploy them when .Values.operator.clusterComplianceEnabled is true?

chen-keinan commented 6 months ago

@elchenberg yes it does make sense, would you like to raise a PR ?

elchenberg commented 6 months ago

Yes, I would like to. Maybe at the weekend.