search

aquasecurity / trivy-operator

Kubernetes-native security toolkit
https://aquasecurity.github.io/trivy-operator/latest
Apache License 2.0
1.27k stars 211 forks source link

trivy-db fails to download with UNAUTHORIZED: The client does not have permission for manifest aquasecurity/trivy-db/2/manifest.json #695

Closed seb-835 closed 1 year ago

seb-835 commented 1 year ago

What steps did you take and what happened:

trivy-operator pod failed to update db with error : 

init error: DB error: failed to download vulnerability DB:
 OCI artifact error: OCI artifact error: OCI repository error: GET https://******/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: The client does not have permission for manifest; map[manifest:aquasecurity/trivy-db/2/manifest.json]\n"
,"stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:551\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:376\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}

What did you expect to happen:

I expected the trivy-db to be downloaded fine.

Anything else you would like to add:

What permission do i have to look for and authorized to fix this issue ?
Thanks a lot for advice.

Environment:

chen-keinan commented 1 year ago

@seb-835 you should not set up a permission for it , is this happen intermittently? or happen on every scan job?

seb-835 commented 1 year ago

@chen-keinan it happens for every scan job. None Start at all.

chen-keinan commented 1 year ago

@seb-835 we have not changed anything about it in trivy-operator v0.6.0 can you check if you get the same issue with trivy-operator v0.5.0 ?

seb-835 commented 1 year ago

@chen-keinan i install trivy-operator v0.5.0, and got same behaviour.

chen-keinan commented 1 year ago

@chen-keinan i install trivy-operator v0.5.0, and got same behaviour.

thanks for checking , I think it might be env. / config issue. can you please share more info on env./ special config you are using ?

seb-835 commented 1 year ago

i agree @chen-keinan , here is more details, i use a private registry to get/store image , this registry is only able to query docker.io, so image path is aquasec (docker.io) not aquasecurity (ghcr.io) The registry credential are set through imagePullSecrets in both "default", "trivy-system" service account so i have change helm value file to :

image:
  repository:  ***/aquasec/trivy-operator"
serviceAccount:
  name: "trivy-system"

trivy:
  repository: ***/aquasec/trivy"
  dbRepository: ***/aquasec/trivy-db
  dbRepositoryInsecure: "true"

My cluster is deployed with rke2 and cis-compliance enabled, means there is some restriction. But default there is a restricted PSP policy apply to the cluster.

NAME                         PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
global-restricted-psp        false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

But I have assign to the namespace : trivy-system + serviceaccount trivy-system a rolebinding using an unrestricted PSP : 

NAME                         PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
system-unrestricted-psp      true    *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *

it have solve and allow the trivy-operator container to run with root user,

but it does not solve the trivy-db update UNAUTHORIZE , i don't know what can block it and where to search.

Thanks for your help

chen-keinan commented 1 year ago

if you do not have an access to (ghcr.io) , then this is your problem as the trivy-db now is hosted on (ghcr.io) only it use to be also in docker.io in the past but it is no longer maintained for quite a while

you can download it and publish it your self to private repo and overwrite the trivy.dbRepository param via helm or trivy-operator-trivy-config configmap

seb-835 commented 1 year ago

ok, i will fix that on the registry to get the image from ghcr.io and get back to you if i got the same behaviour

seb-835 commented 1 year ago

@chen-keinan : is it expected ? 

docker pull ghcr.io/aquasecurity/trivy-db:2
2: Pulling from /aquasecurity/trivy-db:2
b4f49697b38B : Pulling fs Layer
unexpected end of Json Input
crictl  --config=/var/lib/rancher/rke2/agent/etc/crictl.yaml pull ghcr.io/aquasecurity/trivy-db:2

E1116 14:45:30.116981   75414 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = InvalidArgument desc = failed to pull and unpack image \"ghcr.io/aquasecurity/trivy-db:2\": unable to fetch descriptor (sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) which reports content size of zero: invalid argument" image="ghcr.io/aquasecurity/trivy-db:2"

FATA[0001] pulling image: rpc error: code = InvalidArgument desc = failed to pull and unpack image "ghcr.io/aquasecurity/trivy-db:2": unable to fetch descriptor (sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) 
which reports content size of zero: invalid argument
chen-keinan commented 1 year ago

@chen-keinan : is it expected ?

docker pull ghcr.io/aquasecurity/trivy-db:2
2: Pulling from /aquasecurity/trivy-db:2
b4f49697b38B : Pulling fs Layer
unexpected end of Json Input

you need to use oras see docs

seb-835 commented 1 year ago

Thanks for help @chen-keinan now, all images are pull from ghcr.io, but still having the UNAUTHORIZED issue.

chen-keinan commented 1 year ago

Thanks for help @chen-keinan now, all images are pull from ghcr.io, but still having the UNAUTHORIZED issue.

Is it the same error or different, can you share the log ?

seb-835 commented 1 year ago

Same error

{"level":"error","ts":1668610817.513167,"logger":"reconciler.vulnerabilityreport","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-7759bd949","container":"79ffd7da-935c-43f3-9c1b-0c0c6fdf9fc3","status.reason":"Error","status.message":"2022-11-16T15:00:14.258Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n2022-11-16T15:00:14.258Z\t\u001b[34mINFO\u001b[0m\tDB Repository: registry.xxxx/aquasecurity/trivy-db\n2022-11-16T15:00:14.258Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n2022-11-16T15:00:14.631Z\t\u001b[31mFATAL\u001b[0m\tinit error: DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: GET https://registry.xxxx/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: The client does not have permission for manifest; map[manifest:aquasecurity/trivy-db/2/manifest.json]\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:551\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:376\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}

Does the job use the same ServiceAccount as trivy-operator deployment to get the imagePullSecret to pull the trivy-db ?

chen-keinan commented 1 year ago

Should have , checking and getting back to you

seb-835 commented 1 year ago

hum, even if my artifactory registry pull the image from ghcr.io as puvlic , my cluster need to authenticate against the artifactory registry to get the image.

anyway, if you do not specify a service account in the job manifest, it may use the "default" one of the trivy-system namespace, it put the imagePullSecret in the default too, but still the same issue :(

chen-keinan commented 1 year ago

Download db is done in init container need to check if we can pass TRIVY_USER/TRIVY_PASSWORD then it will work

chen-keinan commented 1 year ago

@seb-835 I will try to find a away to make it work with current release if I will not succeed I will create PR for it

seb-835 commented 1 year ago

@chen-keinan thanks, i have add a look to the pullSecrets but this value is only use by trivy-operator deployment, it is not create as an entry in the trivy-operator-trivy-config configmap used to run the job.

chen-keinan commented 1 year ago

@seb-835 yes it do create entry in trivy-operator-trivy-config cm.

still not sure it will solve your problem , I need to review it deeply and get back to you.

chen-keinan commented 1 year ago

@seb-835 is the images that you want to scan and trivy db are located on the same private registry ?

seb-835 commented 1 year ago

Sorry for late, yes, they are.

chen-keinan commented 1 year ago

Sorry for late, yes, they are.

@seb-835 we might need to do a change in trivy to support download db from private registry , I will get back to you with final answer shortly

mfilotto commented 1 year ago

Any update on this one please ?

chen-keinan commented 1 year ago

Any update on this one please ?

I'll open an issue on trivy to support download db (Server) from private registry

sathieu commented 1 year ago

Any update on this one please ?

I'll open an issue on trivy to support download db (Server) from private registry

This is already possible:

https://github.com/aquasecurity/trivy-operator/blob/487faa08eaac282e4b4018e8dea4b426a6fe0445/deploy/helm/values.yaml#L339

chen-keinan commented 1 year ago

This is already possible:

https://github.com/aquasecurity/trivy-operator/blob/487faa08eaac282e4b4018e8dea4b426a6fe0445/deploy/helm/values.yaml#L339

this is for public repo not private , if it require credential it will now work

sathieu commented 1 year ago

Thanks for the clarification.

mfilotto commented 1 year ago

@chen-keinan what is the number of the issue you opened on trivy ?

chen-keinan commented 1 year ago

@chen-keinan what is the number of the issue you opened on trivy ?

thanks for reminder , here is the issue

mfilotto commented 1 year ago

@chen-keinan the PR for this issue on trivy has just been merged, and the issue is closed.

chen-keinan commented 1 year ago

@chen-keinan the PR for this issue on trivy has just been merged, and the issue is closed.

@mfilotto great. once trivy v0.39.0 will be released I'll update trivy-operator

mfilotto commented 1 year ago

@chen-keinan: how can I configure authentication in helm chart ?

chen-keinan commented 1 year ago

@chen-keinan: how can I configure authentication in helm chart ?

User and Password

mfilotto commented 1 year ago

@chen-keinan I created a secret named trivy-operator-trivy-config with values TRIVY_USERNAME and TRIVY_PASSWORD, but I still have the error UNAUTHORIZED in trivy operator logs. This secret seems to be used only in builtInTrivyServer mode, should I actiavte that mode ? I'm not sure to understand what this mode is for though

chen-keinan commented 1 year ago

@chen-keinan I created a secret named trivy-operator-trivy-config with values TRIVY_USERNAME and TRIVY_PASSWORD, but I still have the error UNAUTHORIZED in trivy operator logs. This secret seems to be used only in builtInTrivyServer mode, should I actiavte that mode ? I'm not sure to understand what this mode is for though

yes , what I have specified is for builtInTrivyServer. builtInTrivyServer param is simply spin up in cluster server, it remove the server concern from the user

mfilotto commented 1 year ago

Now I have this error in trivy server logs TRIVY_USERNAME and TRIVY_PASSWORD are correctly set in secret trivy-operator-trivy-config

$ kubectl logs -f trivy-server-0 2023-04-05T23:17:27.933Z WARN 'TRIVY_SKIP_UPDATE' is deprecated. Use 'TRIVY_SKIP_DB_UPDATE' instead. 2023-04-05T23:17:27.934Z INFO Need to update DB 2023-04-05T23:17:27.934Z INFO DB Repository: docker-nexus.sln.nc/aquasecurity/trivy-db 2023-04-05T23:17:27.934Z INFO Downloading DB... 2023-04-05T23:17:27.988Z FATAL failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:

chen-keinan commented 1 year ago

Now I have this error in trivy server logs TRIVY_USERNAME and TRIVY_PASSWORD are correctly set in secret trivy-operator-trivy-config

$ kubectl logs -f trivy-server-0 2023-04-05T23:17:27.933Z WARN 'TRIVY_SKIP_UPDATE' is deprecated. Use 'TRIVY_SKIP_DB_UPDATE' instead. 2023-04-05T23:17:27.934Z INFO Need to update DB 2023-04-05T23:17:27.934Z INFO DB Repository: docker-nexus.sln.nc/aquasecurity/trivy-db 2023-04-05T23:17:27.934Z INFO Downloading DB... 2023-04-05T23:17:27.988Z FATAL failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred: * GET https://my-private-registry/v2/aquasecurity/trivy-db/manifests/2: UNAUTHORIZED: access to the requested resource is not authorized

@mfilotto Thanks for feedback, I see now that the PR done on trivy support only trivy image ... sub command and not trivy server... example:

$ TRIVY_USERNAME=foo TRIVY_PASSWORD=bar trivy image --db-repository ghcr.io/your/private-db --download-db-only

I'm Checking it with trivy project and getting back to you , opening this issue for now

chen-keinan commented 1 year ago

@mfilotto @seb-835 the latest PR in trivy is not supported for server command. I have created another trivy/4008 PR to support server side , trivy 0.39.2 will be out next week with the fix

chen-keinan commented 1 year ago

@mfilotto now adding trivy v0.40.0 which include credential server support for private repo, will be available with trivy-operator v0.13.1