Closed stealthrabbi closed 1 year ago
Hello, I am having the same issue there. Worked perfectly fine this morning until the Vulnerability DB updates:
trivy -v
Version: 0.33.0
Vulnerability DB:
Version: 2
UpdatedAt: 2023-01-12 06:07:03.840852771 +0000 UTC
NextUpdate: 2023-01-12 12:07:03.840852271 +0000 UTC
DownloadedAt: 2023-01-12 06:41:12.347924267 +0000 UTC
Hello,
maybe it is related to search.maven.org:
2023-01-12T12:55:15.461Z DEBUG request failed {"error": "Get \"https://search.maven.org/solrsearch/select?q=1%3A%22185ae7422deb18aa14464b18cf9ce997d4710a57%22&rows=1&wt=json\": net/http: TLS handshake timeout", "method": "GET", "url": "https://search.maven.org/solrsearch/select?q=1%3A%22185ae7422deb18aa14464b18cf9ce997d4710a57%22&rows=1&wt=json"}
Looks like GitHub Container Registry distributing Trivy DB is something wrong now. Please try it again later.
Seeing the same issue with Trivy 0.32.1
.
Is everyone scanning images with JAR files? If so, maven might be relevant like @dusik66 pointed out. --offline-scan
may help until the server is back.
In my case, yes, I am scanning an image for a java proj.
Can you try --offline-scan
then? It doesn't try to connect to search.maven.org
.
I reported the same issue here: https://github.com/aquasecurity/trivy-action/issues/190
If you run with --offline-scan
then you even get additional features ;-)
In addition, if you want to scan Java dependencies such as JAR and pom.xml, you need to specify --offline-scan since Trivy tries to issue API requests for scanning Java applications by default.
https://aquasecurity.github.io/trivy/v0.36/docs/advanced/air-gap/
so --offline-scan
won't prevent the main trivy DB from being downloaded, it just will not reach out to internet for the actual scanning portion?
so --offline-scan won't prevent the main trivy DB from being downloaded, it just will not reach out to internet for the actual scanning portion?
Right.
To skip downloading DB, you can pass --skip-db-update
, but you need to have the old database locally. Otherwise, the scanning will fail.
We are probably facing a similar issue — starting from today (the first fail was at Jan 12, 2023, 11:30 AM GMT
), the GitLab pipelines started to fail.
0.36.1
.10m
and then to 30m
.$ trivy --cache-dir .trivycache/ image --exit-code 0 --no-progress --format template --template "@/contrib/junit.tpl" --output "$CI_PROJECT_DIR/container-scan-junit.xml" "$FULL_IMAGE_NAME" --timeout 30m0s
2023-01-12T18:01:31.130Z INFO Need to update DB
2023-01-12T18:01:31.130Z INFO DB Repository: ghcr.io/aquasecurity/trivy-db
2023-01-12T18:01:31.130Z INFO Downloading DB...
2023-01-12T18:01:33.467Z INFO Vulnerability scanning is enabled
2023-01-12T18:01:33.467Z INFO Secret scanning is enabled
2023-01-12T18:01:33.467Z INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-12T18:01:33.467Z INFO Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-12T18:06:55.053Z FATAL image scan error: scan error: scan failed: failed analysis: analyze error: failed to analyze layer: : walk error: failed to process the file: failed to analyze file: failed to analyze app/BOOT-INF/lib/log4j-core-2.19.0.jar: unable to open app/BOOT-INF/lib/log4j-core-2.19.0.jar: failed to open: unable to read the file: read tcp 172.17.0.2:47734->3.5.139.107:443: read: connection reset by peer
Uploading artifacts for failed job
Setting --offline-scan
option is helping, but I am not sure whether it's a good long-term solution.
What happens if we set --offline-scan
? The documentation is not very clear about it.
If it stops to refresh the CVE database, this isn't the best constant solution.
+1
+1
+1
Still happening in my case and works after retrying 3 times:
...
2023-01-13T02:20:42.699Z INFO Need to update DB
2023-01-13T02:20:42.699Z INFO Downloading DB...
16.89 MiB / 36.05 MiB [---------------------------->________________________________] 46.86% ? p/s ?
32.95 MiB / 36.05 MiB [------------------------------------------------------->_____] 91.42% ? p/s ?
36.05 MiB / 36.05 MiB [----------------------------------------------------------->] 100.00% ? p/s ?
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 32.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 32.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 32.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.03 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.03 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.03 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 28.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [-------------------------------------------------] 100.00% 19.06 MiB p/s 2.1s
2023-01-13T02:25:42.822Z FATAL scan error: image scan failed: failed analysis: analyze error: timeout: context deadline exceeded
...
Hello guys!!
I checked this problem now. Looks like it was temporal down and it works now. I tried to reproduce this problem, but got correct response from maven 1000 times in a row. Can you check it and write if there are still errors?
@paultung may be your problem has different reason. Can you send your jar
file for more investigate?
Hello, the problem still exists.
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:385]: {
"uuid": "bf9b6744-ebbc-11eb-bcb0-fef69661a874",
"name": "Trivy",
"description": "The Trivy scanner adapter",
"url": "http://registry-harbor-trivy:8080",
"disabled": false,
"is_default": true,
"health": "healthy",
"auth": "",
"access_credential": "[HIDDEN]",
"skip_certVerify": false,
"use_internal_addr": true,
"adapter": "Trivy",
"vendor": "Aqua Security",
"version": "v0.32.1",
"create_time": "2021-07-23T13:49:06.678883Z",
"update_time": "2021-07-27T11:34:11.007996Z"
}
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:385]: {
"registry": {
"url": "http://registry-harbor-core:80",
"authorization": "[HIDDEN]"
},
"artifact": {
"namespace_id": 135,
"repository": "*******",
"tag": "latest",
"digest": "sha256:3cd2b01db4c4299f6bddcf1b93d6bf51f0953dd1998f295cccc56f34c4ff486c",
"mime_type": "application/vnd.docker.distribution.manifest.v2+json"
}
}
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:222]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2023-01-13T06:34:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:34:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:34:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:34:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:36:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:36:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:36:11Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2023-01-13T06:34:40.074Z [34mINFO[0m Vulnerability scanning is enabled
2023-01-13T06:36:06.461Z [31mFATAL[0m image scan error: scan error: scan failed: failed analysis: analyze error: failed to analyze layer: : walk error: failed to process the file: failed to analyze file: failed to analyze liquibase/internal/lib/ojdbc8.jar: unable to open liquibase/internal/lib/ojdbc8.jar: failed to open: unable to read the file: unexpected EOF
: general response handler: unexpected status code: 500, expected: 200
Hello @Dmitry-Dymnov
Looks like problem is not with maven repository.
Trivy can't read liquibase/internal/lib/ojdbc8.jar
file in your liquibase***.jar
(am not sure of file name).
Can you submit this file for investigation?
Also can you say version of Trivy? We fixed case where Trivy would stop scanning after receiving error for internal jars.(https://github.com/aquasecurity/trivy/pull/2989)
Regards, Dmitriy
@DmitriyLewen, until yesterday, these images were scanned without problems. I'm sorry, but I can't give you the file ((( Trivy used as part of the harbor. ("version": "v0.32.1")
Hello guys!!
I checked this problem now. Looks like it was temporal down and it works now. I tried to reproduce this problem, but got correct response from maven 1000 times in a row. Can you check it and write if there are still errors?
@paultung may be your problem has different reason. Can you send your
jar
file for more investigate?
I've just run a trivy execution and it's failing again
+ /opt/trivy-scan/trivy -d --cache-dir /opt/trivy-scan/.trivycache/ image --no-progress --format json -o /tmp/reports/aea-chapi-omnichannel-trivy-report-1673595621.json aea-tp/channel:alpha-91d20501-91340-241 --ignorefile /tmp/.trivyignore
2023-01-13T07:40:21.965Z DEBUG Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-01-13T07:40:21.970Z DEBUG cache dir: /opt/trivy-scan/.trivycache/
2023-01-13T07:40:21.970Z DEBUG There is no valid metadata file: unable to open a file: open /opt/trivy-scan/.trivycache/db/metadata.json: no such file or directory
2023-01-13T07:40:21.970Z INFO Need to update DB
2023-01-13T07:40:21.970Z INFO DB Repository: ghcr.io/aquasecurity/trivy-db
2023-01-13T07:40:21.970Z INFO Downloading DB...
2023-01-13T07:40:21.970Z DEBUG no metadata file
2023-01-13T07:40:24.300Z DEBUG Updating database metadata...
2023-01-13T07:40:24.301Z DEBUG DB Schema: 2, UpdatedAt: 2023-01-13 06:07:02.14805908 +0000 UTC, NextUpdate: 2023-01-13 12:07:02.14805848 +0000 UTC, DownloadedAt: 2023-01-13 07:40:24.301082232 +0000 UTC
2023-01-13T07:40:24.301Z INFO Vulnerability scanning is enabled
2023-01-13T07:40:24.302Z DEBUG Vulnerability type: [os library]
2023-01-13T07:40:24.302Z INFO Secret scanning is enabled
2023-01-13T07:40:24.302Z INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-13T07:40:24.302Z INFO Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-13T07:40:24.319Z DEBUG No secret config detected: trivy-secret.yaml
2023-01-13T07:40:24.320Z DEBUG Image ID: sha256:e0ea4be578c1e55dfc047dc4621a3c8e460cf2cc080dc2c56c36d08858ca80ba
2023-01-13T07:40:24.320Z DEBUG Diff IDs: [sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3 sha256:9dc38e038a4b30a44a39340027b138e3d52f5256e5449ce32b74f7f21475a796 sha256:0c6a8c08d9664f51203017fe0dd63a0ad69a0da2fc6cb01bd021314ae5bdecd3 sha256:29e51557cb02f590cef20e0a9ba769e030c27e8bd42076d42f2acaa49cfd5f38 sha256:13a14e21710e57c5e06cfa617638216e95b066f0b3f38663fed6a85563e89dce sha256:916158f15540724646ab7f924d27542c47aa1ca392333c03b1d19386793f9a3f sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef sha256:df59f20536381fbafd8002f10f4d38a62516da52d6f0e610341a7426ebe99f4f sha256:c9ec7b8d4aea08c7884108ce8ab2dd96883998b57f1ebb33af014149956c396d sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef]
2023-01-13T07:40:24.320Z DEBUG Base Layers: [sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3]
2023-01-13T07:40:24.320Z DEBUG Missing image ID in cache: sha256:e0ea4be578c1e55dfc047dc4621a3c8e460cf2cc080dc2c56c36d08858ca80ba
2023-01-13T07:40:24.320Z DEBUG Missing diff ID in cache: sha256:13a14e21710e57c5e06cfa617638216e95b066f0b3f38663fed6a85563e89dce
2023-01-13T07:40:24.320Z DEBUG Missing diff ID in cache: sha256:9dc38e038a4b30a44a39340027b138e3d52f5256e5449ce32b74f7f21475a796
2023-01-13T07:40:24.320Z DEBUG Missing diff ID in cache: sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3
2023-01-13T07:40:24.320Z DEBUG Missing diff ID in cache: sha256:29e51557cb02f590cef20e0a9ba769e030c27e8bd42076d42f2acaa49cfd5f38
2023-01-13T07:40:24.320Z DEBUG Missing diff ID in cache: sha256:0c6a8c08d9664f51203017fe0dd63a0ad69a0da2fc6cb01bd021314ae5bdecd3
2023-01-13T07:40:26.786Z DEBUG Missing diff ID in cache: sha256:916158f15540724646ab7f924d27542c47aa1ca392333c03b1d19386793f9a3f
2023-01-13T07:40:26.855Z DEBUG Missing diff ID in cache: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
2023-01-13T07:40:26.859Z DEBUG Missing diff ID in cache: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
2023-01-13T07:40:26.862Z DEBUG Missing diff ID in cache: sha256:df59f20536381fbafd8002f10f4d38a62516da52d6f0e610341a7426ebe99f4f
2023-01-13T07:40:27.125Z DEBUG Parsing Java artifacts... {"file": "home/docker/app/app.jar"}
2023-01-13T07:40:27.127Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/logback-access-spring-boot-starter-2.7.1.jar"}
2023-01-13T07:40:27.127Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/tomcat-embed-websocket-9.0.65.jar"}
2023-01-13T07:40:27.241Z DEBUG Missing diff ID in cache: sha256:c9ec7b8d4aea08c7884108ce8ab2dd96883998b57f1ebb33af014149956c396d
2023-01-13T07:40:27.305Z DEBUG Missing diff ID in cache: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
2023-01-13T07:40:29.355Z DEBUG Parsing Java artifacts... {"file": "opt/java/openjdk/lib/jrt-fs.jar"}
2023-01-13T07:40:29.792Z DEBUG No such POM in the central repositories {"file": "jrt-fs.jar"}
2023-01-13T07:41:27.790Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22javax.websocket%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-13T07:42:47.932Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%22bd70dfeb39cc83c6934be24fa377b21e541dbe76%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-13T07:43:08.005Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/tomcat-embed-core-9.0.31.jar"}
2023-01-13T07:43:08.152Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/logback-access-1.2.11.jar"}
2023-01-13T07:43:08.152Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/logstash-logback-encoder-4.11.jar"}
2023-01-13T07:43:08.153Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/logback-core-1.2.11.jar"}
2023-01-13T07:43:08.153Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/spring-security-oauth2-autoconfigure-2.6.8.jar"}
2023-01-13T07:44:08.220Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%22d9ce52f45c4689c1f445d08550ed7c4adcb920e3%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-13T07:44:28.292Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/spring-security-oauth2-2.5.2.RELEASE.jar"}
2023-01-13T07:44:28.292Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/springdoc-openapi-ui-1.6.6.jar"}
2023-01-13T07:44:28.292Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/springdoc-openapi-webmvc-core-1.6.6.jar"}
2023-01-13T07:44:28.293Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/spring-webmvc-5.3.22.jar"}
2023-01-13T07:44:28.366Z DEBUG Parsing Java artifacts... {"file": "BOOT-INF/lib/spring-boot-actuator-autoconfigure-2.7.3.jar"}
2023-01-13T07:45:21.991Z WARN Increase --timeout value
2023-01-13T07:45:21.991Z FATAL image scan error:
github.com/aquasecurity/trivy/pkg/commands/artifact.Run
/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:397
- scan error:
github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:257
- scan failed:
github.com/aquasecurity/trivy/pkg/commands/artifact.scan
/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:588
- failed analysis:
github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
/home/runner/work/trivy/trivy/pkg/scanner/scan.go:146
- analyze error:
github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.Inspect
/home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:127
- timeout:
github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspect
/home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:248
- context deadline exceeded
seems like search.maven.org is still unstable or has a more aggressive rate limiting. After some requests (where the response is really fast) I get "504 Gateway Timeout".
What is search.maven.org used for? Only for checking whether an update fixing a vulnerability of maven dependency is available?
with the latest trivy image no problems anymore: docker.io/aquasec/trivy With option --vuln-type os
https://hub.tekton.dev/tekton/task/trivy-scanner uses docker.io/aquasec/trivy@sha256:dea76d4b50c75125cada676a87ac23de2b7ba4374752c6f908253c3b839201d9
and will hang until timeout with option --vuln-type os So probably a version problem? Because no jar scan is with search.maven.org is enabled ....
until yesterday, these images were scanned without problems.
This is so strange. In this case Trivy doesn't go to jar parse logic. Can you open new issue? I think this is different question.
seems like search.maven.org is still unstable
It looks like you are right. I am playing with maven repository REST API. I get 504 error intermittently. But it can happen on the first request. It doesn't seem to be related to number of requests.
What is search.maven.org used for?
If your jar file doesn't have information about GroupID
, ArtifactID
and Version
in pom.properties
or MANIFEST.MF
then:
We use maven repository to try to get GroupID
, ArtifactID
and Version
by SHA-1 digest
or get GroupID
by artifactID
(artifactID got from file name)
If your jar file doesn't have information about
GroupID
,ArtifactID
andVersion
inpom.properties
orMANIFEST.MF
then: We use maven repository to try to getGroupID
,ArtifactID
andVersion
by SHA-1 digest or getGroupID
byartifactID
(artifactID got from file name)
I don't this is 100% accurate, it is also used to complement the db, docs state in bold
And I just confirmed a couple of CVE that are not reported when running with --offline-scan
.
Imo --offline-scan
involves some risk.
Same Problem here - with activated debug Option, we get
docker run --rm --entrypoint trivy aquasec/trivy image webapp --timeout 30m --security-checks vuln -d
2023-01-13T08:53:45.739Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%225dd2c405ef14507e909cf6537f0e614328fb1af4%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:45.997Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%229c15159c5e1a8f6dd5381fadf7b0dbeb49f2860e%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:46.786Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22Apache+Tomcat%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:46.786Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22javax.annotation%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:46.797Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22Apache+Tomcat%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5}
but opening the urls in browser works. Are there any request-header which make this broken?
I'm also having these issues with trivy 0.36.1. Just to add some extra info, here's our situation:
We only have this issue in our (bitbucket) ci/cd pipeline. Locally on my laptop, this doesn't seem to be an issue.
Also we do two scans in out build pipeline, fs
and image
. The fs scan runs fine and scans all our dependencies as expected. But the image scan times out on getting the info from search.maven.org
but opening the urls in browser works. Are there any request-header which make this broken?
Links don't always open. Your link opened 1 out of 2 times for me.
And I just confirmed a couple of CVE that are not reported when running with --offline-scan. Imo --offline-scan involves some risk.
This is because Trivy cannot get dependency information from jar file (from pom.properties
and MANIFEST.MF
) and from maven repository (get 504 error).
A quick request to https://search.maven.org from a browser gives you a 504 Gateway Timeout error When the website was still available but the searches were already failing yesterday, I saw a banner on top saying their new search repository will be at https://central.sonatype.dev/ So Maven may have moved the search services to this new domain which, as I've tried a few simple searches on it, seems to work fine.
Thanks for information @Hugo-D !
But their REST API guide still uses https://search.maven.org/solrsearch
url (https://central.sonatype.org/search/rest-api-guide/)
I tried to change search.maven.org/solrsearch
/search.maven.org
to central.sonatype.dev/
, but got 404 error...
Same issue here. We use GitHub Workflows and the latest trivy-action version 0.8.
The problem still exists:
2023-01-13T09:45:51.883Z INFO Downloading DB...
13.36 MiB / 36.05 MiB [---------------------->______________________________________] 37.05% ? p/s ?33.86 MiB / 36.05 MiB [--------------------------------------------------------->___] 93.91% ? p/s ?36.05 MiB / 36.05 MiB [----------------------------------------------------------->] 100.00% ? p/s ?36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 37.78 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 37.78 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 37.78 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 35.35 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 35.35 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 35.35 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 33.07 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 33.07 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 33.07 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.93 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [-------------------------------------------------] 100.00% 14.16 MiB p/s 2.7s2023-01-13T09:45:55.467Z INFO Vulnerability scanning is enabled
2023-01-13T09:45:55.468Z INFO Secret scanning is enabled
2023-01-13T09:45:55.468Z INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-13T09:45:55.468Z INFO Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-13T09:50:51.978Z WARN Increase --timeout value
2023-01-13T09:50:51.978Z FATAL image scan error: scan error: scan failed: failed analysis: analyze error: timeout: context deadline exceeded
After using --offline-scan
the error disappeared.
@DmitriyLewen
If your jar file doesn't have information about GroupID, ArtifactID and Version in pom.properties or MANIFEST.MF then: We use maven repository to try to get GroupID, ArtifactID and Version by SHA-1 digest or get GroupID by artifactID(artifactID got from file name)
I understand it, but is there no way to save this information in the database? Maven creates pom.properties
by default, there shouldn't be much JAR files without it. Same for MANIFEST.MF
. In a lot of cases there is also the pom.xml
in den JAR.
Maybe it would help if Trivy logs the JARs which are not containing theses files. Then we can see if there is an easier solution.
In a lot of cases there is also the pom.xml in den JAR.
I'm not sure there is resource with a list of jars that we can use to save to the database.
That's why we use maven repository to analyze similar jars.
@DmitriyLewen
In a lot of cases there is also the pom.xml in den JAR.
I'm not sure there is resource with a list of jars that we can use to save to the database.
That's why we use maven repository to analyze similar jars.
I understand and I don't know it, too.
I think logging the JARs would help to see if there is an issue. I checked one of my projects and almost all JARs have one of these three files. Maybe there is some issue with finding these files. Logs would be helpful to improve it.
Seems like maven.org has issues: https://status.maven.org/incidents/zs6lhc4nykst
I think logging the JARs would help to see if there is an issue
Trivy does not use a server for scanning (all scans are done on your ЗС), so saving logs is currently not possible. Also, if we get some information from each scan - users will have a lot of questions about this, because users scan their personal files and etc. @knqyf263 I am right?
I think logging the JARs would help to see if there is an issue
Trivy does not use a server for scanning (all scans are done on your ЗС), so saving logs is currently not possible. Also, if we get some information from each scan - users will have a lot of questions about this, because users scan their personal files and etc. @knqyf263 I am right?
Sure, I meant my own logs. You could provide debug logs (enabled by a flag), so Java developers can see, which JARs are not containing these files. It helps to find problems.
If I see a JAR, which has one of these three files I could file a feature request to improve Trivy. Right know I don't know how much JARs have this problem and which ones. But I see in my project that almost all JARs have these files.
https://status.maven.org/incidents/mblzqd69jjpz this looks relevant to the issues people are seeing here
One note to others looking at this ticket. We're predominantly a Ruby shop (and cRuby at that) but were getting the same timeouts described here. It turns out that the concurrent-ruby gem includes a jar file in its install, and that jar was causing us to hit search.maven.org even though everything else being scanned was not Java. So, be sure to turn on -d debug output before deciding this can't be what's causing your problem.
One comment regarding this issue. The logic to try to get missing properties from maven is a good solution, but shall perhaps be considered as a workaround to a guilty software not properly identifying itself. Thus, would it be possible to have an option to activate a warning on each jar missing identification ? It could help to raise issue against each of these software and ensure that in the future every opensource component has set appropriate identification.
@WhyJee --debug
helps. Many JAR files don't correctly define the identity. The warning will overwhelm the output. That is why they are debugging messages.
@knqyf263
@WhyJee
--debug
helps. Many JAR files don't correctly define the identity. The warning will overwhelm the output. That is why they are debugging messages.
As I wrote in my comment almost all JARs in my project provide such information. However, if --debug
shows that JARs that's helpful. I will try it and check my project again.
Instead of deactivating the search would it possible to cache the search results like the database? The SHA-1 digest should never change. Trivy should only call search.maven.org
if there is a new JAR without a cached SHA-1 digest. That would also ease the rate limit problem.
Is it possible to reduce / configure the query timeout (20s seems to be too high) for external sources like maven.org? Also, it would be nice to skip query the external source if n-tries failed before for an current scan process.
We run Trivy through GH action aquasecurity/trivy-action@0.8.0
. The action does not have an input for offline-scan
, but it does take trivy.yaml
file.
The debug
is working just fine, but scan.offline-scan
does not have any effect.
Is there a way to enable offline-scan
through trivy.yaml
?
# see https://aquasecurity.github.io/trivy/v0.34/docs/references/customization/config-file/
# Same as '--debug'
# Default is false
debug: true
scan:
# Same as '--offline-scan'
# Default is false
# See https://github.com/aquasecurity/trivy/issues/3421 (hitting the rate limit on https://search.maven.org/)
# and https://aquasecurity.github.io/trivy/v0.36/docs/advanced/air-gap/#run-trivy-with-skip-update-and-offline-scan-option
offline-scan: true
2023-01-17T12:13:31.7922056Z 2023-01-17T12:13:31.791Z [35mDEBUG[0m Parsing Java artifacts... {"file": "BOOT-INF/lib/tomcat-embed-core-9.0.60.jar"}
2023-01-17T12:14:31.5959831Z 2023-01-17T12:14:31.595Z [35mDEBUG[0m retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%220962197731cd69c6c2fc8572db85a7ace8ec13f4%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-17T12:14:31.9867806Z 2023-01-17T12:14:31.986Z [35mDEBUG[0m retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%220e14045220243804544ddb168cb532f4640a220c%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
@sadovnikov Here is a hint we are using as well and it is working https://github.com/aquasecurity/trivy-action/issues/190
Trivy image scan was taking less than 1 minute as of yesterday. Now suddenly it is timing out.
Settting
`--security-checks vuln
has no effect, as was suggested from output.Tried this with v0.24.2 and current latest