search

aquasecurity / trivy

Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
https://aquasecurity.github.io/trivy
Apache License 2.0
23.34k stars 2.31k forks source link

trivy image scan suddenly timing out #3421

Closed stealthrabbi closed 1 year ago

stealthrabbi commented 1 year ago

Trivy image scan was taking less than 1 minute as of yesterday. Now suddenly it is timing out.

Settting `--security-checks vuln has no effect, as was suggested from output.

Tried this with v0.24.2 and current latest

2023-01-12T12:46:56.932Z    INFO    Need to update DB
2023-01-12T12:46:56.932Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2023-01-12T12:46:56.932Z    INFO    Downloading DB...
10.64 MiB / 36.04 MiB [------------------>__________________________________________] 29.53% ? p/s ?23.69 MiB / 36.04 MiB [---------------------------------------->____________________] 65.75% ? p/s ?36.04 MiB / 36.04 MiB [----------------------------------------------------------->] 100.00% ? p/s ?36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 42.35 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 42.35 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 42.35 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 39.62 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 39.62 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 39.62 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 37.07 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 37.07 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 37.07 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [---------------------------------------------->] 100.00% 34.67 MiB p/s ETA 0s36.04 MiB / 36.04 MiB [-------------------------------------------------] 100.00% 14.56 MiB p/s 2.7s2023-01-12T12:47:00.337Z    INFO    Vulnerability scanning is enabled
2023-01-12T12:56:56.932Z    WARN    Increase --timeout value
2023-01-12T12:56:56.932Z    FATAL   image scan error: scan error: scan failed: failed analysis: analyze error: timeout: context deadline exceeded
Hugo-D commented 1 year ago

Hello, I am having the same issue there. Worked perfectly fine this morning until the Vulnerability DB updates:

trivy -v
Version: 0.33.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-01-12 06:07:03.840852771 +0000 UTC
  NextUpdate: 2023-01-12 12:07:03.840852271 +0000 UTC
  DownloadedAt: 2023-01-12 06:41:12.347924267 +0000 UTC
dusik66 commented 1 year ago

Hello,

maybe it is related to search.maven.org: 2023-01-12T12:55:15.461Z DEBUG request failed {"error": "Get \"https://search.maven.org/solrsearch/select?q=1%3A%22185ae7422deb18aa14464b18cf9ce997d4710a57%22&rows=1&wt=json\": net/http: TLS handshake timeout", "method": "GET", "url": "https://search.maven.org/solrsearch/select?q=1%3A%22185ae7422deb18aa14464b18cf9ce997d4710a57%22&rows=1&wt=json"}

knqyf263 commented 1 year ago

Looks like GitHub Container Registry distributing Trivy DB is something wrong now. Please try it again later.

aaron-hastings-travelport commented 1 year ago

Seeing the same issue with Trivy 0.32.1.

knqyf263 commented 1 year ago

Is everyone scanning images with JAR files? If so, maven might be relevant like @dusik66 pointed out. --offline-scan may help until the server is back.

stealthrabbi commented 1 year ago

In my case, yes, I am scanning an image for a java proj.

knqyf263 commented 1 year ago

Can you try --offline-scan then? It doesn't try to connect to search.maven.org.

pioneer2k commented 1 year ago

I reported the same issue here: https://github.com/aquasecurity/trivy-action/issues/190

DaspawnW commented 1 year ago

If you run with --offline-scan then you even get additional features ;-)

In addition, if you want to scan Java dependencies such as JAR and pom.xml, you need to specify --offline-scan since Trivy tries to issue API requests for scanning Java applications by default.

https://aquasecurity.github.io/trivy/v0.36/docs/advanced/air-gap/

stealthrabbi commented 1 year ago

so --offline-scan won't prevent the main trivy DB from being downloaded, it just will not reach out to internet for the actual scanning portion?

knqyf263 commented 1 year ago

so --offline-scan won't prevent the main trivy DB from being downloaded, it just will not reach out to internet for the actual scanning portion?

Right.

knqyf263 commented 1 year ago

To skip downloading DB, you can pass --skip-db-update, but you need to have the old database locally. Otherwise, the scanning will fail.

dmitry-weirdo commented 1 year ago

We are probably facing a similar issue — starting from today (the first fail was at Jan 12, 2023, 11:30 AM GMT), the GitLab pipelines started to fail.

$ trivy --cache-dir .trivycache/ image --exit-code 0 --no-progress --format template --template "@/contrib/junit.tpl" --output "$CI_PROJECT_DIR/container-scan-junit.xml" "$FULL_IMAGE_NAME" --timeout 30m0s
2023-01-12T18:01:31.130Z    INFO    Need to update DB
2023-01-12T18:01:31.130Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2023-01-12T18:01:31.130Z    INFO    Downloading DB...
2023-01-12T18:01:33.467Z    INFO    Vulnerability scanning is enabled
2023-01-12T18:01:33.467Z    INFO    Secret scanning is enabled
2023-01-12T18:01:33.467Z    INFO    If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-12T18:01:33.467Z    INFO    Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-12T18:06:55.053Z    FATAL   image scan error: scan error: scan failed: failed analysis: analyze error: failed to analyze layer:  : walk error: failed to process the file: failed to analyze file: failed to analyze app/BOOT-INF/lib/log4j-core-2.19.0.jar: unable to open app/BOOT-INF/lib/log4j-core-2.19.0.jar: failed to open: unable to read the file: read tcp 172.17.0.2:47734->3.5.139.107:443: read: connection reset by peer
Uploading artifacts for failed job

Setting --offline-scan option is helping, but I am not sure whether it's a good long-term solution.

dmitry-weirdo commented 1 year ago

What happens if we set --offline-scan? The documentation is not very clear about it. If it stops to refresh the CVE database, this isn't the best constant solution.

thiagosf-hotmart commented 1 year ago

+1

caiohasouza commented 1 year ago

+1

knqyf263 commented 1 year ago

@dmitry-weirdo Please see my comment above. https://github.com/aquasecurity/trivy/issues/3421#issuecomment-1380380681

Ref. https://aquasecurity.github.io/trivy/v0.36/docs/references/troubleshooting/#maven-rate-limiting-inconsistent-jar-vulnerability-reporting

nelsonneto-hotmart commented 1 year ago

+1

paultung commented 1 year ago

Still happening in my case and works after retrying 3 times:

...
2023-01-13T02:20:42.699Z    INFO    Need to update DB
2023-01-13T02:20:42.699Z    INFO    Downloading DB...
16.89 MiB / 36.05 MiB [---------------------------->________________________________] 46.86% ? p/s ?
32.95 MiB / 36.05 MiB [------------------------------------------------------->_____] 91.42% ? p/s ?
36.05 MiB / 36.05 MiB [----------------------------------------------------------->] 100.00% ? p/s ?
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 32.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 32.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 32.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.03 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.03 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.03 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 28.10 MiB p/s ETA 0s
36.05 MiB / 36.05 MiB [-------------------------------------------------] 100.00% 19.06 MiB p/s 2.1s
2023-01-13T02:25:42.822Z    FATAL   scan error: image scan failed: failed analysis: analyze error: timeout: context deadline exceeded
...
DmitriyLewen commented 1 year ago

Hello guys!!

I checked this problem now. Looks like it was temporal down and it works now. I tried to reproduce this problem, but got correct response from maven 1000 times in a row. Can you check it and write if there are still errors?

@paultung may be your problem has different reason. Can you send your jar file for more investigate?

Dmitry-Dymnov commented 1 year ago

Hello, the problem still exists.

2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:385]: {
  "uuid": "bf9b6744-ebbc-11eb-bcb0-fef69661a874",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://registry-harbor-trivy:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.32.1",
  "create_time": "2021-07-23T13:49:06.678883Z",
  "update_time": "2021-07-27T11:34:11.007996Z"
}
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:385]: {
  "registry": {
    "url": "http://registry-harbor-core:80",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 135,
    "repository": "*******",
    "tag": "latest",
    "digest": "sha256:3cd2b01db4c4299f6bddcf1b93d6bf51f0953dd1998f295cccc56f34c4ff486c",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2023-01-13T06:34:39Z [INFO] [/pkg/scan/job.go:222]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2023-01-13T06:34:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:34:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:34:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:34:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:11Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:16Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:21Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:26Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:31Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:36Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:41Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:46Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:51Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:35:56Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:36:01Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:36:06Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-01-13T06:36:11Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2023-01-13T06:34:40.074Z INFO   Vulnerability scanning is enabled
2023-01-13T06:36:06.461Z    FATAL  image scan error: scan error: scan failed: failed analysis: analyze error: failed to analyze layer:  : walk error: failed to process the file: failed to analyze file: failed to analyze liquibase/internal/lib/ojdbc8.jar: unable to open liquibase/internal/lib/ojdbc8.jar: failed to open: unable to read the file: unexpected EOF
: general response handler: unexpected status code: 500, expected: 200
DmitriyLewen commented 1 year ago

Hello @Dmitry-Dymnov Looks like problem is not with maven repository. Trivy can't read liquibase/internal/lib/ojdbc8.jar file in your liquibase***.jar(am not sure of file name). Can you submit this file for investigation?

Also can you say version of Trivy? We fixed case where Trivy would stop scanning after receiving error for internal jars.(https://github.com/aquasecurity/trivy/pull/2989)

Regards, Dmitriy

Dmitry-Dymnov commented 1 year ago

@DmitriyLewen, until yesterday, these images were scanned without problems. I'm sorry, but I can't give you the file ((( Trivy used as part of the harbor. ("version": "v0.32.1")

monwolf commented 1 year ago

Hello guys!!

I checked this problem now. Looks like it was temporal down and it works now. I tried to reproduce this problem, but got correct response from maven 1000 times in a row. Can you check it and write if there are still errors?

@paultung may be your problem has different reason. Can you send your jar file for more investigate?

I've just run a trivy execution and it's failing again 

+ /opt/trivy-scan/trivy -d --cache-dir /opt/trivy-scan/.trivycache/ image --no-progress --format json -o /tmp/reports/aea-chapi-omnichannel-trivy-report-1673595621.json aea-tp/channel:alpha-91d20501-91340-241 --ignorefile /tmp/.trivyignore
2023-01-13T07:40:21.965Z    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-01-13T07:40:21.970Z    DEBUG   cache dir:  /opt/trivy-scan/.trivycache/
2023-01-13T07:40:21.970Z    DEBUG   There is no valid metadata file: unable to open a file: open /opt/trivy-scan/.trivycache/db/metadata.json: no such file or directory
2023-01-13T07:40:21.970Z    INFO    Need to update DB
2023-01-13T07:40:21.970Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2023-01-13T07:40:21.970Z    INFO    Downloading DB...
2023-01-13T07:40:21.970Z    DEBUG   no metadata file
2023-01-13T07:40:24.300Z    DEBUG   Updating database metadata...
2023-01-13T07:40:24.301Z    DEBUG   DB Schema: 2, UpdatedAt: 2023-01-13 06:07:02.14805908 +0000 UTC, NextUpdate: 2023-01-13 12:07:02.14805848 +0000 UTC, DownloadedAt: 2023-01-13 07:40:24.301082232 +0000 UTC
2023-01-13T07:40:24.301Z    INFO    Vulnerability scanning is enabled
2023-01-13T07:40:24.302Z    DEBUG   Vulnerability type:  [os library]
2023-01-13T07:40:24.302Z    INFO    Secret scanning is enabled
2023-01-13T07:40:24.302Z    INFO    If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-13T07:40:24.302Z    INFO    Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-13T07:40:24.319Z    DEBUG   No secret config detected: trivy-secret.yaml
2023-01-13T07:40:24.320Z    DEBUG   Image ID: sha256:e0ea4be578c1e55dfc047dc4621a3c8e460cf2cc080dc2c56c36d08858ca80ba
2023-01-13T07:40:24.320Z    DEBUG   Diff IDs: [sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3 sha256:9dc38e038a4b30a44a39340027b138e3d52f5256e5449ce32b74f7f21475a796 sha256:0c6a8c08d9664f51203017fe0dd63a0ad69a0da2fc6cb01bd021314ae5bdecd3 sha256:29e51557cb02f590cef20e0a9ba769e030c27e8bd42076d42f2acaa49cfd5f38 sha256:13a14e21710e57c5e06cfa617638216e95b066f0b3f38663fed6a85563e89dce sha256:916158f15540724646ab7f924d27542c47aa1ca392333c03b1d19386793f9a3f sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef sha256:df59f20536381fbafd8002f10f4d38a62516da52d6f0e610341a7426ebe99f4f sha256:c9ec7b8d4aea08c7884108ce8ab2dd96883998b57f1ebb33af014149956c396d sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef]
2023-01-13T07:40:24.320Z    DEBUG   Base Layers: [sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3]
2023-01-13T07:40:24.320Z    DEBUG   Missing image ID in cache: sha256:e0ea4be578c1e55dfc047dc4621a3c8e460cf2cc080dc2c56c36d08858ca80ba
2023-01-13T07:40:24.320Z    DEBUG   Missing diff ID in cache: sha256:13a14e21710e57c5e06cfa617638216e95b066f0b3f38663fed6a85563e89dce
2023-01-13T07:40:24.320Z    DEBUG   Missing diff ID in cache: sha256:9dc38e038a4b30a44a39340027b138e3d52f5256e5449ce32b74f7f21475a796
2023-01-13T07:40:24.320Z    DEBUG   Missing diff ID in cache: sha256:6515074984c6f8bb1b8a9962c8fb5f310fc85e70b04c88442a3939c026dbfad3
2023-01-13T07:40:24.320Z    DEBUG   Missing diff ID in cache: sha256:29e51557cb02f590cef20e0a9ba769e030c27e8bd42076d42f2acaa49cfd5f38
2023-01-13T07:40:24.320Z    DEBUG   Missing diff ID in cache: sha256:0c6a8c08d9664f51203017fe0dd63a0ad69a0da2fc6cb01bd021314ae5bdecd3
2023-01-13T07:40:26.786Z    DEBUG   Missing diff ID in cache: sha256:916158f15540724646ab7f924d27542c47aa1ca392333c03b1d19386793f9a3f
2023-01-13T07:40:26.855Z    DEBUG   Missing diff ID in cache: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
2023-01-13T07:40:26.859Z    DEBUG   Missing diff ID in cache: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
2023-01-13T07:40:26.862Z    DEBUG   Missing diff ID in cache: sha256:df59f20536381fbafd8002f10f4d38a62516da52d6f0e610341a7426ebe99f4f
2023-01-13T07:40:27.125Z    DEBUG   Parsing Java artifacts...   {"file": "home/docker/app/app.jar"}
2023-01-13T07:40:27.127Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/logback-access-spring-boot-starter-2.7.1.jar"}
2023-01-13T07:40:27.127Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/tomcat-embed-websocket-9.0.65.jar"}
2023-01-13T07:40:27.241Z    DEBUG   Missing diff ID in cache: sha256:c9ec7b8d4aea08c7884108ce8ab2dd96883998b57f1ebb33af014149956c396d
2023-01-13T07:40:27.305Z    DEBUG   Missing diff ID in cache: sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
2023-01-13T07:40:29.355Z    DEBUG   Parsing Java artifacts...   {"file": "opt/java/openjdk/lib/jrt-fs.jar"}
2023-01-13T07:40:29.792Z    DEBUG   No such POM in the central repositories {"file": "jrt-fs.jar"}
2023-01-13T07:41:27.790Z    DEBUG   retrying request    {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22javax.websocket%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-13T07:42:47.932Z    DEBUG   retrying request    {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%22bd70dfeb39cc83c6934be24fa377b21e541dbe76%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-13T07:43:08.005Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/tomcat-embed-core-9.0.31.jar"}
2023-01-13T07:43:08.152Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/logback-access-1.2.11.jar"}
2023-01-13T07:43:08.152Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/logstash-logback-encoder-4.11.jar"}
2023-01-13T07:43:08.153Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/logback-core-1.2.11.jar"}
2023-01-13T07:43:08.153Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/spring-security-oauth2-autoconfigure-2.6.8.jar"}
2023-01-13T07:44:08.220Z    DEBUG   retrying request    {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%22d9ce52f45c4689c1f445d08550ed7c4adcb920e3%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-13T07:44:28.292Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/spring-security-oauth2-2.5.2.RELEASE.jar"}
2023-01-13T07:44:28.292Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/springdoc-openapi-ui-1.6.6.jar"}
2023-01-13T07:44:28.292Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/springdoc-openapi-webmvc-core-1.6.6.jar"}
2023-01-13T07:44:28.293Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/spring-webmvc-5.3.22.jar"}
2023-01-13T07:44:28.366Z    DEBUG   Parsing Java artifacts...   {"file": "BOOT-INF/lib/spring-boot-actuator-autoconfigure-2.7.3.jar"}
2023-01-13T07:45:21.991Z    WARN    Increase --timeout value
2023-01-13T07:45:21.991Z    FATAL   image scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run

        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:397

  - scan error:

    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact

        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:257

  - scan failed:

    github.com/aquasecurity/trivy/pkg/commands/artifact.scan

        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:588

  - failed analysis:

    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact

        /home/runner/work/trivy/trivy/pkg/scanner/scan.go:146

  - analyze error:

    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.Inspect

        /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:127

  - timeout:

    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspect

        /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:248

  - context deadline exceeded
michael-mader commented 1 year ago

seems like search.maven.org is still unstable or has a more aggressive rate limiting. After some requests (where the response is really fast) I get "504 Gateway Timeout".

What is search.maven.org used for? Only for checking whether an update fixing a vulnerability of maven dependency is available?

mbreevoort commented 1 year ago

with the latest trivy image no problems anymore: docker.io/aquasec/trivy With option --vuln-type os https://hub.tekton.dev/tekton/task/trivy-scanner uses docker.io/aquasec/trivy@sha256:dea76d4b50c75125cada676a87ac23de2b7ba4374752c6f908253c3b839201d9 and will hang until timeout with option --vuln-type os So probably a version problem? Because no jar scan is with search.maven.org is enabled ....

DmitriyLewen commented 1 year ago

until yesterday, these images were scanned without problems.

This is so strange. In this case Trivy doesn't go to jar parse logic. Can you open new issue? I think this is different question.

DmitriyLewen commented 1 year ago

seems like search.maven.org is still unstable

It looks like you are right. I am playing with maven repository REST API. I get 504 error intermittently. But it can happen on the first request. It doesn't seem to be related to number of requests.

What is search.maven.org used for?

If your jar file doesn't have information about GroupID, ArtifactID and Version in pom.properties or MANIFEST.MF then: We use maven repository to try to get GroupID, ArtifactID and Version by SHA-1 digest or get GroupID by artifactID(artifactID got from file name)

abelsromero commented 1 year ago

If your jar file doesn't have information about GroupID, ArtifactID and Version in pom.properties or MANIFEST.MF then: We use maven repository to try to get GroupID, ArtifactID and Version by SHA-1 digest or get GroupID by artifactID(artifactID got from file name)

I don't this is 100% accurate, it is also used to complement the db, docs state in bold

image

And I just confirmed a couple of CVE that are not reported when running with --offline-scan.

Imo --offline-scan involves some risk.

Moeth commented 1 year ago

Same Problem here - with activated debug Option, we get

docker run --rm --entrypoint trivy aquasec/trivy image webapp --timeout 30m --security-checks vuln -d

2023-01-13T08:53:45.739Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%225dd2c405ef14507e909cf6537f0e614328fb1af4%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:45.997Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%229c15159c5e1a8f6dd5381fadf7b0dbeb49f2860e%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:46.786Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22Apache+Tomcat%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:46.786Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22javax.annotation%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5} 2023-01-13T08:53:46.797Z DEBUG retrying request {"request": "GET https://search.maven.org/solrsearch/select?q=g%3A%22org.apache%22+AND+a%3A%22Apache+Tomcat%22&rows=1 (status: 504)", "timeout": "20s", "remaining": 5}

but opening the urls in browser works. Are there any request-header which make this broken?

dannykruitbosch commented 1 year ago

I'm also having these issues with trivy 0.36.1. Just to add some extra info, here's our situation:

We only have this issue in our (bitbucket) ci/cd pipeline. Locally on my laptop, this doesn't seem to be an issue. Also we do two scans in out build pipeline, fs and image. The fs scan runs fine and scans all our dependencies as expected. But the image scan times out on getting the info from search.maven.org

DmitriyLewen commented 1 year ago

but opening the urls in browser works. Are there any request-header which make this broken?

Links don't always open. Your link opened 1 out of 2 times for me. image

DmitriyLewen commented 1 year ago

And I just confirmed a couple of CVE that are not reported when running with --offline-scan. Imo --offline-scan involves some risk.

This is because Trivy cannot get dependency information from jar file (from pom.properties and MANIFEST.MF) and from maven repository (get 504 error).

Hugo-D commented 1 year ago

A quick request to https://search.maven.org from a browser gives you a 504 Gateway Timeout error When the website was still available but the searches were already failing yesterday, I saw a banner on top saying their new search repository will be at https://central.sonatype.dev/ So Maven may have moved the search services to this new domain which, as I've tried a few simple searches on it, seems to work fine.

DmitriyLewen commented 1 year ago

Thanks for information @Hugo-D !

But their REST API guide still uses https://search.maven.org/solrsearch url (https://central.sonatype.org/search/rest-api-guide/)

I tried to change search.maven.org/solrsearch/search.maven.org to central.sonatype.dev/, but got 404 error...

anderssonw commented 1 year ago

Same issue here. We use GitHub Workflows and the latest trivy-action version 0.8.

durcon commented 1 year ago

The problem still exists:

2023-01-13T09:45:51.883Z    INFO    Downloading DB...
13.36 MiB / 36.05 MiB [---------------------->______________________________________] 37.05% ? p/s ?33.86 MiB / 36.05 MiB [--------------------------------------------------------->___] 93.91% ? p/s ?36.05 MiB / 36.05 MiB [----------------------------------------------------------->] 100.00% ? p/s ?36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 37.78 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 37.78 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 37.78 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 35.35 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 35.35 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 35.35 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 33.07 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 33.07 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 33.07 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [---------------------------------------------->] 100.00% 30.93 MiB p/s ETA 0s36.05 MiB / 36.05 MiB [-------------------------------------------------] 100.00% 14.16 MiB p/s 2.7s2023-01-13T09:45:55.467Z    INFO    Vulnerability scanning is enabled
2023-01-13T09:45:55.468Z    INFO    Secret scanning is enabled
2023-01-13T09:45:55.468Z    INFO    If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-13T09:45:55.468Z    INFO    Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-13T09:50:51.978Z    WARN    Increase --timeout value
2023-01-13T09:50:51.978Z    FATAL   image scan error: scan error: scan failed: failed analysis: analyze error: timeout: context deadline exceeded

After using --offline-scan the error disappeared.

@DmitriyLewen

If your jar file doesn't have information about GroupID, ArtifactID and Version in pom.properties or MANIFEST.MF then: We use maven repository to try to get GroupID, ArtifactID and Version by SHA-1 digest or get GroupID by artifactID(artifactID got from file name)

I understand it, but is there no way to save this information in the database? Maven creates pom.properties by default, there shouldn't be much JAR files without it. Same for MANIFEST.MF. In a lot of cases there is also the pom.xml in den JAR.

Maybe it would help if Trivy logs the JARs which are not containing theses files. Then we can see if there is an easier solution.

DmitriyLewen commented 1 year ago

In a lot of cases there is also the pom.xml in den JAR.

I'm not sure there is resource with a list of jars that we can use to save to the database.

That's why we use maven repository to analyze similar jars.

durcon commented 1 year ago

@DmitriyLewen

In a lot of cases there is also the pom.xml in den JAR.

I'm not sure there is resource with a list of jars that we can use to save to the database.

That's why we use maven repository to analyze similar jars.

I understand and I don't know it, too.

I think logging the JARs would help to see if there is an issue. I checked one of my projects and almost all JARs have one of these three files. Maybe there is some issue with finding these files. Logs would be helpful to improve it.

danielefranceschi commented 1 year ago

Seems like maven.org has issues: https://status.maven.org/incidents/zs6lhc4nykst

DmitriyLewen commented 1 year ago

I think logging the JARs would help to see if there is an issue

Trivy does not use a server for scanning (all scans are done on your ЗС), so saving logs is currently not possible. Also, if we get some information from each scan - users will have a lot of questions about this, because users scan their personal files and etc. @knqyf263 I am right?

durcon commented 1 year ago

I think logging the JARs would help to see if there is an issue

Trivy does not use a server for scanning (all scans are done on your ЗС), so saving logs is currently not possible. Also, if we get some information from each scan - users will have a lot of questions about this, because users scan their personal files and etc. @knqyf263 I am right?

Sure, I meant my own logs. You could provide debug logs (enabled by a flag), so Java developers can see, which JARs are not containing these files. It helps to find problems.

If I see a JAR, which has one of these three files I could file a feature request to improve Trivy. Right know I don't know how much JARs have this problem and which ones. But I see in my project that almost all JARs have these files.

lintu22 commented 1 year ago

https://status.maven.org/incidents/mblzqd69jjpz this looks relevant to the issues people are seeing here

andytinkham commented 1 year ago

One note to others looking at this ticket. We're predominantly a Ruby shop (and cRuby at that) but were getting the same timeouts described here. It turns out that the concurrent-ruby gem includes a jar file in its install, and that jar was causing us to hit search.maven.org even though everything else being scanned was not Java. So, be sure to turn on -d debug output before deciding this can't be what's causing your problem.

WhyJee commented 1 year ago

One comment regarding this issue. The logic to try to get missing properties from maven is a good solution, but shall perhaps be considered as a workaround to a guilty software not properly identifying itself. Thus, would it be possible to have an option to activate a warning on each jar missing identification ? It could help to raise issue against each of these software and ensure that in the future every opensource component has set appropriate identification.

knqyf263 commented 1 year ago

@WhyJee --debug helps. Many JAR files don't correctly define the identity. The warning will overwhelm the output. That is why they are debugging messages.

durcon commented 1 year ago

@knqyf263

@WhyJee --debug helps. Many JAR files don't correctly define the identity. The warning will overwhelm the output. That is why they are debugging messages.

As I wrote in my comment almost all JARs in my project provide such information. However, if --debug shows that JARs that's helpful. I will try it and check my project again.

Instead of deactivating the search would it possible to cache the search results like the database? The SHA-1 digest should never change. Trivy should only call search.maven.org if there is a new JAR without a cached SHA-1 digest. That would also ease the rate limit problem.

sizowie commented 1 year ago

Is it possible to reduce / configure the query timeout (20s seems to be too high) for external sources like maven.org? Also, it would be nice to skip query the external source if n-tries failed before for an current scan process.

sadovnikov commented 1 year ago

We run Trivy through GH action aquasecurity/trivy-action@0.8.0. The action does not have an input for offline-scan, but it does take trivy.yaml file.

The debug is working just fine, but scan.offline-scan does not have any effect.

Is there a way to enable offline-scan through trivy.yaml? 

# see https://aquasecurity.github.io/trivy/v0.34/docs/references/customization/config-file/

# Same as '--debug'
# Default is false
debug: true

scan:
  # Same as '--offline-scan'
  # Default is false
  # See https://github.com/aquasecurity/trivy/issues/3421 (hitting the rate limit on https://search.maven.org/)
  # and https://aquasecurity.github.io/trivy/v0.36/docs/advanced/air-gap/#run-trivy-with-skip-update-and-offline-scan-option
  offline-scan: true
2023-01-17T12:13:31.7922056Z 2023-01-17T12:13:31.791Z   DEBUG  Parsing Java artifacts...   {"file": "BOOT-INF/lib/tomcat-embed-core-9.0.60.jar"}
2023-01-17T12:14:31.5959831Z 2023-01-17T12:14:31.595Z   DEBUG  retrying request    {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%220962197731cd69c6c2fc8572db85a7ace8ec13f4%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
2023-01-17T12:14:31.9867806Z 2023-01-17T12:14:31.986Z   DEBUG  retrying request    {"request": "GET https://search.maven.org/solrsearch/select?q=1%3A%220e14045220243804544ddb168cb532f4640a220c%22&rows=1&wt=json (status: 504)", "timeout": "20s", "remaining": 5}
pioneer2k commented 1 year ago

@sadovnikov Here is a hint we are using as well and it is working https://github.com/aquasecurity/trivy-action/issues/190