search

aquasecurity / trivy

Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
https://aquasecurity.github.io/trivy
Apache License 2.0
22.93k stars 2.26k forks source link

0.39.0 - Error with parsing package.json alongside yarn.lock #3970

Closed SharkMachine closed 1 year ago

SharkMachine commented 1 year ago

Description

Running trivy fs --scanners vuln --skip-dirs "node_modules,vendor" ./ on a repository with yarn.lock and package.json files fails with fatal error.

2023-04-03T10:35:11.103+0300    INFO    Vulnerability scanning is enabled
2023-04-03T10:35:11.113+0300    FATAL   filesystem scan error: scan error: scan failed: failed analysis: post analysis error: post analysis error: yarn walk error: unable to parse .: parse error: unable to parse package.json

What did you expect to happen?

Vulnerability scan doesn't fail with fatal error. The same cannot be reproduced with 0.38.3, package.json parsing was added in 0.39 (#3757).

Output of run with --debug:

2023-04-03T10:25:10.288+0300    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-04-03T10:25:10.292+0300    DEBUG   cache dir:  /home/sharkmachine/.cache/trivy
2023-04-03T10:25:10.292+0300    DEBUG   DB update was skipped because the local DB is the latest
2023-04-03T10:25:10.292+0300    DEBUG   DB Schema: 2, UpdatedAt: 2023-04-03 06:07:03.052991494 +0000 UTC, NextUpdate: 2023-04-03 12:07:03.052991094 +0000 UTC, DownloadedAt: 2023-04-03 07:04:14.573032556 +0000 UTC
2023-04-03T10:25:10.292+0300    INFO    Vulnerability scanning is enabled
2023-04-03T10:25:10.292+0300    DEBUG   Vulnerability type:  [os library]
2023-04-03T10:25:10.292+0300    DEBUG   Walk the file tree rooted at '.' in parallel
2023-04-03T10:25:10.293+0300    DEBUG   Skipping directory: node_modules
2023-04-03T10:25:10.293+0300    DEBUG   Skipping directory: vendor
2023-04-03T10:25:10.302+0300    FATAL   filesystem scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:431
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:266
  - scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:679
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        /home/runner/work/trivy/trivy/pkg/scanner/scan.go:146
  - post analysis error:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/local.Artifact.Inspect
        /home/runner/work/trivy/trivy/pkg/fanal/artifact/local/fs.go:164
  - post analysis error:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer.AnalyzerGroup.PostAnalyze
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/analyzer.go:487
  - yarn walk error:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer/language/nodejs/yarn.yarnAnalyzer.PostAnalyze
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/language/nodejs/yarn/yarn.go:72
  - unable to parse .:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer/language/nodejs/yarn.yarnAnalyzer.removeDevDependencies
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/language/nodejs/yarn/yarn.go:104
  - parse error:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer/language/nodejs/yarn.yarnAnalyzer.parsePackageJsonDependencies
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/language/nodejs/yarn/yarn.go:174
  - unable to parse package.json:
    github.com/aquasecurity/go-dep-parser/pkg/nodejs/packagejson.(*Parser).Parse
        /home/runner/go/pkg/mod/github.com/aquasecurity/go-dep-parser@v0.0.0-20230324043952-2172dc218241/pkg/nodejs/packagejson/parse.go:39

Output of trivy -v:

Version: 0.39.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-04-03 06:07:03.052991494 +0000 UTC
  NextUpdate: 2023-04-03 12:07:03.052991094 +0000 UTC
  DownloadedAt: 2023-04-03 07:04:14.573032556 +0000 UTC

Additional details (base image name, container registry info...):

The same problem can be reproduced with the Debian/Ubuntu deb package version and with the Docker image.

DmitriyLewen commented 1 year ago

Hello @SharkMachine Thanks for your report!

Looks like problem is with your package.json file which is next to the yarn.lock file. This package.json file doesn't have Name or Version field.

Can you send wrong files for investigation or check yourself?

Regards, Dmitriy

knqyf263 commented 1 year ago

@DmitriyLewen Can we change it not to fail on broken package.json files? package.json just enriches the results. We should not fail on the entire scanning. We should show warnings and keep scanning.

BumpeiShimada commented 1 year ago

The same issue occurred in our project as well. Thank you for your prompt response and I hope this issue will be fixed anytime soon.

knqyf263 commented 1 year ago

@BumpeiShimada First of all, we don't think it is our issue. We suppose your package.json has an issue. Could you share your package.json? You can remove or mask something sensitive.

BumpeiShimada commented 1 year ago

@knqyf263 Thank you for the response. Since it should not be good to disclose the file publicly, I'd like to ask you some questions beforehand.

We suppose your package.json has an issue

Is the issue that the package.json file doesn't have Name or Version field as DmitriyLewen said above? If so, is it enough to tell you whether our file has the fields? Or you need any other information?

DmitriyLewen commented 1 year ago

@BumpeiShimada You can send you the debug log and I will try to tell you what the problem is.

BumpeiShimada commented 1 year ago

@DmitriyLewen Okay. Here's the log:

2023-04-03T08:11:03.097Z    INFO    Need to update DB
2023-04-03T08:11:03.097Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2023-04-03T08:11:03.097Z    INFO    Downloading DB...
2023-04-03T08:11:04.602Z    INFO    Vulnerability scanning is enabled
2023-04-03T08:11:04.602Z    INFO    Secret scanning is enabled
2023-04-03T08:11:04.602Z    INFO    If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-04-03T08:11:04.602Z    INFO    Please see also https://aquasecurity.github.io/trivy/v0.39/docs/secret/scanning/#recommendation for faster secret detection
2023-04-03T08:11:26.647Z    FATAL   filesystem scan error: scan error: scan failed: failed analysis: post analysis error: post analysis error: yarn walk error: unable to parse .: parse error: unable to parse package.json
DmitriyLewen commented 1 year ago

@BumpeiShimada I meant logs with --debug flag. As in main message.

UPD: I checked this log. Looks like problem in Name or Version field.

hwo411 commented 1 year ago

We also experience this problem with new trivy, both "name" and "version" fields are present.

Logs:

trivy fs . --security-checks vuln --debug
2023-04-03T13:30:30.532+0300    WARN    '--security-checks' is deprecated. Use '--scanners' instead.
2023-04-03T13:30:30.535+0300    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-04-03T13:30:30.542+0300    DEBUG   cache dir:  /home/hunter/.cache/trivy
2023-04-03T13:30:30.543+0300    DEBUG   DB update was skipped because the local DB is the latest
2023-04-03T13:30:30.543+0300    DEBUG   DB Schema: 2, UpdatedAt: 2023-04-03 06:07:03.052991494 +0000 UTC, NextUpdate: 2023-04-03 12:07:03.052991094 +0000 UTC, DownloadedAt: 2023-04-03 10:22:50.946824057 +0000 UTC
2023-04-03T13:30:30.544+0300    INFO    Vulnerability scanning is enabled
2023-04-03T13:30:30.544+0300    DEBUG   Vulnerability type:  [os library]
2023-04-03T13:30:30.545+0300    DEBUG   Walk the file tree rooted at '.' in parallel
2023-04-03T13:30:30.638+0300    DEBUG   Start parent: com.google.protobuf:protobuf-parent:3.14.0
2023-04-03T13:30:30.639+0300    DEBUG   Start parent: org.apache.commons:commons-parent:42
2023-04-03T13:30:30.645+0300    DEBUG   Start parent: org.sonatype.oss:oss-parent:7
2023-04-03T13:30:30.655+0300    DEBUG   Start parent: org.sonatype.oss:oss-parent:7
2023-04-03T13:30:30.688+0300    DEBUG   Start parent: com.google.guava:guava-parent:26.0-android
2023-04-03T13:30:30.880+0300    DEBUG   Start parent: org.sonatype.oss:oss-parent:9
2023-04-03T13:30:30.913+0300    DEBUG   Start parent: org.apache:apache:18
2023-04-03T13:30:30.915+0300    DEBUG   Exit parent: org.sonatype.oss:oss-parent:7
2023-04-03T13:30:30.915+0300    DEBUG   Exit parent: org.sonatype.oss:oss-parent:7
2023-04-03T13:30:30.915+0300    DEBUG   Resolving com.google.code.findbugs:jsr305:3.0.1...
2023-04-03T13:30:30.915+0300    DEBUG   Exit parent: com.google.protobuf:protobuf-parent:3.14.0
2023-04-03T13:30:30.915+0300    DEBUG   Resolving com.google.protobuf:protobuf-bom:3.14.0...
2023-04-03T13:30:30.929+0300    DEBUG   Exit parent: org.sonatype.oss:oss-parent:9
2023-04-03T13:30:30.929+0300    DEBUG   Exit parent: com.google.guava:guava-parent:26.0-android
2023-04-03T13:30:30.941+0300    DEBUG   Exit parent: org.apache:apache:18
2023-04-03T13:30:30.942+0300    DEBUG   Exit parent: org.apache.commons:commons-parent:42
2023-04-03T13:30:30.944+0300    DEBUG   Start parent: org.sonatype.oss:oss-parent:7
2023-04-03T13:30:30.944+0300    DEBUG   Exit parent: org.sonatype.oss:oss-parent:7
2023-04-03T13:30:30.944+0300    DEBUG   Resolving org.jetbrains.kotlin:kotlin-annotations-jvm:1.3.72...
2023-04-03T13:30:30.971+0300    INFO    To collect the license information of packages in "app/modules/react-native-gpgs/package-lock.json", "npm install" needs to be performed beforehand
2023-04-03T13:30:30.974+0300    INFO    To collect the license information of packages in "functions/node_modules/minipass-sized/package-lock.json", "npm install" needs to be performed beforehand
2023-04-03T13:30:30.982+0300    INFO    To collect the license information of packages in "functions/node_modules/protobufjs/package-lock.json", "npm install" needs to be performed beforehand
2023-04-03T13:30:31.029+0300    FATAL   filesystem scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:431
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:266
  - scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:679
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        /home/runner/work/trivy/trivy/pkg/scanner/scan.go:146
  - post analysis error:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/local.Artifact.Inspect
        /home/runner/work/trivy/trivy/pkg/fanal/artifact/local/fs.go:164
  - post analysis error:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer.AnalyzerGroup.PostAnalyze
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/analyzer.go:487
  - yarn walk error:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer/language/nodejs/yarn.yarnAnalyzer.PostAnalyze
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/language/nodejs/yarn/yarn.go:72
  - unable to parse functions:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer/language/nodejs/yarn.yarnAnalyzer.removeDevDependencies
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/language/nodejs/yarn/yarn.go:104
  - parse error:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer/language/nodejs/yarn.yarnAnalyzer.parsePackageJsonDependencies
        /home/runner/work/trivy/trivy/pkg/fanal/analyzer/language/nodejs/yarn/yarn.go:174
  - unable to parse package.json:
    github.com/aquasecurity/go-dep-parser/pkg/nodejs/packagejson.(*Parser).Parse
        /home/runner/go/pkg/mod/github.com/aquasecurity/go-dep-parser@v0.0.0-20230324043952-2172dc218241/pkg/nodejs/packagejson/parse.go:39
DmitriyLewen commented 1 year ago

Hello @hwo411 Can you share your package.json file?

hwo411 commented 1 year ago

@DmitriyLewen actually, I found the cause. We have another project as subfolder and the error seem to be in it (but we ran trivy from the root folder).

The package.json in that project in subfolder indeed didn't have version and adding version solved the problem.

Thank you very much!

SharkMachine commented 1 year ago

@DmitriyLewen Thanks for the quick reply. I checked my package.json file and it was missing version field. Adding it fixes the issue.

However, I'd argue that my package.json wasn't broken as Yarn itself has been fine with it and hasn't displayed any warnings.

DmitriyLewen commented 1 year ago

@hwo411 , @SharkMachine I'm happy that I could help you.

However, I'd argue that my package.json wasn't broken as Yarn itself has been fine with it and hasn't displayed any warnings.

By default yarn createы version in `package.lock file:

➜  ~ docker run --name node --rm -it node sh

# yarn init
yarn init v1.22.19
question name: app
question version (1.0.0): 
question description: 
question entry point (index.js): 
question repository url: 
question author: 
question license (MIT): 
question private: 
success Saved package.json
Done in 4.50s.

# cat package.json
{
  "name": "app",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT"
}

That's why we thought package.json always included version field.

But we created #3972 to not stop scan when errors occur in enrichment files.

BumpeiShimada commented 1 year ago

@DmitriyLewen Thank you for the help. Adding name and version worked in our project as well.

On the other hand, since our project neither needs nor requires them, it would be very nice if v0.39.1, the version including the fix will be released soon.

Thank you very much again for your prompt reactions!

knqyf263 commented 1 year ago

v0.39.1 should address this issue.

Betriebsrat commented 1 year ago

Still running into said error with trivy 0.40.0

DmitriyLewen commented 1 year ago

Hello @Betriebsrat Can you share you yarn.lock + package.json file to check this?