guacsec / guac

GUAC aggregates software security metadata into a high fidelity graph database.
https://guac.sh
Apache License 2.0
1.29k stars 177 forks source link

Fix: jsonl files are rejected #2266

Closed robert-cronin closed 2 weeks ago

robert-cronin commented 2 weeks ago

Description of the PR

Fixes #2187

PR Checklist

pxp928 commented 2 weeks ago

made a bug report here: https://github.com/guacsec/guac/issues/2273 for the test failures

robert-cronin commented 2 weeks ago

As for CLI testing, I think it might be a bit contrived, but here is a test run for collecting a jsonl file which has two SPDX documents (one on each line):

❯ go run ./cmd/guacone/main.go collect files ./tmp/spdx_vuln.jsonl
{"level":"info","ts":1731040156.5952644,"caller":"logging/logger.go:79","msg":"Logging at info level","guac-version":"v0.0.1-custom"}
{"level":"info","ts":1731040156.5953152,"caller":"cli/init.go:65","msg":"Using config file: /home/rob/go/src/guacsec/guac/guac.yaml","guac-version":"v0.0.1-custom"}
{"level":"info","ts":1731040157.3011687,"caller":"helpers/bulk.go:47","msg":"assembling Package: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.302138,"caller":"helpers/bulk.go:63","msg":"assembling Source: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3025815,"caller":"helpers/bulk.go:73","msg":"assembling Artifact: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3030267,"caller":"helpers/bulk.go:88","msg":"assembling Materials (Artifact): 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3033898,"caller":"helpers/bulk.go:97","msg":"assembling Builder: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.303743,"caller":"helpers/bulk.go:106","msg":"assembling Vulnerability: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3041625,"caller":"helpers/bulk.go:115","msg":"assembling Licenses: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304605,"caller":"helpers/bulk.go:122","msg":"assembling CertifyScorecard: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3046224,"caller":"helpers/bulk.go:128","msg":"assembling IsDependency: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304633,"caller":"helpers/bulk.go:137","msg":"assembling IsOccurrence: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304642,"caller":"helpers/bulk.go:146","msg":"assembling HasSLSA: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3046508,"caller":"helpers/bulk.go:152","msg":"assembling CertifyVuln: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304659,"caller":"helpers/bulk.go:158","msg":"assembling VulnMetadata: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3046675,"caller":"helpers/bulk.go:164","msg":"assembling VulnEqual: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3046749,"caller":"helpers/bulk.go:170","msg":"assembling HasSourceAt: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3046834,"caller":"helpers/bulk.go:176","msg":"assembling CertifyBad: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304692,"caller":"helpers/bulk.go:182","msg":"assembling CertifyGood: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3047001,"caller":"helpers/bulk.go:188","msg":"assembling PointOfContact: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3047082,"caller":"helpers/bulk.go:194","msg":"assembling HasMetadata: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304717,"caller":"helpers/bulk.go:200","msg":"assembling HasSBOM: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3047254,"caller":"helpers/bulk.go:211","msg":"assembling VEX : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3047338,"caller":"helpers/bulk.go:217","msg":"assembling HashEqual : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.304741,"caller":"helpers/bulk.go:223","msg":"assembling PkgEqual : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.3047488,"caller":"helpers/bulk.go:229","msg":"assembling CertifyLegal : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.32608,"caller":"helpers/bulk.go:47","msg":"assembling Package: 3981","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.6908877,"caller":"helpers/bulk.go:63","msg":"assembling Source: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.6944394,"caller":"helpers/bulk.go:73","msg":"assembling Artifact: 3384","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.783583,"caller":"helpers/bulk.go:88","msg":"assembling Materials (Artifact): 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.7843251,"caller":"helpers/bulk.go:97","msg":"assembling Builder: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.784722,"caller":"helpers/bulk.go:106","msg":"assembling Vulnerability: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.7852316,"caller":"helpers/bulk.go:115","msg":"assembling Licenses: 29","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.7868395,"caller":"helpers/bulk.go:122","msg":"assembling CertifyScorecard: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040157.7868674,"caller":"helpers/bulk.go:128","msg":"assembling IsDependency: 7161","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.2391815,"caller":"helpers/bulk.go:137","msg":"assembling IsOccurrence: 3878","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.849252,"caller":"helpers/bulk.go:146","msg":"assembling HasSLSA: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.849305,"caller":"helpers/bulk.go:152","msg":"assembling CertifyVuln: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.8493211,"caller":"helpers/bulk.go:158","msg":"assembling VulnMetadata: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.8493342,"caller":"helpers/bulk.go:164","msg":"assembling VulnEqual: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.849344,"caller":"helpers/bulk.go:170","msg":"assembling HasSourceAt: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.8493533,"caller":"helpers/bulk.go:176","msg":"assembling CertifyBad: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.8493662,"caller":"helpers/bulk.go:182","msg":"assembling CertifyGood: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.8493824,"caller":"helpers/bulk.go:188","msg":"assembling PointOfContact: 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.8493936,"caller":"helpers/bulk.go:194","msg":"assembling HasMetadata: 462","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.9131582,"caller":"helpers/bulk.go:200","msg":"assembling HasSBOM: 1","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.9971635,"caller":"helpers/bulk.go:211","msg":"assembling VEX : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.9972134,"caller":"helpers/bulk.go:217","msg":"assembling HashEqual : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.9972332,"caller":"helpers/bulk.go:223","msg":"assembling PkgEqual : 0","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040159.9972486,"caller":"helpers/bulk.go:229","msg":"assembling CertifyLegal : 125","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040160.0336263,"caller":"ingestor/ingestor.go:79","msg":"[3.432688836s] completed doc {Collector:FileCollector Source:file:///./tmp/spdx_vuln.jsonl DocumentRef:sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241}","guac-version":"v0.0.1-custom","documentHash":"sha256_3245201bb639af7efb54de2f3bcabab3c7c08d69e538d789d70142844ebc7241"}
{"level":"info","ts":1731040160.0336566,"caller":"cmd/files.go:155","msg":"collector ended gracefully","guac-version":"v0.0.1-custom"}
{"level":"info","ts":1731040160.0336747,"caller":"cmd/files.go:170","msg":"completed ingesting 1 documents of 1","guac-version":"v0.0.1-custom"}