Closed ckotzbauer closed 1 year ago
Hi @ckotzbauer -- it looks like the only differences are the element IDs likely from the related files section, is that right? Is this a diff from 0.55.0
to 0.56.0
? There was a change to sort the files in order to make the output more consistent, I suspect the updated version is what you might want to use for your test fixtures moving forward. Or am I missing something?
it looks like the only differences are the element IDs likely from the related files section, is that right?
Yes, this should be right.
Is this a diff from 0.55.0 to 0.56.0?
No this is a diff between my go-implementation (linked above) which uses 0.56.0 in a PR and the 0.56.0 cli.
I suspect the updated version is what you might want to use for your test fixtures moving forward. Or am I missing something?
The fixtures were updated on my machine locally, not pushed yet. But the diff was created from up-to-date fixtures (generated by 0.56.0 syft cli)
Ok, so you generate SBOMs using the command line syft
at version 0.56.0
and then update syft as a library in your sbom-operator
project to the same version and then generate SBOMs with the code you linked to and then compare those outputs. And currently syft-json
format results in the same files but spdx-json
has the diff you provided. Am I understanding this correctly?
If this gets run multiple times does it always result in the exact same diff? I suspect there may be a sorting issue specific to SPDX, probably nothing you need to change.
Ok, so you generate SBOMs using the command line syft at version 0.56.0 and then update syft as a library in your sbom-operator project to the same version and then generate SBOMs with the code you linked to and then compare those outputs. And currently syft-json format results in the same files but spdx-json has the diff you provided. Am I understanding this correctly?
Exactly.
If this gets run multiple times does it always result in the exact same diff? I suspect there may be a sorting issue specific to SPDX, probably nothing you need to change.
I have to check this, will reply.
@kzantow When running my go-code and the cli multiple times, the ordering always differs.
@ckotzbauer thanks for getting back to me -- that is what I suspected, I think I have a fix for this. Are you able to test changes from a Syft PR somehow?
yes, that should be possible :+1:
@ckotzbauer I've created a PR here: https://github.com/anchore/syft/pull/1216 ... to be frank, I'm not quite sure yet how to add a meaningful test for this yet so it might take just a bit to get it merged, but it would be nice to know if it seems to solve your problem
great, I will test it with my code.
The PR fixed the issue :heavy_check_mark:. My tests are green again. Tested the snapshotted cli against my go-code which also used the PR-code.
I'm not quite sure yet how to add a meaningful test for this yet
You can also try to test the code against a static fixture. When the issue occurrs again, this will fail.
Right, my concern is that we have some static fixtures that have been routinely passing... I'm not sure if they had sufficient data to exhibit the problem. Thanks for following up 👍
Thanks for the fast PR!
I went ahead and published a new release with the fix here, please let me know if it doesn't work! https://github.com/anchore/syft/releases/tag/v0.57.0
Hi @kzantow,
I discovered this issue again. The artifact-relationships are not stable between two runs for the same image.
So running syft registry:alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 -o spdx-json
twice will produce different IDs in the relationship section.
It worked with 0.58.0 and failed for the first time with 0.59.0, but it is still present in 0.62.0
Is this still an issue or "intented" behaviour?
@ckotzbauer -- the SPDX IDs should be stable, I've reopened this to investigate. We did just recently (v0.61.0) rework the SPDX output to exclusively use the spdx/tools-golang
library, so it's interesting to hear that this was unstable before this, in v0.59.0 and continues to be. Thanks for the info!
@ckotzbauer -- there was a pretty obvious omission in the aforementioned refactor, so this will be fixed with PR #1350 and I've adjusted the tests to hopefully surface this more obviously in the future.
Thanks @kzantow for your very quick reply!!
@ckotzbauer -- a new Syft release v0.62.1
has been published which includes this fix -- please let me know if you continue to have issue! Again, sorry for the inconvenience -- the updates to the tests should help catch this regression in the future.
Great. I can confirm, that the issue is now gone. Thanks for your work! :partying_face:
What happened: Hi all, I'm the maintainer of the https://github.com/ckotzbauer/sbom-operator project. Syft is integrated there via its golang api. This is unit-tested to ensure, that my code and the cli of the same version produce the same SBOM for images. Since 0.56.0 the spdxjson-output differs. (with 0.55.0 the syftjson format also differed, but that was fixed with 0.56.0).
What you expected to happen: My code and the cli produce the same SBOM for images.
How to reproduce it (as minimally and precisely as possible): Used image:
alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
Used format:spdxjson
SPDX-json from go-integration
```json { "SPDXID": "SPDXRef-DOCUMENT", "name": "/tmp/sha256_21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300.tar.gz", "spdxVersion": "SPDX-2.2", "creationInfo": { "created": "2022-09-18T08:20:02.269681097Z", "creators": [ "Organization: Anchore, Inc", "Tool: syft-v9.9.9" ], "licenseListVersion": "3.18" }, "dataLicense": "CC0-1.0", "documentNamespace": "https://anchore.com/syft/image/tmp/sha256_21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300.tar.gz-480350d2-9f14-4665-b9b6-737b6333a21a", "packages": [ { "SPDXID": "SPDXRef-9f527213f4d2a873", "name": "alpine-baselayout", "licenseConcluded": "GPL-2.0-only", "description": "Alpine base dir structure and init scripts", "downloadLocation": "https://git.alpinelinux.org/cgit/aports/tree/main/alpine-baselayout", "externalRefs": [ { "referenceCategory": "SECURITY", "referenceLocator": "cpe:2.3:a:alpine-baselayout:alpine-baselayout:3.2.0-r18:*:*:*:*:*:*:*", "referenceType": "cpe23Type" }, { "referenceCategory": "SECURITY", "referenceLocator": "cpe:2.3:a:alpine-baselayout:alpine_baselayout:3.2.0-r18:*:*:*:*:*:*:*", "referenceType": "cpe23Type" }, { "referenceCategory": "SECURITY", "referenceLocator": "cpe:2.3:a:alpine_baselayout:alpine-baselayout:3.2.0-r18:*:*:*:*:*:*:*", "referenceType": "cpe23Type" }, { "referenceCategory": "SECURITY", "referenceLocator": "cpe:2.3:a:alpine_baselayout:alpine_baselayout:3.2.0-r18:*:*:*:*:*:*:*", "referenceType": "cpe23Type" }, { "referenceCategory": "SECURITY", "referenceLocator": "cpe:2.3:a:alpine:alpine-baselayout:3.2.0-r18:*:*:*:*:*:*:*", "referenceType": "cpe23Type" }, { "referenceCategory": "SECURITY", "referenceLocator": "cpe:2.3:a:alpine:alpine_baselayout:3.2.0-r18:*:*:*:*:*:*:*", "referenceType": "cpe23Type" }, { "referenceCategory": "PACKAGE_MANAGER", "referenceLocator": "pkg:alpine/alpine-baselayout@3.2.0-r18?arch=x86_64&upstream=alpine-baselayout&distro=alpine-3.15.0", "referenceType": "purl" } ], "filesAnalyzed": false, "hasFiles": [ "SPDXRef-2eaa15c5fc625ebe", "SPDXRef-a53373020dfa8bb4", "SPDXRef-38605c90f707fb90", "SPDXRef-60fa740c32339374", "SPDXRef-24d0f8d913cd9906", "SPDXRef-d41a5f82a774a6a1", "SPDXRef-13d6d27618d264f7", "SPDXRef-b499705c36475f74", "SPDXRef-2e3613b244458b5a", "SPDXRef-84fd54b3f2a2e825", "SPDXRef-32701f6d1e056c29", "SPDXRef-93b858998f2c7034", "SPDXRef-fb021b79aa9cd553", "SPDXRef-82fda88ae28dd50", "SPDXRef-9ab25fdcabefa4ac", "SPDXRef-2c0eaf2a7d7dbad", "SPDXRef-f3ee626693308800", "SPDXRef-420fa6f3289d6ee6", "SPDXRef-ae2cba512a3f4065", "SPDXRef-b3a5f05adcd1cf82", "SPDXRef-64b20ab568341372", "SPDXRef-5e12c5188eeb9cb3", "SPDXRef-18d9a7fcef583aeb", "SPDXRef-dc65dbf355556024" ], "licenseDeclared": "GPL-2.0-only", "originator": "Person: Natanael CopaDifferences to CLI-Output
```diff 61c61,62 < "SPDXRef-2eaa15c5fc625ebe", --- > "SPDXRef-84fd54b3f2a2e825", > "SPDXRef-18d9a7fcef583aeb", 63,64c64 < "SPDXRef-38605c90f707fb90", < "SPDXRef-60fa740c32339374", --- > "SPDXRef-2c0eaf2a7d7dbad", 66,67c66,67 < "SPDXRef-d41a5f82a774a6a1", < "SPDXRef-13d6d27618d264f7", --- > "SPDXRef-64b20ab568341372", > "SPDXRef-dc65dbf355556024", 69,71c69,75 < "SPDXRef-2e3613b244458b5a", < "SPDXRef-84fd54b3f2a2e825", < "SPDXRef-32701f6d1e056c29", --- > "SPDXRef-9ab25fdcabefa4ac", > "SPDXRef-13d6d27618d264f7", > "SPDXRef-2eaa15c5fc625ebe", > "SPDXRef-38605c90f707fb90", > "SPDXRef-60fa740c32339374", > "SPDXRef-420fa6f3289d6ee6", > "SPDXRef-5e12c5188eeb9cb3", 73d76 < "SPDXRef-fb021b79aa9cd553", 75,76d77 < "SPDXRef-9ab25fdcabefa4ac", < "SPDXRef-2c0eaf2a7d7dbad", 78d78 < "SPDXRef-420fa6f3289d6ee6", 81,84c81,84 < "SPDXRef-64b20ab568341372", < "SPDXRef-5e12c5188eeb9cb3", < "SPDXRef-18d9a7fcef583aeb", < "SPDXRef-dc65dbf355556024" --- > "SPDXRef-2e3613b244458b5a", > "SPDXRef-d41a5f82a774a6a1", > "SPDXRef-32701f6d1e056c29", > "SPDXRef-fb021b79aa9cd553" 136,137d135 < "SPDXRef-ccc2b3e76affde68", < "SPDXRef-3562d93285c5a3c5", 139,141d136 < "SPDXRef-ff0560ee36b984a7", < "SPDXRef-79cc1d44454e11b9", < "SPDXRef-56080e31fd12fe67", 143d137 < "SPDXRef-57149f915867bf12", 144a139,146 > "SPDXRef-395f72182f48f77c", > "SPDXRef-66756a275982c586", > "SPDXRef-56080e31fd12fe67", > "SPDXRef-187efc434122356a", > "SPDXRef-59d943ecba7b9db1", > "SPDXRef-79cc1d44454e11b9", > "SPDXRef-abfd85d1b45289dc", > "SPDXRef-ccc2b3e76affde68", 148,151c150 < "SPDXRef-66756a275982c586", < "SPDXRef-4d646d694b6380fc", < "SPDXRef-add734ec170033bd", < "SPDXRef-abfd85d1b45289dc", --- > "SPDXRef-3562d93285c5a3c5", 154,157c153,157 < "SPDXRef-395f72182f48f77c", < "SPDXRef-187efc434122356a", < "SPDXRef-59d943ecba7b9db1", < "SPDXRef-2c8a8c151837aa6e" --- > "SPDXRef-2c8a8c151837aa6e", > "SPDXRef-57149f915867bf12", > "SPDXRef-4d646d694b6380fc", > "SPDXRef-add734ec170033bd", > "SPDXRef-ff0560ee36b984a7" 236a237,238 > "SPDXRef-988a54d89f5c4c09", > "SPDXRef-cd1c702a19149d7d", 240,242c242 < "SPDXRef-d5ee1ce0839cb21a", < "SPDXRef-988a54d89f5c4c09", < "SPDXRef-cd1c702a19149d7d" --- > "SPDXRef-d5ee1ce0839cb21a" 1097c1097 < "relatedSpdxElement": "SPDXRef-be5355441673f6dc" --- > "relatedSpdxElement": "SPDXRef-988a54d89f5c4c09" 1102c1102 < "relatedSpdxElement": "SPDXRef-e7d6b30bf31f933a" --- > "relatedSpdxElement": "SPDXRef-cd1c702a19149d7d" 1107c1107 < "relatedSpdxElement": "SPDXRef-e6d162458c0b30b0" --- > "relatedSpdxElement": "SPDXRef-be5355441673f6dc" 1112c1112 < "relatedSpdxElement": "SPDXRef-d5ee1ce0839cb21a" --- > "relatedSpdxElement": "SPDXRef-e7d6b30bf31f933a" 1117c1117 < "relatedSpdxElement": "SPDXRef-988a54d89f5c4c09" --- > "relatedSpdxElement": "SPDXRef-e6d162458c0b30b0" 1122c1122 < "relatedSpdxElement": "SPDXRef-cd1c702a19149d7d" --- > "relatedSpdxElement": "SPDXRef-d5ee1ce0839cb21a" 1127c1127 < "relatedSpdxElement": "SPDXRef-2eaa15c5fc625ebe" --- > "relatedSpdxElement": "SPDXRef-84fd54b3f2a2e825" 1132c1132 < "relatedSpdxElement": "SPDXRef-a53373020dfa8bb4" --- > "relatedSpdxElement": "SPDXRef-18d9a7fcef583aeb" 1137c1137 < "relatedSpdxElement": "SPDXRef-38605c90f707fb90" --- > "relatedSpdxElement": "SPDXRef-a53373020dfa8bb4" 1142c1142 < "relatedSpdxElement": "SPDXRef-60fa740c32339374" --- > "relatedSpdxElement": "SPDXRef-2c0eaf2a7d7dbad" 1152c1152 < "relatedSpdxElement": "SPDXRef-d41a5f82a774a6a1" --- > "relatedSpdxElement": "SPDXRef-64b20ab568341372" 1157c1157 < "relatedSpdxElement": "SPDXRef-13d6d27618d264f7" --- > "relatedSpdxElement": "SPDXRef-dc65dbf355556024" 1167c1167 < "relatedSpdxElement": "SPDXRef-2e3613b244458b5a" --- > "relatedSpdxElement": "SPDXRef-9ab25fdcabefa4ac" 1172c1172 < "relatedSpdxElement": "SPDXRef-84fd54b3f2a2e825" --- > "relatedSpdxElement": "SPDXRef-13d6d27618d264f7" 1177c1177 < "relatedSpdxElement": "SPDXRef-32701f6d1e056c29" --- > "relatedSpdxElement": "SPDXRef-2eaa15c5fc625ebe" 1182c1182 < "relatedSpdxElement": "SPDXRef-93b858998f2c7034" --- > "relatedSpdxElement": "SPDXRef-38605c90f707fb90" 1187c1187 < "relatedSpdxElement": "SPDXRef-fb021b79aa9cd553" --- > "relatedSpdxElement": "SPDXRef-60fa740c32339374" 1192c1192 < "relatedSpdxElement": "SPDXRef-82fda88ae28dd50" --- > "relatedSpdxElement": "SPDXRef-420fa6f3289d6ee6" 1197c1197 < "relatedSpdxElement": "SPDXRef-9ab25fdcabefa4ac" --- > "relatedSpdxElement": "SPDXRef-5e12c5188eeb9cb3" 1202c1202 < "relatedSpdxElement": "SPDXRef-2c0eaf2a7d7dbad" --- > "relatedSpdxElement": "SPDXRef-93b858998f2c7034" 1207c1207 < "relatedSpdxElement": "SPDXRef-f3ee626693308800" --- > "relatedSpdxElement": "SPDXRef-82fda88ae28dd50" 1212c1212 < "relatedSpdxElement": "SPDXRef-420fa6f3289d6ee6" --- > "relatedSpdxElement": "SPDXRef-f3ee626693308800" 1227c1227 < "relatedSpdxElement": "SPDXRef-64b20ab568341372" --- > "relatedSpdxElement": "SPDXRef-2e3613b244458b5a" 1232c1232 < "relatedSpdxElement": "SPDXRef-5e12c5188eeb9cb3" --- > "relatedSpdxElement": "SPDXRef-d41a5f82a774a6a1" 1237c1237 < "relatedSpdxElement": "SPDXRef-18d9a7fcef583aeb" --- > "relatedSpdxElement": "SPDXRef-32701f6d1e056c29" 1242c1242 < "relatedSpdxElement": "SPDXRef-dc65dbf355556024" --- > "relatedSpdxElement": "SPDXRef-fb021b79aa9cd553" 1247c1247 < "relatedSpdxElement": "SPDXRef-ccc2b3e76affde68" --- > "relatedSpdxElement": "SPDXRef-27d8de5355fdb7ba" 1252c1252 < "relatedSpdxElement": "SPDXRef-3562d93285c5a3c5" --- > "relatedSpdxElement": "SPDXRef-7803dc5a1a496765" 1257c1257 < "relatedSpdxElement": "SPDXRef-27d8de5355fdb7ba" --- > "relatedSpdxElement": "SPDXRef-2363acec0a71a382" 1262c1262 < "relatedSpdxElement": "SPDXRef-ff0560ee36b984a7" --- > "relatedSpdxElement": "SPDXRef-395f72182f48f77c" 1267c1267 < "relatedSpdxElement": "SPDXRef-79cc1d44454e11b9" --- > "relatedSpdxElement": "SPDXRef-66756a275982c586" 1277c1277 < "relatedSpdxElement": "SPDXRef-7803dc5a1a496765" --- > "relatedSpdxElement": "SPDXRef-187efc434122356a" 1282c1282 < "relatedSpdxElement": "SPDXRef-57149f915867bf12" --- > "relatedSpdxElement": "SPDXRef-59d943ecba7b9db1" 1287c1287 < "relatedSpdxElement": "SPDXRef-2363acec0a71a382" --- > "relatedSpdxElement": "SPDXRef-79cc1d44454e11b9" 1292c1292 < "relatedSpdxElement": "SPDXRef-8ec9dcf9b3d1d7ce" --- > "relatedSpdxElement": "SPDXRef-abfd85d1b45289dc" 1297c1297 < "relatedSpdxElement": "SPDXRef-39dcc03ca17480ca" --- > "relatedSpdxElement": "SPDXRef-ccc2b3e76affde68" 1302c1302 < "relatedSpdxElement": "SPDXRef-496698ff67ca49fc" --- > "relatedSpdxElement": "SPDXRef-8ec9dcf9b3d1d7ce" 1307c1307 < "relatedSpdxElement": "SPDXRef-66756a275982c586" --- > "relatedSpdxElement": "SPDXRef-39dcc03ca17480ca" 1312c1312 < "relatedSpdxElement": "SPDXRef-4d646d694b6380fc" --- > "relatedSpdxElement": "SPDXRef-496698ff67ca49fc" 1317c1317 < "relatedSpdxElement": "SPDXRef-add734ec170033bd" --- > "relatedSpdxElement": "SPDXRef-3562d93285c5a3c5" 1322c1322 < "relatedSpdxElement": "SPDXRef-abfd85d1b45289dc" --- > "relatedSpdxElement": "SPDXRef-2dac0f0b0463195c" 1327c1327 < "relatedSpdxElement": "SPDXRef-2dac0f0b0463195c" --- > "relatedSpdxElement": "SPDXRef-f059a81847acaad9" 1332c1332 < "relatedSpdxElement": "SPDXRef-f059a81847acaad9" --- > "relatedSpdxElement": "SPDXRef-2c8a8c151837aa6e" 1337c1337 < "relatedSpdxElement": "SPDXRef-395f72182f48f77c" --- > "relatedSpdxElement": "SPDXRef-57149f915867bf12" 1342c1342 < "relatedSpdxElement": "SPDXRef-187efc434122356a" --- > "relatedSpdxElement": "SPDXRef-4d646d694b6380fc" 1347c1347 < "relatedSpdxElement": "SPDXRef-59d943ecba7b9db1" --- > "relatedSpdxElement": "SPDXRef-add734ec170033bd" 1352c1352 < "relatedSpdxElement": "SPDXRef-2c8a8c151837aa6e" --- > "relatedSpdxElement": "SPDXRef-ff0560ee36b984a7" ```Note: Differences like
name
,creationInfo
ordocumentNamespace
are ok and not part of my diff. The differences appear inhasFiles
andrelatedSpdxElement
properties.Anything else we need to know?: Maybe I have to change something from my code which fixes the differences, then please let me know :wink:.
Environment:
syft version
:cat /etc/os-release
or similar):