carlocab
commented
4 months ago CC @SMillerDev
Closed carlocab closed 4 months ago
carlocab
commented
4 months ago CC @SMillerDev
SMillerDev
commented
4 months ago I'll have to find where this is ignored for tabs since we can't really make sure it's the same. I can probably fix the build date though
carlocab
commented
4 months ago I'll have to find where this is ignored for tabs since we can't really make sure it's the same.
Yup, they will indeed be different. It isn't ignored for tabs -- they're just not stored in the bottle, so they don't affect the bottle checksum.
Bo98
commented
4 months ago Yes, for tabs we don't store this stuff in the bottles - we store them in GitHub Packages manifest annotations instead.
MikeMcQuaid
commented
4 months ago Yes, for tabs we don't store this stuff in the bottles - we store them in GitHub Packages manifest annotations instead.
We should do the same thing for SBOMs dates/times as we do for Tab runtime dependencies: update them after installation (based on the dates/times from the tab): https://github.com/Homebrew/brew/blob/1e4d119f6bf08b782d5f5cea5feedee556258e99/Library/Homebrew/formula_installer.rb#L825-L830
SMillerDev
commented
4 months ago Not sure how to resolve this. We could not write the field if the compiler is the system one maybe? Or, which affects the usefulness iyam, we could drop the bottle inclusion of the file and only write it on install.
MikeMcQuaid
commented
4 months ago I think in an ideal world we'd detect if the compiler was actually used somehow e.g. write a temporary file on first usage of one of the compiler shims.
In cases like this, it's pretty clear that the compiler isn't actually used or a dependency.
Bo98
commented
4 months ago If compiler information needs to be available in the bottle archive via brew fetch (though this archive isn't necessarily representative of a complete install as it's pre-relocation): avoiding system compiler makes sense
If compiler information only needs to be available in the Cellar after brew install: SBOM already fetches the compiler from the tab, so we can exclude it and attach it back again on install.
MikeMcQuaid
commented
4 months ago If compiler information only needs to be available in the Cellar after
brew install: SBOM already fetches the compiler from the tab, so we can exclude it and attach it back again on install.
Yes, this seems best for now.
carlocab
commented
4 months ago If compiler information only needs to be available in the Cellar after
brew install: SBOM already fetches the compiler from the tab, so we can exclude it and attach it back again on install.Yes, this seems best for now.
This is fine, but it might not be enough. The sbom.spdx.json files also reference bottle checksum of dependencies, which in general be different across OS versions even for existing :all bottles. Unless this information is generated during brew fetch?
carlocab
commented
4 months ago Confusingly, the SBOM also seems to contain this snippet:
{
"SPDXID": "SPDXRef-Bottle-node@20",
"name": "node@20",
"versionInfo": "20.13.1",
"filesAnalyzed": false,
"licenseDeclared": "NOASSERTION",
"builtDate": "2024-05-09 05:20:38 -0400",
"licenseConcluded": "MIT",
"downloadLocation": "https://ghcr.io/v2/homebrew/core/node/20/blobs/sha256:a865d16c32d50cdffe26e341fb6a8d52b7c3f95daf10e2a390fb988c4fba0ab3",
"copyrightText": "NOASSERTION",
"externalRefs": [
{
"referenceCategory": "PACKAGE-MANAGER",
"referenceLocator": "pkg:brew/homebrew/core/node@20@20.13.1",
"referenceType": "purl"
}
],
"checksums": [
{
"algorithm": "SHA256",
"checksumValue": "a865d16c32d50cdffe26e341fb6a8d52b7c3f95daf10e2a390fb988c4fba0ab3"
}
]
}Except that the download location (and checksum) is not for the version indicated (20.13.1). Instead, it points to the location and checksum of 20.13.0. Which kinda makes sense, because you can't really write a file containing the checksum of the bottle inside the bottle. (Or I could just be very confused about what's going on here, which is also a possibility.)
MikeMcQuaid
commented
4 months ago Except that the download location (and checksum) is not for the version indicated (20.13.1). Instead, it points to the location and checksum of 20.13.0. Which kinda makes sense, because you can't really write a file containing the checksum of the bottle inside the bottle. (Or I could just be very confused about what's going on here, which is also a possibility.)
This also should be removed at bottling time and restored at install time.
MikeMcQuaid
commented
4 months ago Confirmed rebottling in https://github.com/Homebrew/homebrew-core/pull/171540 post https://github.com/Homebrew/brew/pull/17284 fixes the bottles 🎉
carlocab
commented
4 months ago Thanks @MikeMcQuaid :heart:
carlocab
commented
4 months ago This is still happening. See https://github.com/Homebrew/homebrew-core/commit/fd1c80d8a7955be8aa8d787a045efb3ec9eaa076.
This is basically the problem I describe at https://github.com/Homebrew/brew/issues/17281#issuecomment-2106243956.
MikeMcQuaid
commented
4 months ago This is still happening. See Homebrew/homebrew-core@fd1c80d.
This is basically the problem I describe at #17281 (comment).
That problem was fixed. I cannot reproduce this locally. If I run brew bottle ack --json --only-json-tab I get:
ack/3.7.0/sbom.spdx.json that does not contain any of the fields you mentionThis was not the case before that was fixed.
This is because we're passing bottling: to skip these at brew bottle time:
https://github.com/Homebrew/brew/blob/cb168dfe6df49b3dbf5261265d6838c48141ca52/Library/Homebrew/dev-cmd/bottle.rb#L511
These values are only being added at brew install time:
https://github.com/Homebrew/brew/blob/610b80e6374cfa7f2e9b1409cd7272c158bbef4e/Library/Homebrew/formula_installer.rb#L835
So this is an issue with either brew test-bot or our homebrew-core CI workflows that is somehow resulting in attempting to double-bottle or use older/cached, broken SBOMs in bottles or something. An issue should probably be opened somewhere but I don't think it's this issue and I don't think (for now) it's Homebrew/brew.
carlocab
commented
4 months ago That problem was fixed. I cannot reproduce this locally. If I run
brew bottle ack --json --only-json-tab
No, it is not fixed. You cannot reproduce this problem with ack because ack has no dependencies. You need to do it with something like ruby-build.
Doing brew bottle ruby-build --json --only-json-tab produces a ruby-build/20240517/sbom.spdx.json that contains fields like
{
"SPDXID": "SPDXRef-Package-SPDXRef-openssl@3-3.3.0",
"name": "openssl@3",
"versionInfo": "3.3.0",
"filesAnalyzed": false,
"licenseDeclared": "NOASSERTION",
"licenseConcluded": "Apache-2.0",
"downloadLocation": "https://ghcr.io/v2/homebrew/core/openssl/3/blobs/sha256:ec6f9daf8e32d96f4a2f4cd56d18533ee47bb8d9e7cb3d832ac64115d8a1a4ca",
"copyrightText": "NOASSERTION",
"checksums": [
{
"algorithm": "SHA256",
"checksumValue": "ec6f9daf8e32d96f4a2f4cd56d18533ee47bb8d9e7cb3d832ac64115d8a1a4ca"
}
],
"externalRefs": [
{
"referenceCategory": "PACKAGE-MANAGER",
"referenceLocator": "pkg:brew/openssl@3@3.3.0",
"referenceType": "purl"
}
]
}which will, in general, prevent the creation of :all bottles.
MikeMcQuaid
commented
4 months ago Doing
brew bottle ruby-build --json --only-json-tab
Ok, thanks for the reproduction command. It was not clear how to reproduce this before and not clear to me until rereading that this was an additional issue unrelated to reproducibility but related to :all bottles specifically.
https://github.com/Homebrew/brew/pull/17370 should address this.
brew doctoroutputVerification
brew doctoroutput" above saysYour system is ready to brew.and am still able to reproduce my issue.brew updatetwice and am still able to reproduce my issue.brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.brew configoutputWhat were you trying to do (and why)?
Figure out why we don't have an
:allbottle at https://github.com/Homebrew/homebrew-core/commit/0894397822dcdba3facaf90055f450b2dc741647 (because:allbottles are nice).To examine the differences between the bottles, I used
diffoscope.What happened (include all command output)?
diffoscopeshowed that the bottles have differingsbom.spdx.jsonfiles.diffoscope output
```diff --- bottles_ubuntu-22.04/osinfo-db--20240510.x86_64_linux.bottle.tar.gz +++ bottles_14-arm64-9046359873/osinfo-db--20240510.arm64_sonoma.bottle.tar.gz │ --- osinfo-db--20240510.x86_64_linux.bottle.tar ├── +++ osinfo-db--20240510.arm64_sonoma.bottle.tar │ ├── file list │ │ @@ -1,12 +1,12 @@ │ │ drwxr-xr-x 0 0 0 0 2024-05-10 10:31:02.000000 osinfo-db/20240510/ │ │ drwxr-xr-x 0 0 0 0 2024-05-10 10:31:02.000000 osinfo-db/20240510/.brew/ │ │ -rw-r--r-- 0 0 0 667 2024-05-10 10:31:02.000000 osinfo-db/20240510/.brew/osinfo-db.rb │ │ -rw-r--r-- 0 0 0 17987 2024-05-10 10:31:02.000000 osinfo-db/20240510/LICENSE │ │ --rw-r--r-- 0 0 0 2727 2024-05-10 10:31:02.000000 osinfo-db/20240510/sbom.spdx.json │ │ +-rw-r--r-- 0 0 0 2719 2024-05-10 10:31:02.000000 osinfo-db/20240510/sbom.spdx.json │ │ drwxr-xr-x 0 0 0 0 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/ │ │ drwxr-xr-x 0 0 0 0 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/osinfo/ │ │ -rw-r--r-- 0 0 0 17987 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/osinfo/LICENSE │ │ -rw-r--r-- 0 0 0 8 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/osinfo/VERSION │ │ drwxr-xr-x 0 0 0 0 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/osinfo/datamap/ │ │ drwxr-xr-x 0 0 0 0 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/osinfo/datamap/microsoft.com/ │ │ -rw-r--r-- 0 0 0 1691 2024-05-10 10:31:02.000000 osinfo-db/20240510/share/osinfo/datamap/microsoft.com/win-7-l10n-language.xml │ ├── osinfo-db/20240510/sbom.spdx.json │ │ ├── Pretty-printed │ │ │ @@ -1,11 +1,11 @@ │ │ │ { │ │ │ "SPDXID": "SPDXRef-DOCUMENT", │ │ │ "creationInfo": { │ │ │ - "created": "2024-05-11T20:59:00+00:00", │ │ │ + "created": "2024-05-11T17:00:13-04:00", │ │ │ "creators": [ │ │ │ "Tool: https://github.com/homebrew/brew@4.2.21-109-g71c4bfa" │ │ │ ] │ │ │ }, │ │ │ "dataLicense": "CC0-1.0", │ │ │ "documentDescribes": [ │ │ │ "SPDXRef-Archive-osinfo-db-src", │ │ │ @@ -14,15 +14,15 @@ │ │ │ ], │ │ │ "documentNamespace": "https://formulae.brew.sh/spdx/osinfo-db-20240510.json", │ │ │ "files": [], │ │ │ "name": "SBOM-SPDX-osinfo-db-20240510", │ │ │ "packages": [ │ │ │ { │ │ │ "SPDXID": "SPDXRef-Archive-osinfo-db-src", │ │ │ - "builtDate": "2024-05-10 10:31:02 +0000", │ │ │ + "builtDate": "2024-05-10 06:31:02 -0400", │ │ │ "checksums": [ │ │ │ { │ │ │ "algorithm": "SHA256", │ │ │ "checksumValue": "08a2d521c485687f6be39940d5b3f61bc0f583bb7e3655a131c658385eb7e5ca" │ │ │ } │ │ │ ], │ │ │ "copyrightText": "NOASSERTION", │ │ │ @@ -39,20 +39,20 @@ │ │ │ "checksums": [], │ │ │ "copyrightText": "NOASSERTION", │ │ │ "downloadLocation": "NOASSERTION", │ │ │ "externalRefs": [], │ │ │ "filesAnalyzed": false, │ │ │ "licenseConcluded": "NOASSERTION", │ │ │ "licenseDeclared": "NOASSERTION", │ │ │ - "name": "gcc-11", │ │ │ - "versionInfo": "NOASSERTION" │ │ │ + "name": "clang", │ │ │ + "versionInfo": "15.3" │ │ │ }, │ │ │ { │ │ │ "SPDXID": "SPDXRef-Bottle-osinfo-db", │ │ │ - "builtDate": "2024-05-10 10:31:02 +0000", │ │ │ + "builtDate": "2024-05-10 06:31:02 -0400", │ │ │ "checksums": [ │ │ │ { │ │ │ "algorithm": "SHA256", │ │ │ "checksumValue": "a8c86aee5fd157554d85aa0a28d4c12bc5bdf03ccb5e67ac5c8c524d78bd1971" │ │ │ } │ │ │ ], │ │ │ "copyrightText": "NOASSERTION", ```What did you expect to happen?
These bottles are identical, so they should not have different contents.
Step-by-step reproduction instructions (by running
brewcommands)