search

SamuraiAku / PkgToSoftwareBOM.jl

Produces a Software Bill of Materials (SBOM) describing your Julia Pkg environment. SBOM is in the SPDX format
MIT License
17 stars 2 forks source link

Exception when generating SBOM using an uncompressed registry #13

Closed Octogonapus closed 1 year ago

Octogonapus commented 1 year ago

I tried to generate an SBOM for a demo project but encountered this exception:

ERROR: MethodError: no method matching getindex(::Nothing, ::String)
Stacktrace:
  [1] populate_registryinfo(uuid::Base.UUID, package::Pkg.API.PackageInfo, registry::Pkg.Registry.RegistryInstance)
    @ PkgToSoftwareBOM ~/.julia/packages/PkgToSoftwareBOM/1qwTK/src/Registry.jl:60
  [2] (::PkgToSoftwareBOM.var"#9#10"{Dict{Base.UUID, Pkg.API.PackageInfo}})(k::Base.UUID)
    @ PkgToSoftwareBOM ./none:0
  [3] iterate
    @ ./generator.jl:47 [inlined]
  [4] Dict{Base.UUID, Union{Missing, Nothing, PkgToSoftwareBOM.PackageRegistryInfo}}(kv::Base.Generator{Base.KeySet{Base.UUID, Dict{Base.UUID, Pkg.API.PackageInfo}}, PkgToSoftwareBOM.var"#9#10"{Dict{Base.UUID, Pkg.API.PackageInfo}}})
    @ Base ./dict.jl:85
  [5] registry_packagequery(packages::Dict{Base.UUID, Pkg.API.PackageInfo}, registry::String)
    @ PkgToSoftwareBOM ~/.julia/packages/PkgToSoftwareBOM/1qwTK/src/Registry.jl:36
  [6] registry_packagequery(packages::Dict{Base.UUID, Pkg.API.PackageInfo}, registries::Vector{String})
    @ PkgToSoftwareBOM ~/.julia/packages/PkgToSoftwareBOM/1qwTK/src/Registry.jl:6
  [7] generateSPDX(docData::spdxCreationData, sbomRegistries::Vector{String}, envpkgs::Dict{Base.UUID, Pkg.API.PackageInfo})
    @ PkgToSoftwareBOM ~/.julia/packages/PkgToSoftwareBOM/1qwTK/src/spdxBuild.jl:7
  [8] generateSPDX(docData::spdxCreationData, sbomRegistries::Vector{String})
    @ PkgToSoftwareBOM ~/.julia/packages/PkgToSoftwareBOM/1qwTK/src/spdxBuild.jl:7
  [9] generateSPDX(docData::spdxCreationData)
    @ PkgToSoftwareBOM ~/.julia/packages/PkgToSoftwareBOM/1qwTK/src/spdxBuild.jl:7
 [10] top-level scope
    @ REPL[8]:1

Running this code:

sbom = generateSPDX(spdxCreationData(rootpackages=filter(p -> !(p.first in ["PkgToSoftwareBOM", "SPDX"]), Pkg.project().dependencies)))

My demo project has two files. Project.toml:

name = "packageName"
uuid = "1c653b0a-0b5a-4cff-b25a-92f0db012773"
version = "0.1.0"

[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

[compat]
JSON = "0.21"
julia = "1.8"

and Manifest.toml:

# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.0"
manifest_format = "2.0"
project_hash = "fd1f7ad92e109f821f4124222ecf4f969f1af8a6"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.4"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.Parsers]]
deps = ["Dates", "PrecompileTools", "UUIDs"]
git-tree-sha1 = "a5aef8d4a6e8d81f171b2bd4be5265b01384c74c"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.5.10"

[[deps.PrecompileTools]]
deps = ["Preferences"]
git-tree-sha1 = "259e206946c293698122f63e2b513a7c99a244e8"
uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
version = "1.1.1"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "7eb1686b4f04b82f96ed7a4ea5890a4f0c7a09f1"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.4.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
version = "0.7.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.3"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

PkgToSoftwareBOM v0.1.1 Julia v1.9

SamuraiAku commented 1 year ago

It's erroring out trying to read a Project.toml from the registry. It must be when trying to access files for packageName, but presumably packageName doesn't exist in the registry?

Does packageName exist as a set of source files that you added as a local path in Pkg? Or does just the name exist in your Project.toml?

Octogonapus commented 1 year ago

Indeed, this package isn't in any registry. It's actually just those two TOML files and nothing else.

SamuraiAku commented 1 year ago

This is an edge case I hadn't considered. Where you're presented with a Project.toml that can't be fully resolved. For example, if some packages are in a private registry that the SBOM creator doesn't have access to. The package today does implicitly assume a completely resolved Manifest.toml. Probably worth mentioning in the README.

Still, it shouldn't have gotten that far into the registry resolution process before erroring out. It should have exited with missing right at the start. Will need to investigate further.

Octogonapus commented 1 year ago

The manifest can be fully resolved; all those packages are either stdlibs or are in General. We don't register all of our packages because we don't need to. Some of them are applications that are never installed by anyone but are instead containerized, so they aren't registered. I hope you will consider supporting this use case.

Also you could generate an SBOM for a Manifest without knowing about any registries; you just can't augment it with other info like licenses.

SamuraiAku commented 1 year ago

@Octogonapus I want to solve your problem, but I can't reproduce the issue yet. I feel that I'm missing something in the setup. Here's what I'm doing

  1. I create a new folder called dev_sbom
  2. Inside that folder I create another folder called packageName. I put the Project.toml and the Manifest.toml files in your original post in there
  3. Launch Julia, cd to my dev_sbom folder
  4. Launch interactive Pkg, create a new environment
  5. Try to add packageName as a package. But at this point it all errors out for me. Here's a transcript
julia> cd("./JuliaWork/dev_sbom/")

julia> VERSION
v"1.9.0"

(@v1.8) pkg> activate .
  Activating new project at `~/JuliaWork/dev_sbom`

(dev_sbom) pkg> add ./packageName
ERROR: Did not find a git repository at `packageName`, perhaps you meant `Pkg.develop`?

(dev_sbom) pkg> dev ./packageName/
ERROR: expected the file `src/packageName.jl` to exist for package `packageName` at `packageName`

(dev_sbom) pkg> st
Status `~/JuliaWork/dev_sbom/Project.toml` (empty project)

If you could give me some pointers on how you are setting up your environment, I'll try to reproduce.

Octogonapus commented 1 year ago

Here's a script that can reproduce it:

#!/bin/bash
mkdir testpkg

echo 'name = "packageName"
uuid = "1c653b0a-0b5a-4cff-b25a-92f0db012773"
version = "0.1.0"

[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

[compat]
JSON = "0.21"
julia = "1.8"' > testpkg/Project.toml

echo '# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.0"
manifest_format = "2.0"
project_hash = "fd1f7ad92e109f821f4124222ecf4f969f1af8a6"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.4"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.Parsers]]
deps = ["Dates", "PrecompileTools", "UUIDs"]
git-tree-sha1 = "a5aef8d4a6e8d81f171b2bd4be5265b01384c74c"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.5.10"

[[deps.PrecompileTools]]
deps = ["Preferences"]
git-tree-sha1 = "259e206946c293698122f63e2b513a7c99a244e8"
uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
version = "1.1.1"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "7eb1686b4f04b82f96ed7a4ea5890a4f0c7a09f1"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.4.0"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
version = "0.7.0"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
version = "1.0.3"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"' > testpkg/Manifest.toml

julia --project=testpkg -e 'import Pkg; using PkgToSoftwareBOM; generateSPDX(spdxCreationData(rootpackages=filter(p -> !(p.first in ["PkgToSoftwareBOM", "SPDX"]), Pkg.project().dependencies)))'
SamuraiAku commented 1 year ago

I've run your script and it works just fine for me.

Looking at the stack trace in the original post, it's erroring out on line 60 of Registry.jl

Package= TOML.parse(registry.in_memory_registry[registryPath*"/Package.toml"])

The error message

ERROR: MethodError: no method matching getindex(::Nothing, ::String)

implies that registry.in_memory_registry is nothing.

Could you launch Julia in the configuration that fails for you and then run 

Pkg.Registry.status()

and tell me what registries you have? There would appear to be something different about the Registry memory structures than what I see.

Octogonapus commented 1 year ago 
julia> Pkg.Registry.status()
Registry Status 
 [23338594] General (https://github.com/JuliaRegistries/General.git)
 <redacted>
 <redacted>

I can't show those two registries.

Octogonapus commented 1 year ago

I can actually reproduce this exception by running the same code on this project. I would guess that your General registry is a compressed file? Mine is not, so in_memory_registry is nothing. This is the relevant code https://github.com/JuliaLang/Pkg.jl/blob/master/src/Registry/registry_instance.jl#L305-L331

shell> ls ~/.julia/registries/General
A  B  C  CONTRIBUTING.md  D  E  F  G  H  I  J  jll  K  L  LICENSE.md  M  N  O  P  Q  R  README.md  Registry.toml  S  T  U  V  W  X  Y  Z
SamuraiAku commented 1 year ago

You are correct that my General registry is a compressed file, which is the out of the box behavior. This looks like the cause of the error.

Out of curiosity, how do you get your General registry to be like this?. It would be helpful if I could recreate this scenario.

Octogonapus commented 1 year ago

I believe ENV["JULIA_PKG_SERVER"] = "" is what does it. If not, try adding General by URL.

SamuraiAku commented 1 year ago

I've forked the General registry, modded the UUID, and then added it to my environment. Now I have a compressed and uncompressed registry side-by-side. With the uncompressed registry I get the same error as in the opening post.

Now that I can reproduce the issue, I'll start working on a fix.

SamuraiAku commented 1 year ago

Released as v0.1.2