search

CycloneDX / cdxgen

Creates CycloneDX Bill of Materials (BOM) for your projects from source and container images. Supports many languages and package managers. Integrate in your CI/CD pipeline with automatic submission to Dependency Track server.
https://cyclonedx.github.io/cdxgen/
Apache License 2.0
546 stars 158 forks source link

cdxgen fails to determine Golang dependencies on Windows #917

Open mahenarayan opened 6 months ago

mahenarayan commented 6 months ago

Hello, I have been trying out cdxgen on Windows to retrieve Golang dependencies for a GO project. This happens with any GO project.

Content of go.mod

module yaml2json

go 1.21

require (
    github.com/onsi/ginkgo/v2 v2.13.2
    github.com/onsi/gomega v1.30.0
    gopkg.in/yaml.v3 v3.0.1
)

require (
    github.com/go-logr/logr v1.3.0 // indirect
    github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect
    github.com/google/go-cmp v0.6.0 // indirect
    github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect
    golang.org/x/net v0.17.0 // indirect
    golang.org/x/sys v0.14.0 // indirect
    golang.org/x/text v0.13.0 // indirect
    golang.org/x/tools v0.14.0 // indirect
)

Command used to generate table of dependencies along with env SCAN_DEBUG_MODE=true

cdxgen.exe --spec-version 1.4 "C:\ws\k2s\pkg\util\yaml2json\" -p
Scanning C:\ws\k2s\pkg\util\yaml2json\
Performing babel-based package usage analysis with source code at C:\ws\k2s\pkg\util\yaml2json\
Parsing C:\ws\k2s\pkg\util\yaml2json\go.sum
About to fetch go package license information for github.com/chzyer/logex
About to fetch go package license information for github.com/chzyer/readline
About to fetch go package license information for github.com/chzyer/test
About to fetch go package license information for github.com/davecgh/go-spew
About to fetch go package license information for github.com/davecgh/go-spew
About to fetch go package license information for github.com/go-logr/logr
About to fetch go package license information for github.com/go-task/slim-sprig
About to fetch go package license information for github.com/golang/protobuf
About to fetch go package license information for github.com/google/go-cmp
About to fetch go package license information for github.com/google/pprof
About to fetch go package license information for github.com/ianlancetaylor/demangle
About to fetch go package license information for github.com/onsi/ginkgo/v2
About to fetch go package license information for github.com/onsi/gomega
About to fetch go package license information for github.com/pmezard/go-difflib
About to fetch go package license information for github.com/stretchr/objx
About to fetch go package license information for github.com/stretchr/testify
About to fetch go package license information for golang.org/x/mod
About to fetch go package license information for golang.org/x/net
About to fetch go package license information for golang.org/x/sys
About to fetch go package license information for golang.org/x/sys
About to fetch go package license information for golang.org/x/text
About to fetch go package license information for golang.org/x/tools
About to fetch go package license information for google.golang.org/protobuf
About to fetch go package license information for gopkg.in/check.v1
About to fetch go package license information for gopkg.in/yaml.v3
About to fetch go package license information for gopkg.in/yaml.v3
Executing go list -deps in C:\ws\k2s\pkg\util\yaml2json
Executing go mod graph in C:\ws\k2s\pkg\util\yaml2json
About to fetch go package license information for :base
About to fetch go package license information for :gopkg.in/yaml.v3
About to fetch go package license information for :yaml2json
About to fetch go package license information for :yaml2json
About to fetch go package license information for :gopkg.in/yaml.v3
Found 2 go packages at C:\ws\k2s\pkg\util\yaml2json\
Found 0 ruby packages at C:\ws\k2s\pkg\util\yaml2json\
BOM includes 2 components and 2 dependencies after dedupe
╔════════════════════════════════════════╗
║            Dependency Tree             ║
║       Generated with ♥ by cdxgen       ║
╟────────────────────────────────────────╢
║ pkg:golang/yaml2json                   ║
║ └── pkg:golang/gopkg.in/yaml.v3@v3.0.1 ║
╚════════════════════════════════════════╝

╔═══════════════════════════╤═════════════════════════════════════╤═══════════════════════════╤═════════════════╗
║ Group                     │ Name                                │                   Version │ Scope           ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ gopkg.in/yaml.v3                    │                    v3.0.1 │ optional        ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ yaml2json                           │                           │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
BOM includes 2 components and 2 dependencies

Above list of components is incomplete with few of the dependencies are filtered for some reason.

Now the same project with cdxgen run on Linux, outputs right list of components.

Parsing /tmp/yaml2json1/go.mod
Found 12 go packages at /tmp/yaml2json1/
Found 0 ruby packages at /tmp/yaml2json1/
BOM includes 12 components and 2 dependencies after dedupe
===== WARNINGS =====
[ 'Version is missing for metadata.component' ]
╔════════════════════════════════════════╗
║            Dependency Tree             ║
║       Generated with ♥ by cdxgen       ║
╟────────────────────────────────────────╢
║ pkg:golang/yaml2json                   ║
║ └── pkg:golang/gopkg.in/yaml.v3@v3.0.1 ║
╚════════════════════════════════════════╝

╔═══════════════════════════╤═════════════════════════════════════╤═══════════════════════════╤═════════════════╗
║ Group                     │ Name                                │                   Version │ Scope           ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ gopkg.in/yaml.v3                    │                    v3.0.1 │ required        ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ yaml2json                           │                           │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ github.com/onsi/ginkgo/v2           │                   v2.13.2 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ github.com/onsi/gomega              │                   v1.30.0 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ github.com/go-logr/logr             │                    v1.3.0 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ github.com/go-task/slim-sprig       │ v0.0.0-20230315185526-52c │                 ║
║                           │                                     │                 cab3ef572 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ github.com/google/go-cmp            │                    v0.6.0 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ github.com/google/pprof             │ v0.0.0-20210407192527-94a │                 ║
║                           │                                     │                 9f03dee38 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ golang.org/x/net                    │                   v0.17.0 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ golang.org/x/sys                    │                   v0.14.0 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ golang.org/x/text                   │                   v0.13.0 │                 ║
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║                           │ golang.org/x/tools                  │                   v0.14.0 │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
BOM includes 12 components and 2 dependencies

Both use cases tried with cdxgen version and reproducible even on older ones.

 λ C:\ws\k2s\bin\cdxgen.exe -v
10.2.2

Can someone help me with this issue? I am new to JS but I can fix the problem if there is support from the group. I will look for root cause but it will take some time as it is a new codebase for me. Thanks.

prabhu commented 6 months ago

@mahenarayan, Could you check the output of go list -deps and go mod graph on Windows? Also run cdxgen with the environment variable CDXGEN_DEBUG_MODE=debug?

mahenarayan commented 6 months ago

@prabhu Here you go 

minato in C:\ws\k2s\pkg\util\yaml2json on main ● ?4 λ go list -deps
base/system
internal/goarch
unsafe
internal/abi
internal/unsafeheader
internal/cpu
internal/bytealg
internal/coverage/rtcov
internal/godebugs
internal/goexperiment
internal/goos
runtime/internal/atomic
runtime/internal/math
runtime/internal/sys
runtime
internal/reflectlite
errors
internal/itoa
math/bits
math
unicode/utf8
strconv
internal/race
sync/atomic
sync
unicode
reflect
sort
internal/fmtsort
io
internal/syscall/windows/sysdll
internal/oserror
unicode/utf16
syscall
internal/syscall/windows
internal/syscall/windows/registry
time
internal/poll
internal/safefilepath
internal/syscall/execenv
internal/testlog
path
io/fs
os
fmt
bytes
context
encoding
encoding/binary
encoding/base64
strings
encoding/json
log/internal
log
log/slog/internal
log/slog/internal/buffer
cmp
slices
log/slog
path/filepath
base/logging
base/cli
flag
regexp/syntax
regexp
gopkg.in/yaml.v3
yaml2json

Output of go mod graph in file gomodGraph.txt

Env CDXGEN_DEBUG_MODE=debug was already set as well, I missed to mention this in the issue.

prabhu commented 6 months ago

@mahenarayan, thank you. Can you share the go list -deps and go mod graph output from linux?

mahenarayan commented 6 months ago

@prabhu

go list -deps

kubemaster:/tmp/yaml2json$ go list -deps
main.go:8:2: package base/cli is not in std (/usr/local/go-1.21.4/src/base/cli)
main.go:9:2: package base/logging is not in std (/usr/local/go-1.21.4/src/base/logging)
base/cli
base/logging
internal/goarch
unsafe
internal/abi
internal/unsafeheader
internal/cpu
internal/bytealg
internal/coverage/rtcov
internal/godebugs
internal/goexperiment
internal/goos
runtime/internal/atomic
runtime/internal/math
runtime/internal/sys
runtime/internal/syscall
runtime
internal/reflectlite
errors
internal/race
sync/atomic
sync
io
unicode
unicode/utf8
bytes
encoding
math/bits
math
internal/itoa
strconv
reflect
encoding/binary
encoding/base64
sort
internal/fmtsort
internal/oserror
syscall
internal/syscall/unix
time
internal/poll
internal/safefilepath
internal/syscall/execenv
internal/testlog
path
io/fs
os
fmt
strings
unicode/utf16
encoding/json
flag
regexp/syntax
regexp
gopkg.in/yaml.v3
context
log/internal
log
log/slog/internal
log/slog/internal/buffer
cmp
slices
log/slog
yaml2json

go mod graph

kubemaster:/tmp/yaml2json$ go mod graph
yaml2json github.com/go-logr/logr@v1.3.0
yaml2json github.com/go-task/slim-sprig@v0.0.0-20230315185526-52ccab3ef572
yaml2json github.com/google/go-cmp@v0.6.0
yaml2json github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38
yaml2json github.com/onsi/ginkgo/v2@v2.13.2
yaml2json github.com/onsi/gomega@v1.30.0
yaml2json go@1.21
yaml2json golang.org/x/net@v0.17.0
yaml2json golang.org/x/sys@v0.14.0
yaml2json golang.org/x/text@v0.13.0
yaml2json golang.org/x/tools@v0.14.0
yaml2json gopkg.in/yaml.v3@v3.0.1
github.com/go-task/slim-sprig@v0.0.0-20230315185526-52ccab3ef572 github.com/davecgh/go-spew@v1.1.1
github.com/go-task/slim-sprig@v0.0.0-20230315185526-52ccab3ef572 github.com/stretchr/testify@v1.6.1
github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38 github.com/chzyer/logex@v1.1.10
github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38 github.com/chzyer/readline@v0.0.0-20180603132655-2972be24d48e
github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38 github.com/chzyer/test@v0.0.0-20180213035817-a1ea475d72b1
github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38 github.com/ianlancetaylor/demangle@v0.0.0-20200824232613-28f6c0f3b639
github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38 golang.org/x/sys@v0.0.0-20191204072324-ce4227a45e2e
github.com/onsi/ginkgo/v2@v2.13.2 github.com/go-logr/logr@v1.3.0
github.com/onsi/ginkgo/v2@v2.13.2 github.com/go-task/slim-sprig@v0.0.0-20230315185526-52ccab3ef572
github.com/onsi/ginkgo/v2@v2.13.2 github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38
github.com/onsi/ginkgo/v2@v2.13.2 github.com/onsi/gomega@v1.29.0
github.com/onsi/ginkgo/v2@v2.13.2 golang.org/x/net@v0.17.0
github.com/onsi/ginkgo/v2@v2.13.2 golang.org/x/sys@v0.14.0
github.com/onsi/ginkgo/v2@v2.13.2 golang.org/x/tools@v0.14.0
github.com/onsi/ginkgo/v2@v2.13.2 github.com/golang/protobuf@v1.5.3
github.com/onsi/ginkgo/v2@v2.13.2 github.com/google/go-cmp@v0.6.0
github.com/onsi/ginkgo/v2@v2.13.2 golang.org/x/text@v0.13.0
github.com/onsi/ginkgo/v2@v2.13.2 google.golang.org/protobuf@v1.28.0
github.com/onsi/ginkgo/v2@v2.13.2 gopkg.in/yaml.v3@v3.0.1
github.com/onsi/gomega@v1.30.0 github.com/golang/protobuf@v1.5.3
github.com/onsi/gomega@v1.30.0 github.com/google/go-cmp@v0.6.0
github.com/onsi/gomega@v1.30.0 github.com/onsi/ginkgo/v2@v2.13.0
github.com/onsi/gomega@v1.30.0 golang.org/x/net@v0.17.0
github.com/onsi/gomega@v1.30.0 gopkg.in/yaml.v3@v3.0.1
github.com/onsi/gomega@v1.30.0 github.com/go-logr/logr@v1.2.4
github.com/onsi/gomega@v1.30.0 github.com/go-task/slim-sprig@v0.0.0-20230315185526-52ccab3ef572
github.com/onsi/gomega@v1.30.0 github.com/google/pprof@v0.0.0-20210407192527-94a9f03dee38
github.com/onsi/gomega@v1.30.0 golang.org/x/sys@v0.13.0
github.com/onsi/gomega@v1.30.0 golang.org/x/text@v0.13.0
github.com/onsi/gomega@v1.30.0 golang.org/x/tools@v0.12.0
github.com/onsi/gomega@v1.30.0 google.golang.org/protobuf@v1.28.0
go@1.21 toolchain@go1.21
golang.org/x/net@v0.17.0 golang.org/x/crypto@v0.14.0
golang.org/x/net@v0.17.0 golang.org/x/sys@v0.13.0
golang.org/x/net@v0.17.0 golang.org/x/term@v0.13.0
golang.org/x/net@v0.17.0 golang.org/x/text@v0.13.0
golang.org/x/text@v0.13.0 golang.org/x/tools@v0.6.0
golang.org/x/text@v0.13.0 golang.org/x/mod@v0.8.0
golang.org/x/text@v0.13.0 golang.org/x/sys@v0.5.0
golang.org/x/tools@v0.14.0 github.com/yuin/goldmark@v1.4.13
golang.org/x/tools@v0.14.0 golang.org/x/mod@v0.13.0
golang.org/x/tools@v0.14.0 golang.org/x/net@v0.16.0
golang.org/x/tools@v0.14.0 golang.org/x/sys@v0.13.0
golang.org/x/tools@v0.14.0 golang.org/x/sync@v0.4.0
gopkg.in/yaml.v3@v3.0.1 gopkg.in/check.v1@v0.0.0-20161208181325-20d25e280405
github.com/stretchr/testify@v1.6.1 github.com/davecgh/go-spew@v1.1.0
github.com/stretchr/testify@v1.6.1 github.com/pmezard/go-difflib@v1.0.0
github.com/stretchr/testify@v1.6.1 github.com/stretchr/objx@v0.1.0
github.com/stretchr/testify@v1.6.1 gopkg.in/yaml.v3@v3.0.0-20200313102051-9f266ea9e77c
gopkg.in/yaml.v3@v3.0.0-20200313102051-9f266ea9e77c gopkg.in/check.v1@v0.0.0-20161208181325-20d25e280405
prabhu commented 6 months ago

Thank you. It is not immediately obvious to me. Could you add console.logs to see what might be going on?

mahenarayan commented 6 months ago

@prabhu What do you mean adding console.logs? Is there an environment variable to enable this? I am using standalone cdxgen Windows executable from the Releases page.

prabhu commented 6 months ago

@mahenarayan, I meant cloning the repo locally, performing npm install and manually running cdxgen with console.log statements.

I have added windows label. We don't have any contributor for windows so any help to troubleshoot this would be appreciated.