Cannot create tuple lines if hosted on gitlab with deeper project structure

[mfechner]

Hi,

I just tried to create a new port for semgrep: https://gitlab.fechner.net/mfechner/Gitlab/-/commit/a0268faf641510cabb4d223c525e51559138501b

The port is far from being complete, but as next step I need now to generate the GH and GL tuples to make the required go modules available for the build process.

If I execute make gomod-vendor I get:

===>   gitlab-semgrep-2.2.0 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by gitlab-semgrep-2.2.0 for building
===>  Extracting for gitlab-semgrep-2.2.0
=> SHA256 Checksum OK for gitlab-org/security-products/analyzers-semgrep-70e54468c00aa2fa358dc7c13d034aae325d351e_GL0.tar.gz.
===>  Patching for gitlab-semgrep-2.2.0
GH_TUPLE=       \
                cpuguy83:go-md2man:v2.0.0:cpuguy83_go_md2man_v2/vendor/github.com/cpuguy83/go-md2man/v2 \
                golang:sys:288bc346aa39:golang_sys/vendor/golang.org/x/sys \
                pelletier:go-toml:v1.8.1:pelletier_go_toml/vendor/github.com/pelletier/go-toml \
                russross:blackfriday:v2.0.1:russross_blackfriday_v2/vendor/github.com/russross/blackfriday/v2 \
                shurcooL:sanitized_anchor_name:v1.0.0:shurcool_sanitized_anchor_name/vendor/github.com/shurcooL/sanitized_anchor_name \
                sirupsen:logrus:v1.8.1:sirupsen_logrus/vendor/github.com/sirupsen/logrus \
                urfave:cli:v2.3.0:urfave_cli_v2/vendor/github.com/urfave/cli/v2

                # Errors found during processing:
                #       error getting commit v1.0.0 for gitlab-org/security-products: not found
                #       error getting commit v2.23.0 for gitlab-org/security-products: not found
                #       error getting commit v2.1.0 for gitlab-org/security-products: not found
                #       error getting commit v1.1.0 for gitlab-org/security-products: not found

If I build it manually with go build I get:

╰─ go build
go: downloading github.com/urfave/cli/v2 v2.3.0
go: downloading gitlab.com/gitlab-org/security-products/analyzers/ruleset v1.0.0
go: downloading gitlab.com/gitlab-org/security-products/analyzers/report/v2 v2.1.0
go: downloading gitlab.com/gitlab-org/security-products/analyzers/common/v2 v2.23.0
go: downloading gitlab.com/gitlab-org/security-products/analyzers/command v1.1.0
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading github.com/pelletier/go-toml v1.8.1
go: downloading golang.org/x/sys v0.0.0-20200915084602-288bc346aa39
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0

So it seems that modules2tuples can maybe not handle the deeper nested dependency?

Any idea how to get this fixed?

Thanks a lot.

[dmgk]

Hi Matthias,

Unfortunately I don't think this can be generally fixed for Gitlab hosted packages.

modules2tuple tries to figure out an (account, project, optional_submodule_path) tuple by splitting a package name like gitlab.com/gitlab-org/security-products/analyzers/semgrep. It can be done unambiguously for Github hosted packages because Github doesn't allow slashes in project names, but in Gutlab case it's unclear where project name ends and submodule name starts.

I'm afraid modules2tuple won't be of much help for this port. Also, the recently added GO_MODULE [1] might work better for situations like this because it uses go for figuring out and downloading dependencies.

[1] https://docs.freebsd.org/en/books/porters-handbook/special/#using-go

[mfechner]

Hi Dmitri,

ok I understand. Maybe modules2tuples and the ports should not try to download the packages/sources from github and/or gitlab but from pkg.go.dev.

I have another problem with one go module: https://pkg.go.dev/gopkg.in/DataDog/dd-trace-go.v1@v1.31.0

I reported this issue already here: https://github.com/DataDog/dd-trace-go/issues/929 Problem is here that the tag is existing on pkg.go.dev but not on github which causes an unavailability for FreeBSD.

Do you think we should redesign here the way go packages are downloaded a switch from github/gitlab to pkg.go.dev (if this is even possible)?

[dmgk]

This was the reason GO_MODULE was added. Quoting [1]

The specific case of loading packages should still be done by invoking the go command, which remains the single point of truth for package loading algorithms.

modules2tuple is still supported and will be supported as long as it works, but only go command knows all the details of how to resolve versions and download packages.

[1] https://pkg.go.dev/golang.org/x/mod