Support for vendored packages

[conradoverta]

I really enjoy goda! It's been very helpful on debugging. Now I want to use it as part of our CI to build and test the application when the relevant packages have changed.

Our CI is air gapped and all the packages vendored, but when I call goda list, I see it's trying to fetch the packages from the internet.

Is there a way to enable vendored package in goda?

[egonelbre]

Is there a specific check you wish to do? There might be another way to check something.

Do you have an example repo that I can take a look at?

Supporting vendoring would be neat, they don't seem to work really nice together with modules. It might be possible to add "replace" directives into go.mod to make it clear where to find packages.

[conradoverta]

We have a monorepo with all of our Go services and libraries. Currently, on CI, we trigger the build and deploy of every component on every change, which is not ideal as components are updated without any real change. My goal is to use goda to identify which packages are used by a main function and cross-reference that with files changed in git to see if a specific package needs to be built and updated. When I tried to do the analysis, it started trying to fetch from the internet. Unfortunately I can't ignore the external dependencies because they might change and cause a package to be updated.

The alternative I'm currently considering is mounting my vendor folder in the $GOPATH/src to pretend the packages are there, but that would cause some conflicts for parallel builds in different branches, which means I'd have to have duplicated $GOPATH copies to the different builds, which seems like the wrong approach. I'd be happy with some other recommendations on how to achieve this.

[egonelbre]

Hmm, maybe a better approach for you would be to build the binaries and see whether the binaries themselves have changed. Go should have reproducible builds with -trimpath rootpath -ldflags=-buildid= (or similar).

[conradoverta]

Our builds are reproducible, but then it requires more stateful checks vs using git as the state reference, and I generally prefer not having to add more state if we can avoid.

We'd have to save the binary somewhere (currently we just pack in a container and save the container) (or save just the SHA, but it's about the same complexity) and identify what is the last one to fetch and compare. It would also require us to build things that have not changed, which is time consuming (even though go compilation is fast, it's not free).

I'll put some more thought if that would be the easier path or if we could figure something simpler. If you have other ideas, I'd be interested in considering those too! I bet someone has resolved this problem, but I just can't find a pre-existing solution.

[egonelbre]

You can checkout the previous git commit and the new one, then compile both during deployment checks.

I'll also whip up a quick alternative check it shouldn't be too difficult to implement.

[egonelbre]

Here's a proof of concept tool with a demo repository that should do what you need:

https://github.com/egonelbre/vendoring-test/blob/master/changes/main.go

Clone the repository and inside the changes folder run:

go run . -- ../vendor/vendor.test/sigma/value.go

It'll check which of the programs in "cmd" depend on that file.

[egonelbre]

As expected, that approach might miss some things such as compiler version changes and flags.

[egonelbre]

Also, running goda list with that repository seems to work as intended. I'm not getting any errors for accessing the internet. Also the next Go version should be even stricter with it.

Which Go version are you using? And are you using go mod vendor to vendor your packages?

[egonelbre]

Note, you can also use something like:

go list -deps -f '{{ $dir := .Dir }}{{range .GoFiles}}{{$dir}}{{.}}, {{ end }}' .

To list all the go files that are compiled into the binary.

[egonelbre]

@conradoverta let me know whether any of the proposed alternatives would work better for you. As far it seems to me, goda does work with vendoring (as tested on the repository), so I'm not sure what should be fixed.

[egonelbre]

Closing for now. You can reopen, if you have an exact scenario that doesn't work.

[conradoverta]

Hi, @egonelbre, sorry for being MIA! I'll check it out. When I tried to run, it tried fetching packages from the internet instead of using the vendored ones, but it might have been a user error.

We're currently on an older version of Go since we haven't had the need to bump it, which might also be affecting. I'll consider that also.

But thanks for all the pointers and info! It's definitely appreciated 🙇