mage is not able to function in an entirely vendored project

[gordallott]

So the use-case is that I'm trying to get mage working within a docker build, in an attempt to keep it as light as possible and because we use private repositories that can be a real pain to get access to from a docker container, I thought the best solution would be to go mod vendor everything in

however doing that breaks mage, because the go list exec in the MageFIles function doesn't supply the "-mod=vendor" command to go. Which means go tries to get the dependency graph of packages from the network, which it can't do.

I'd suggest either a --mod-vendor argument to turn this on specifically or a more general ability to add arguments to go commands

[natefinch]

interesting... yeah, we should fix that. in the meantime, would it work to use mage -compile and just put the compiled magefile binary in docker instead of the code itself?

[thaney071]

This shouldn't be a problem for projects using go 1.14+

https://go.dev/ref/mod#vendoring

If the vendor directory is present in the main module’s root directory, it will be used automatically if the go version in the main module’s go.mod file is 1.14 or higher. To explicitly enable vendoring, invoke the go command with the flag -mod=vendor. To disable vendoring, use the flag -mod=readonly or -mod=mod.

By default from go 1.14 or higher -mod=vendor is implicitly applied when a vendor directory in the root of the module.

I am assuming the original issue was filed before go 1.14. Mage could potentially do the same as the native go commands by adding the -mod=vendor to module aware commands if their using anything lower than go 1.14. It would just need to check the current go version it is running on and if there is a vendor directory.

Given that the docs state mage supports go 1.7+, this should still be addressed if that is to remain true.

However this issue has been open over a year without additional follow up or comments, It doesn't seem likely mage will be used in projects that are vendoring and using go less than 1.14. So it might be worth just updating the documentation as this is a known issue, but it is better to update go versions at this time if vendoring is used.

[sheldonhull]

I was testing vendoring in a project and ran into this issue as well. Error determining list of magefiles: failed to list non-mage gofiles.

Correction, I just figured out an "inconsistent vendoring" issue and once I fixed that and had all my build dependencies defined in a devtools.go file I was able to run mage successfully. Seems to be running fine so far.

[sheldonhull]

I'm back to working through issues on this. I have a vendored project and getting this now:

Error determining list of magefiles: listing mage files: failed to list gofiles tagged with "mage": exit status 1: go: downloading github.com/sheldonhull/magetools v0.0.10

This is start of the chain of errors in that vendoring my magefiles by using a tools.go doesn't seem to be detected by the mage tool, so it tries to redownload the build dependencies. However, in the -mod=readonly environment it's failing to compile mage $(GOBIN)/mage -f -compile ./magec which is what I was doing to prebuild during mage setup and confirm the tasks are ready for the remaining pipeline.

Have to stop investigating tonight, but if anyone else figured out how to get build deps working with vendoring and I'm missing something let me know :-) I even removed //go:build mage from all my magefiles/ to avoid issues there as well.

I know vendoring is the same as the module cache, but figuring out build deps being vendored (nested modules too) in a monorepo has proven much harder to figure out than I thought!

[natefinch]

I've been talking to Sheldon about this a bunch, and Im pretty convinced that this doesn't have anything to do with Mage. All mage does is call go list and go build, both of which should do the right thing with vendoring. I would recommend that if anyone has a problem with mage working with vendoring, try running

go list -e -tags=mage

In the directory where they're running mage, and see if they get the same error. As someone above said, the -mod=vendor is not supposed to be needed in go 1.14+

[sheldonhull]

@natefinch I got it to work. I had lots of issues that I couldn't pinpoint, but I think one of the main culprits as using GOFLAGS=-mod=vendor/readonly. I removed any flag and finally seems to be running nice and smooth in a fully vendored project. If I narrow down the exact culprit I'll try and update this later. This issue might be best to convert to a discussion or close.

I'm now using mage 1.13 with magefiles directory (nothing in root of repo) and seems to work great. Thanks for helping me work through some of this.