hugo mod removes non-Hugo modules from go.mod

deanishe commented 4 years ago

I use mage to manage my sites (download data, publish via rsync, run pngquant on resized images…), and go mod adds mage's dependencies to go.mod.

Hugo removes them every time I run it.

How do I get Hugo modules to play nice with regular Go modules? Is there an option to make it write its dependencies to, say, hugo.mod instead of the go.mod file?

bep commented 4 years ago

Is there an option to make it write its dependencies to, say, hugo.mod instead of the go.mod file?

Not that I know of, but I would be happy to be proven wrong. I would say that I cannot currently fix your problem, but I will think about it.

bep commented 4 years ago

OK , thinking a little I think I know what we can do. I have implemented my own "tidy logic" because Go does not understand Hugo's imports. We can adjust that so:

Store away the go.* content
First run go tidy. In most situations this should make them empty.
Apply Hugo's logic, but keep any lefovers from 2)

deanishe commented 4 years ago

Sounds like a reasonable workaround. Shame Hugo can't use a different file.

bep commented 4 years ago

Shame Hugo can't use a different file.

Or Mage.

I have double-checked the Go source, and go.mod is very much hardcoded ... all over the place.

I will take another round on this and check what Go does with these files on regular use, but I suspect that it will be a variant of the above and that with your setup you cannot use "go mod tidy" as that will wipe out the non-Go entries.

deanishe commented 4 years ago

Or Mage.

It's not Mage. It's go mod doing its thing.

go.mod is very much hardcoded ... all over the place.

That certainly is very hardcoded. Oh well…

as that will wipe out the non-Go entries

I'm not currently using Hugo modules. But it appears that the presence of a go.mod file causes Hugo to run its tidy routine when building or serving.

Also, wouldn't go mod only remove Hugo's entries if you explicitly run tidy?

deanishe commented 4 years ago

If this isn't getting added any time soon, could you make Hugo behave the same as go mod and only remove entries from go.mod when explicitly told to do so (or at least only when hugo mod is run)?

Currently, it clears out go.mod every time I build the site or run hugo server, which means I have to undo its changes basically every time I run Hugo (and stop hugo server first, otherwise it'll just remove the entries again).

myitcv commented 4 years ago

Just run into this as well. I'm not quite sure how Hugo modules are implemented, but I suspect they should be using an approach similar to the way in which tool dependencies are declared?

https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md

i.e. the dependencies need to be declared in Go code if go mod tidy is to see them.

bep commented 4 years ago

i.e. the dependencies need to be declared in Go code if go mod tidy is to see them.

Which is exactly what Hugo does (if you replace "Go code" with "Hugo source"), which is also where the problem lies.

bep commented 4 years ago

I have "turned off" the tidy during regular builds. WIll be in Hugo 0.57.1. You will still see this problem if you do hugo mod tidy.

myitcv commented 4 years ago

which is also where the problem lies.

Just wanted to make sure we're talking about the same thing here: for go mod tidy to work, the dependencies will need to be declared in Go code.

So if you have dependencies declared in "Hugo source" then you will likely need to code generate a build-ignored .go file that also includes those dependencies:

// +build ignore

package hugodeps

import (
    _ "example.com/hugo/dep"
)

bep commented 4 years ago

Just wanted to make sure we're talking about the same thing here: for go mod tidy to work, the dependencies will need to be declared in Go code.

We're obviously not. This is not about go mod tidy.

myitcv commented 4 years ago

Ah, glad I confirmed. I had understood from the conversation above that the Hugo module dependencies would be added to go.mod, but I guess that is not the case.

bep commented 4 years ago

I had understood from the conversation above that the Hugo module dependencies would be added to go.mod, but I guess that is not the case.

They do, but the problem is still not about go mod tidy.

deanishe commented 4 years ago

I had understood from the conversation above that the Hugo module dependencies would be added to go.mod, but I guess that is not the case.

They are. The issue is that Hugo currently behaves as if go.mod is exclusively for its own use and only for Hugo modules, so it removes any actual Go dependencies put there by go mod.

It's also very aggressive about doing that, and "tidies" go.mod when commands other than hugo mod tidy are run. Even when you aren't actually using Hugo modules.

The upshot is that Hugo currently does not play well with go mod.

Not running its tidy routines unless explicitly told to do so would solve the problems this is causing me, but imo, the proper solution is the one @bep initially suggested: Hugo shouldn't alter go mod's entries. After all, the go.mod file belongs to go mod, not Hugo.

bep commented 4 years ago

After all, the go.mod file belongs to go mod, not Hugo.

Just to make my point clear:

go mod tidy acts as if it owns go.mod, which is probably a bug (considering that Go projects also can include non Go resources, but please don't start a discussion on this here).
hugo mod tidy does the same (which is this bug). I have in Hugo 0.57.1 turned off the "auto tidy" feature which should reduce some of this noise.

My main point is this: If you create a Hugo project, that is the spec you need to follow. If you create a "Go and Hugo" mixed module, that comes currently with some issues. If that becomes a problem, put them in each subfolder/module or something.

myitcv commented 4 years ago

If you create a Hugo project, that is the spec you need to follow. If you create a "Go and Hugo" mixed module, that comes currently with some issues.

Thanks for clarifying that you distinguish between these two cases. We have a "Go and Hugo" module, hence why I was referring to go mod tidy above.

I think it's worth ensuring that go mod tidy works regardless because this is what people will expect from any project that contains a go.mod. There's more cognitive load in saying "if it's a Hugo project then you need to run hugo mod tidy", especially because that instruction is unintuitive in a "Go and Hugo" mixed module.

go mod tidy could actually be made to work with either type of project. Every time hugo runs it could ensure that a "shadow" code-generated build-ignored .go file reflects the modules referenced in Hugo source. That way, any subsequent run of go mod tidy will see Hugo's requirements reflected via the code-generated .go file.

deanishe commented 4 years ago

That way, any subsequent run of go mod tidy will see Hugo's requirements reflected via the code-generated .go file.

That doesn't work because Hugo modules aren't go modules. go mod doesn't understand Hugo modules, and Hugo doesn't understand Go modules.

go mod tidy acts as if it owns go.mod

It's called go.mod, not project.stuff…

myitcv commented 4 years ago

That doesn't work because Hugo modules aren't go modules.

Then I'm not quite sure I understand why they are being put into go.mod? (https://github.com/gohugoio/hugo/issues/6115#issuecomment-521707171)

deanishe commented 4 years ago

Then I'm not quite sure I understand why they are being put into go.mod?

They shouldn't be, imo, but Hugo leverages the go mod machinery to manage its modules, and the go.mod filename is very hardcoded, so Hugo can't easily put its stuff in hugo.mod instead.

In any case, go mod doesn't seem to mind Hugo's modules being in there, although go mod tidy will remove them. Now that Hugo v0.57.1 isn't removing go mod's stuff all the time, it's no longer an annoying problem in practice.

Thanks @bep!

bcmills commented 4 years ago

I'm not entirely sure of the details of how you're using the go mod machinery, but:

1) You may want to leave a comment about your usage on https://github.com/golang/go/issues/31866.

2) You might be interested in the -modfile flag we're planning to add in Go 1.14 (https://tip.golang.org/doc/go1.14#go-flags).

sheldonhull commented 1 year ago

@deanishe I had worked with a monorepo before so tried a solution and it's working well for me now. Take a look at magefiles. In the monorepo scenario, I've found it useful for this. The basics are:

Move your magefile (even if just one) into magefiles.
cd magefiles && go mod init mage.local && go mod tidy && go mod vendor (in my case I didn't use canonical name for the root repo as it's just a hugo project and don't need to remotely import).
In project directory, run hugo mod tidy && hugo mod vendor (if using hugo modules).
- Optional: For cutting out multi-module language issues also run go work init && go work use . && go work use ./magefiles. For Go 1.18+ this helps solve multimodule repo gopls issues with completion/intellisense.

Go module for the hugo project isn't impacted then by the mage module tooling.

cc @natefinch as related to monorepo stuff.

gohugoio / hugo

hugo mod removes non-Hugo modules from go.mod #6115