Concrete go + hugo modules conflict proposal: Add modfile property also for hugo

dionysius commented 1 year ago

I'm happy to try to work on this as a PR, but want first your opinion, especially @bep.

As stated in #6115 both go and hugo think they're the owner of the go.mod file. As commented in https://github.com/gohugoio/hugo/issues/6115#issuecomment-521823167, one using hugo should follow its spec (This is not the place to discuss this decision). Learned how hugo mod works and knowing how go mod works, this issue is very possibly solvable.

Since go1.14 they added the -modfile property allowing you to decide which go.mod file to use. How about hugo modules config get a property with the exact same meaning and propagates that to go commands and further hugo module specific functions?

# config.yaml
# ...
module:
  modfile: hugo.mod # default: go.mod

Result: The developer can opt in to use a separate go mod file for hugo and thus keeping the default go.mod for his other purpose.

bep commented 1 year ago

Since go1.14 they added the -modfile property allowing you to decide which go.mod file to use.

So, this is a great idea, but are you sure there is such thing as a universal -modfile option in Go?

A quick scan of the Go source code shows lots of hardcoded "go.mod" strings:

~/d/g/m/g/src ❯❯❯ find . -name "*.go" | grep -v "test" | xargs grep "\"go.mod\"" | wc -l                       master
      45

dionysius commented 1 year ago

I am not sure, but according to the release notes linked in https://github.com/gohugoio/hugo/issues/6115#issuecomment-571426244 it sounds to be their aim

-modfile=file is a new flag that instructs the go command to read (and possibly write) an alternate go.mod file instead of the one in the module root directory. A file named go.mod must still be present in order to determine the module root directory, but it is not accessed. When -modfile is specified, an alternate go.sum file is also used: its path is derived from the -modfile flag by trimming the .mod extension and appending .sum.

That explains why there are still (some) hard go.mod strings. Unsure about vendoring but in a quick glance hugo has also the custom _vendor folder (but don't know full context yet). Also that option should only be used if such "main" go.mod exists.

I'll have a check in the coming days and crosscheck with the mod client in hugo. But it sounds like if there is a way it's worth a try. And if we/I find a need for something, we can voice our use case in the upstream issue.

bep commented 1 year ago

That explains why there are still (some) hard go.mod strings. Unsure about vendoring but in a quick glance hugo has also the custom _vendor folder (but don't know full context yet).

Vendoring should be fine, I guess. There's a stupid Go bug that ignores vendor folders nested deep inside a go.mod project -- maybe that will magically go away if we could use hugo.mod.

bep commented 1 year ago

Also,

We need to consider dependencies in all of this, so I don't think adding this as a config option would make much sense, but if we could:

Check for, in order, hugo.mod and go.mod and use the first found

dionysius commented 1 year ago

We need to consider dependencies in all of this

I hadn't had that on the radar. If you mean by that a hugo module, yes they might be forced to use go.mod, that might be hardcoded. I use a custom goproxy and can see the requests there. My aim in this request was the end site repo module management, but I'll have a look into it as well.

bep commented 1 year ago

Also, and I don't think it will solve the above problems, we do have a modules.workspace setting that allow you to use e.g. a hugo.work file.

dionysius commented 1 year ago

So I couldn't wait :D

Workspaces seem to be aimed to simplify local development (and I have no experience with it yet). So *.work files are technically/probably not part of repos(?). This feature request is aimed for a endsite monorepo where go and hugo can work together independent of local-machine-only setup.

Overall to my knowledge there is one big property hugo modules differ from go modules

Hugo needs access to all files without them beeing in a go package. Once hugo builds, it needs to be able to access all files within the go pkg cache and same rule applies for vendoring (here go can strip out files)

I made a similiar workaround when working with protobuf files to still be able to vendor them out during compilation

And IMO these are the four main tasks during a hugo module build handling

Preamble: I have a go.mod to be "in the way" when handling hugo.mod

$ cat go.mod
module myproject

go 1.19

require github.com/gohugoio/hugo v0.111.2

1) Initializing hugo.mod & discovery and registering the module in a hugo.mod

$ go mod init -modfile=hugo.mod example.com/m
go: creating new go.mod: module example.com/m
go: to add module requirements and sums:
        go mod tidy

$ go get -modfile=hugo.mod github.com/McShelby/hugo-theme-relearn
go: added github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1

$ cat hugo.mod hugo.sum 
module example.com/m

go 1.19

require github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1 // indirect
github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1 h1:hGJYI84Tqtt5KMJlzyXH08ctE3M2Ygh1Tk4Mr0uYhfs=
github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=

2) Downloading hugo module in the go pkg cache

$ go mod download -modfile=hugo.mod -json
{
        "Path": "github.com/McShelby/hugo-theme-relearn",
        "Version": "v0.0.0-20230303193005-e619dabc51a1",
        "Info": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.info",
        "GoMod": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.mod",
        "Zip": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.zip",
        "Dir": "/home/dionysius/Projects/go/pkg/mod/github.com/!mc!shelby/hugo-theme-relearn@v0.0.0-20230303193005-e619dabc51a1",
        "Sum": "h1:hGJYI84Tqtt5KMJlzyXH08ctE3M2Ygh1Tk4Mr0uYhfs=",
        "GoModSum": "h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM="
}

-json is just for visibility that it actually only read hugo.mod, you can always use -f to only output a specific property

3) Accessability of module contents

a) via go pkg cache

$ go list -m -json -modfile=hugo.mod ...
{
        "Path": "example.com/m",
        "Main": true,
        "Dir": "/home/dionysius/Projects/myproject",
        "GoMod": "hugo.mod",
        "GoVersion": "1.19"
}
{
        "Path": "github.com/McShelby/hugo-theme-relearn",
        "Version": "v0.0.0-20230303193005-e619dabc51a1",
        "Time": "2023-03-03T19:30:05Z",
        "Indirect": true,
        "Dir": "/home/dionysius/Projects/go/pkg/mod/github.com/!mc!shelby/hugo-theme-relearn@v0.0.0-20230303193005-e619dabc51a1",
        "GoMod": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.mod",
        "GoVersion": "1.18"
}

-json is just for visibility that it actually only read hugo.mod, you can always use -f to only output a specific property

$ ls -1 $(go list -m -f '{{.Dir}}' -modfile=hugo.mod github.com/McShelby/hugo-theme-relearn)
archetypes
assets
config.toml
exampleSite
frontmatter.json
go.mod
hugo-theme-relearn.code-workspace
i18n
images
layouts
LICENSE
README.md
static
theme.toml
vscode-frontmatter

In the go pkg cache there seems to be always the full repo checked out.

b) via vendoring

Here it gets tricky, thats maybe why hugo has its own vendoring functions. go vendor does only checkout out go packages, that means any folder containing go files (but then all files). The best I could get is ignoring errors with -e to continue the command

$ go mod vendor -v -e -modfile=hugo.mod -o _vendor
example.com/m imports
        github.com/gohugoio/hugo: no required module provides package github.com/gohugoio/hugo; to add it:
        go get github.com/gohugoio/hugo
# github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1
## explicit; go 1.18

-v is just for visibility

And we have the modules.txt, but no checkout out data:

$ ls -1 _vendor/
modules.txt
$ cat _vendor/modules.txt 
# github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1
## explicit; go 1.18

Because this hugo module does not have any go packages/go files. But we can just copy the files from the go pkg module cache, since it contains all files:

# for each hugo module (if vendoring is not disabled by config)
$ mkdir -p _vendor/github.com/McShelby/hugo-theme-relearn
$ cp -rT $(go list -m -f '{{.Dir}}' -modfile=hugo.mod github.com/McShelby/hugo-theme-relearn) _vendor/github.com/McShelby/hugo-theme-relearn
$ tree -L 4 _vendor/
_vendor/
├── github.com
│   └── McShelby
│       └── hugo-theme-relearn
│           ├── archetypes
│           ├── assets
│           ├── config.toml
│           ├── exampleSite
│           ├── frontmatter.json
│           ├── go.mod
│           ├── hugo-theme-relearn.code-workspace
│           ├── i18n
│           ├── images
│           ├── layouts
│           ├── LICENSE
│           ├── README.md
│           ├── static
│           ├── theme.toml
│           └── vscode-frontmatter
└── modules.txt

11 directories, 8 files

4) Build the hugo site

Hugo can access now the vendor folder if exists (isVendored() from modules/collect.go:543) or fall back to the go pkg cache as already (listAndDecodeModules() from modules/client.go:472), but add -modfile to it.

Some hugo mod commands not reflected above

hugo mod tidy needs indeed to be a hugo implementation as the go variant scans the source code to tidy it - and hugo has no go files. But hugo has them listed as imports in the config and thus already can write them out. Respecting the -modfile switch would let it just write it to that .mod and .sum file.

hugo mod graph can be forwarded to go:

go mod graph -modfile hugo.mod
example.com/m github.com/McShelby/hugo-theme-relearn@v0.0.0-20230303193005-e619dabc51a1

Notes

I'm pretty sure hugo already does those workarounds in some way or another. I just listed the full cycle in pure commands.
Right now I have golang 1.19.4, experience might vary with earlier go version so this needs to be tested as well.
Does hugo verify vendored modules? Because I saw in modules/collect.go:209 (// If both these are true, we don't even need Go installed to build.). Because without go you probably can't verify that the vendored files are matching the checksum of hugo.sum unless hugo has this check reimplemented. Having a vendored build without requiring go would indeed be a plus for many. Wait, this doesn't makes sense... If vendoring is used then go.sum's checksums don't matter...
And yes, picking hugo.mod or go.mod could probably also be autodiscovered in hugo code.
I haven't yet looked into whether hugo modules could also only deliver hugo.mod files instead of go.mod. If hugo modules can also only provide hugo.mod, we even can switch to hugo.mod fully and use go.mod as compatibility fallback.
Since go 1.18 the -d in go get is always set in module mode and thus not set above
Having no go.mod makes this full process pretty unusable. This is the part where we would need to voice our use case upstream. AFAICS this currently makes the modfile switch only possible if you actually have a separate use of the go.mod file (This is probably due A file named go.mod must still be present in order to determine the module root directory, but it is not accessed. from the release notes). hugo mod init could always write an empty go.mod file though. Without, you can only get this far:

$ go mod init -modfile=hugo.mod example.com/m
go: creating new go.mod: module example.com/m
go: to add module requirements and sums:
        go mod tidy
$ go get -modfile=hugo.mod github.com/McShelby/hugo-theme-relearn
flag provided but not defined: -modfile
usage: go get [-d] [-f] [-t] [-u] [-v] [-fix] [build flags] [packages]
Run 'go help get' for details.

Once you have an empty go.mod file, this full process works.

But this was overall the idea behind this concrete feature request on why hugo can make -modfile work for end site repos if chosen to do so by the developer. I'll look into it whether hugo modules can provide only a hugo.mod "technically", but I believe go.mod is an integral part of the go module discovery framework.

What do you think? You have more insight in hugo on what might be an issue to this idea since I still don't know hugo very well.

dionysius commented 1 year ago

The possibilities are:

switching to hugo.mod autonomously by choice of hugo devs (if hugo modules are in use at all)
- new projects: creating empty go.mod if missing due to technical reasons
- existing projects: migrating go.mod to hugo.mod by renaming it, still create empty go.mod due to technical reasons
- probably a sanity check required in case someone still has "workarounded "the hugo spec
switching to hugo.mod by site developer choice by introducing a config property:
- either a bool property to use separate .mod file (hugo.mod constant)
- or a string property so the full file name can be specified (file extension should probably still be .mod)
not offering this option at all and thus the previous spec stays that hugo modules should not be combine with go modules
- Might be feasable since the noise turned relatively low in the initial issue

I also checked the dependency part for remote import paths. There is no indication that defines which file is the modules file. There seems to be no need for a go.mod when requesting modules:

$ go mod init -modfile=hugo.mod example.com/m
go: creating new go.mod: module example.com/m
go: to add module requirements and sums:
        go mod tidy

$ go get -v -modfile=hugo.mod github.com/dionysius/vaultwarden-deb
go: downloading github.com/dionysius/vaultwarden-deb v0.0.0-20230117144316-070fd9e94cd6
go: added github.com/dionysius/vaultwarden-deb v0.0.0-20230117144316-070fd9e94cd6

$ go mod download -modfile=hugo.mod

$ cat /home/dionysius/Projects/go/pkg/mod/cache/download/github.com/dionysius/vaultwarden-deb/@v/v0.0.0-20230117144316-070fd9e94cd6.mod
module github.com/dionysius/vaultwarden-deb

$ cat hugo.mod 
module example.com/m

go 1.19

require github.com/dionysius/vaultwarden-deb v0.0.0-20230117144316-070fd9e94cd6 // indirect

$ ls -1 $(go list -m -f '{{.Dir}}' -modfile=hugo.mod github.com/dionysius/vaultwarden-deb)
debian
LICENSE
README.md
signing-key.pub

goproxy logs these requests:

goproxy.io: 0.018s 404 /github.com/@v/list
goproxy.io: 0.019s 404 /github.com/dionysius/@v/list
goproxy.io: 1.128s 200 /github.com/dionysius/vaultwarden-deb/@v/list
goproxy.io: 0.387s 200 /github.com/dionysius/vaultwarden-deb/@latest
goproxy.io: 0.011s 200 /github.com/dionysius/vaultwarden-deb/@v/v0.0.0-20230117144316-070fd9e94cd6.mod
goproxy.io: 0.010s 200 /github.com/dionysius/vaultwarden-deb/@v/v0.0.0-20230117144316-070fd9e94cd6.zip

The question is though, what if this dependency now has another dependency? Going this route would need an extra step of parsing transitive dependencies and adding them to the site's hugo.mod, since go mod doesn't give this part automatically anymore.

Assuming this dependency had a hugo.mod file, we possibly can list their dependencies with help of -modfile again, but within the go pkg cache (to be tested on another day). In theory:

$ cd $(go list -m -f '{{.Dir}}' -modfile=hugo.mod github.com/dionysius/vaultwarden-deb)

$ go list -m -json -modfile=hugo.mod ...
# should technically list all their dependencies

And that list of transitive dependencies could be added to the site's hugo.mod file by hugo. But the next question would be the graph of modules, this would now need to be printed from hugo itself by going through all the hugo.mod in those dependencies recursively.

IMO going the hugo.mod route for dependencies is a bit too much freestyle and I would stick to go.mod. But it was interesting to dig through.

bep commented 1 year ago

Thanks for the writeup. The one thing that's not clear from the above, which needs to be tested is:

$ go get -modfile=hugo.mod github.com/McShelby/hugo-theme-relearn
go: added github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1
$ cat hugo.mod hugo.sum 
module example.com/m

go 1.19

require github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1 // indirect
github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1 h1:hGJYI84Tqtt5KMJlzyXH08ctE3M2Ygh1Tk4Mr0uYhfs=
github.com/McShelby/hugo-theme-relearn v0.0.0-20230303193005-e619dabc51a1/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=

$ go mod download -modfile=hugo.mod -json
{
        "Path": "github.com/McShelby/hugo-theme-relearn",
        "Version": "v0.0.0-20230303193005-e619dabc51a1",
        "Info": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.info",
        "GoMod": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.mod",
        "Zip": "/home/dionysius/Projects/go/pkg/mod/cache/download/github.com/!mc!shelby/hugo-theme-relearn/@v/v0.0.0-20230303193005-e619dabc51a1.zip",
        "Dir": "/home/dionysius/Projects/go/pkg/mod/github.com/!mc!shelby/hugo-theme-relearn@v0.0.0-20230303193005-e619dabc51a1",
        "Sum": "h1:hGJYI84Tqtt5KMJlzyXH08ctE3M2Ygh1Tk4Mr0uYhfs=",
        "GoModSum": "h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM="
}

In the above:

The local project has a hugo.mod file
hugo-theme-relearn has a go.mod file which is resolved correctly.

But what if hugo-theme-relearn had a hugo.mod file`?

Also, the short answer to why Hugo has its own vendoring logic:

Go's vendor tool (and some others) operate on Go packages and and only vendor the Go packages in use (which obviously doesn't work for us)
Also, when Go zips a module, it ignores vendor folders. We respect transitive vendoring, so to speak, so we name the folder _vendor to allow it to survive.

I have pulled the proposal label off of this issue. I'm not sure it's doable, but it's well worth it if it is.

dionysius commented 1 year ago

Thank you. Yes ultimatly I tried to find out how you would want it. It sounds like a constistent hugo.mod file across the dependency tree is the wished goal of hugo.

Can you maybe summerize for me how hugo currently accesses the module data? Is it the go pkg cache and the custom _vendor folder as assumed above or are there some more steps I should know of?

bep commented 1 year ago

Well,

The logic is here: https://github.com/gohugoio/hugo/tree/master/modules

The main gist of it is:

We build the "module tree" based on module imports (transitive)
We use go commands for everything, but have built some wrappers because Go doesn't know about the above. E.g. the Tidy command: https://github.com/gohugoio/hugo/blob/master/modules/client.go#L177 -- running go mod tidy would just remove everything.

I will soonish do some experiments to see if the above is possible.

bep commented 1 year ago

I did some quick testing in #10807 -- and it fails on go mod download.

go mod download -modfile=hugo.mod 
go: cannot find main module, but -modfile was set.
    -modfile cannot be used to set the module root directory.

dionysius commented 1 year ago

Yes, you'd have to workaround

A file named go.mod must still be present in order to determine the module root directory, but it is not accessed.

So for a split second, create an empty go.mod file (it can really be completely empty), do the go mod download, and while it's at it, remove go.mod again, maybe after successfully invoking the command +50/100ms or so

dionysius commented 1 year ago

See https://github.com/bep/hugo/pull/195

First you had trailing space. Added a hacky way to temporary add a go.mod.

Now it's hanging in go list, because -modfile needs to come before all. Go now thinks to read the real go.mod, because it ignores the -modfile argument. The position of -modfile depends in each command, so you'd have to add it per command specifically.

go: no module dependencies to download
go: no module declaration in go.mod. To specify the module path:
    go mod edit -module=example.com/mod
--- FAIL: TestHugoModFile (0.07s)
    c:\Users\Dionysius\Projects\github.com\dionysius\hugo\modules\integrationtest_builder.go:329: 
        error:
          got non-nil error
        got:
          e"failed to list modules: failed to execute 'go [list -m -json all -modfile=hugo.mod]': failed to execute binary \"go\" with args [list -m -json all -modfile=hugo.mod]: go: no module declaration in go.mod. To specify the module path:\n\tgo mod edit -module=example.com/mod\n *errors.errorString"

dionysius commented 1 year ago

If you want, I can continue this branch for a bit, since I have a simple testcase available now

bep commented 1 year ago

Please do, but I suggest you just create a new branch/PR (make it a draft so it gets less noisy) so you can push directly to it. Also, you don't need to spend time on making it "pretty" -- the important part is to demonstrate that it's possible.

bep commented 1 year ago

Also note that the convincing test case would be if you could get that test case to also handle the hugo.mod change here:

https://github.com/bep/hugo-mod-with-hugodotmod/commit/c3e39ab6de5bb0763a5f5e80f03dc02fdc365eb2

Which would require you to change path to github.com/bep/hugo-mod-with-hugodotmod/v6 and the assertion from bar => baz.

dionysius commented 1 year ago

Have a look https://github.com/gohugoio/hugo/pull/10808 and I added my notes there as well

yann-soubeyrand commented 1 year ago

Hello,

I understand that people using Hugo have to play with its rules and I’m absolutely not contesting this fact. However, it seems to me that getting Hugo and Go play nicely together would allow for interesting use cases to be realized. One such use case is a Go project consisting of several assembled modules: one could build the documentation for this project by gathering the documentation included in these modules. The documentation would always be in sync with the code.

I didn’t think that much about a way to solve this use case, but it seems to me that if Hugo could manage a hugo.go file consisting only of the naked imports of the packages containing content, it could solve the conflict with go mod tidy and go mod vendor.

dionysius commented 1 year ago

As discussed in #10808 we skip the hard parts and focus on the main project repo first.

In my opinion, as a go developer, a modfile property feels better than a hugo.mod. But my use case would be satisfied with either option. The required folder structure of hugo would not make it possible to have two hugo sites in the same folder anyway.

In the end those options remain to be decided by hugo (from https://github.com/gohugoio/hugo/issues/10801#issuecomment-1458621992, bold are changed/improved/added):

The possibilities are:

switching to hugo.mod autonomously by choice of hugo devs (if hugo modules are in use at all)

new projects: creating temporary empty go.mod if missing due to technical reasons

existing projects: migrating go.mod to hugo.mod by renaming it, still create temporary empty go.mod due to technical reasons

probably a sanity check required in case someone already has "workarounded" the current hugo spec

switching to hugo.mod by site developer choice by introducing a config property (and cli property on hugo mod init):

either a bool property to use separate .mod file (hugo.mod constant)

or a string property so the full file name can be specified (file extension should probably still be .mod)

we could not do any temporary go.mod: Whoever turns on this feature, knows what they are doing and have read the documentation about this option. Having the temporary go.mod would make it a bit less error prone for hugo and more convenient for the developer though (less issue reports if used wrongly)

not offering this option at all and thus the previous spec stays that hugo modules should not be combine with go modules

Might be feasable since the noise turned relatively low in the initial issue

I think once above is decided, anyone can work on a PR.

Note to myself: I've discovered how/where hugo checks out dependencies, which I tried to ask before https://github.com/gohugoio/hugo/issues/10801#issuecomment-1460237735:

hugo seems to check out dependencies to /tmp/hugo_cache/modules

bep commented 1 year ago

hugo seems to check out dependencies to /tmp/hugo_cache/modules

See https://gohugo.io/getting-started/configuration/#configure-file-caches

If you set the HUGO_CACHEDIR env var to something outside /tmp, I suspect you will get an easier debug situation.

In my opinion, as a go developer, a modfile property feels better than a hugo.mod. But my use case would be satisfied with either option.

Yea, we can add a modfile option here, which would make it behave similar to the workspace option -- and make sure to document that it only works for the main Hugo project (which is where you run hugo from).

https://gohugo.io/hugo-modules/configuration/

gohugoio / hugo

Concrete go + hugo modules conflict proposal: Add modfile property also for hugo #10801