coolaj86 commented 5 years ago

I'm just wondering if anyone has a good tip on how to migrate from gb to the newly landed go modules (previously vgo).

There seem to be a few important steps:

[x] convert vendor/manifest
[x] create go.mod
[x] rewrite import paths

rewrite import paths

Update: I did this (down in the src folder, excluding .git):

find . -type f -print0 | xargs -0 sed -i '' 's:"modulename/:"awesome.io/secret/sauce/modulename/:g'

I'm working in a project with 255 files and they're all using gb src-relative paths. I've been trying to figure out some automated way to rewrite the import paths to absolute project paths. So far, no dice.

convert vendor/manifest to go get

I'm actually just starting to use go (the big project is something I've inherited), so it was quicker for me to write this in node.

convert-manifest-to-go-mod.js:

'use strict';

// How to use private repos that require authentication:
// https://github.com/golang/go/issues/26134#issuecomment-403885803

var fs = require('fs');
var manifest = JSON.parse(fs.readFileSync('./vendor/manifest', 'utf8'));
manifest.dependencies.forEach(function (dep) {
  console.log('go get ' + dep.importpath + '@' + dep.revision);
});

touch go.mod
node convert-manifest-to-go-mod.js > go-get.sh
bash go-get.sh

If you need a quick node installer to run that: https://git.coolaj86.com/coolaj86/node-installer.sh

bash <(curl -fsSL bit.ly/node-installer)

create go.mod

You can't use go mod init with gb projects. However, it's fairly easy to work around that.

The go.mod file needs an import (possibly something like github.com/username/project, though I prefer to use my own domain)

go.mod:

module example.xyz/go/mypkg

require (
)

You also need to add an import comment to the package <name> declarations:

-package main
+package main // import example.xyz/go/mypkg

Here's where things get messy: go build doesn't support relative import paths and gb build appears to not have a way of specifying an absolute package, so it seems like I can't do the transformation gracefully a little at a time.

tianon commented 5 years ago

I just did go mod init in a GB-using directory and it automatically picked up vendor/manifest and created a mostly reasonable go.mod for me (at least as much as it can given that vendor/manifest doesn't persist things like tags by default):

$ GO111MODULE=on go mod init
go: creating new go.mod: module foobar
go: copying requirements from vendor/manifest

coolaj86 commented 5 years ago

I do this

$ pushd /Users/me/awesome.io/secret/sauce
$ ls
src/
vendor/
static/
bin/
README.md

$ GO111MODULE=on go mod init

go: cannot determine module path for source directory /Users/me/awesome.io/secret/sauce (outside GOPATH, no import comments)

(note: there's some other files that show up in ls that I don't think are relevant - docker, circleci, etc, but this is an inherited codebase, so I don't know too much yet)

I remedy with this:

$ GOPATH=/Users/me
$ export GOPATH

src/sauce/main.go

package sauce // import "awesome.io/secret/sauce"

I get the same result when I rerun.

Now, I don't even know how gb build knows which files to build or where they are.

tianon commented 5 years ago

If you don't understand how gb currently builds your code, then I'd wager you're going to have a very difficult time understanding how to switch from gb over to Go modules. Perhaps you need to spend a little more time with the source you're working with first to understand how it works / what it's doing? I'd also recommend spending some time learning how gb works (it's not very complex at a high level if you understand how go build works, but it is a little difference since it is a way to avoid the GOPATH on versions of Go that didn't support modules).

It might also help if you tried a fresh (small) project to get your head wrapped around Go modules before trying to perform this conversion. I'm not a maintainer of gb (just an interested contributor), but I don't feel like there's really anything for gb itself to do here (especially since go mod init already supports gb's vendor/manifest file format for conversion).

coolaj86 commented 5 years ago

This works fine

gb build

And I am familiar with the current code. And I know how to use go's "normal" way of doing things with abs paths and such. It's important to me to switch the imports over so that I don't have to fight against gb (which, for all the great good it has provided, appears to be unmaintained and is probably not particularly necessary moving forward) for go test, godoc, etc, etc.

It seems like gb has brought a lot of value to a lot of people for a long time. I can only imagine it contributed significantly to the community discussion and tooling that has lead to go modules proper.

Could you point me to some small sample projects? I was looking around for quite a bit before I posted here.

tianon commented 5 years ago

Sure, just for the sake of illustration (and because it's something I've been meaning to do), I've just converted https://github.com/tianon/gosleep from using gb to using Go modules:

https://github.com/tianon/gosleep/commit/af5b6562f838eb9e620158ca4525091712d69a94

The only steps I did for the actual conversion were the following:

mv src/gosleep/*.go ./ (so our code isn't using the gb-introduced src/ directory anymore which hides the code from Go's default search paths)
update main.go to include package main // import "github.com/tianon/gosleep" (so go mod knows where we're supposed to be without a GOPATH)

$ GO111MODULE=on go mod init
go: creating new go.mod: module github.com/tianon/gosleep
go: copying requirements from vendor/manifest
$ GO111MODULE=on go mod verify
go: finding github.com/jessevdk/go-flags v0.0.0-20170212220246-460c7bb0abd6
go: finding github.com/jinzhu/now v0.0.0-20170212112655-d939ba741945
go: finding go.tianon.xyz/progress v0.0.0-20170120223140-598c27a6e143
go: finding golang.org/x/crypto v0.0.0-20170508154713-122d919ec1ef
all modules verified

as an extra step, I updated go.mod to reference github.com/jessevdk/go-flags via the v1.4.0 release version (instead of the commit vendor/manifest was pointing to) and ran go mod tidy

(I also use Docker for building release binaries there, so you can see some Docker-related changes in that commit as well.)

coolaj86 commented 5 years ago

Thanks for that. This codebase was heavily steeped in src/ and with so many files and directories a simple mv src/modulename/*.go ./ wasn't the right approach.

FYI: this is what was the most helpful for me after trying govers and gofmt -w -r '"thing" -> "otherthing"', etc:

find . -type f -print0 | xargs -0 sed -i '' 's:"modulename/:"awesome.io/secret/sauce/modulename/:g'

Good old fashioned sed. Note the sed -i '' for macOS compat.

I put the leading " and the trailing / to disambiguate other occurrences of the module name in the code.

Now that everything compiles again I'll slowly get rid of src/ and refactor the code into separate module repos.

constabulary / gb

[SOLVED] How to migrate to go modules? #740

rewrite import paths

convert vendor/manifest to go get

create go.mod