search

u-root / gobusybox

Tools for compiling many Go commands into one binary to save space. Builds are supported for vendor-based Go and module-based Go
BSD 3-Clause "New" or "Revised" License
144 stars 21 forks source link

Add support for combining commands from unrelated Go modules #43

Closed jpeach closed 2 years ago

jpeach commented 3 years ago

I attempted to build a bb command for the Carvel toolchain, which is a set of independent git repositories, each with a different Go module.

I have a top-level go module where I pull in the vendor dependencies using a tools.go file:

$ cat go.mod
module gitlab.eng.vmware.com/jpeach/carvel

go 1.16

require (
    github.com/k14s/imgpkg v0.5.0
    github.com/k14s/ytt v0.31.0
    github.com/u-root/gobusybox/src v0.0.0-20200923191651-a55b339949b0 // indirect
    github.com/vmware-tanzu/carvel-vendir v0.17.0
    golang.org/x/mod v0.4.2 // indirect
    golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4 // indirect
    golang.org/x/tools v0.1.0 // indirect
)
// +build tools

package tools

import (
    _ "github.com/k14s/imgpkg/cmd/imgpkg"
    _ "github.com/k14s/kbld/cmd/kbld"
    _ "github.com/k14s/ytt/cmd/ytt"

    _ "github.com/u-root/gobusybox/src/cmd/makebb"
)

When I run makebb, the source appears to be rewritten correctly, but the modfile in the bb command doesn't contains the necessary requires directives.

$ go run github.com/u-root/gobusybox/src/cmd/makebb github.com/k14s/imgpkg/cmd/imgpkg github.com/k14s/ytt/cmd/ytt
2021/03/23 07:23:51 Disabling CGO for u-root...
2021/03/23 07:23:51 Build environment: GOARCH=amd64 GOOS=darwin GOPATH=/Users/jpeach/go CGO_ENABLED=0 GO111MODULE= GOROOT=/usr/local/Cellar/go/1.16.2/libexec PATH=/usr/local/Cellar/go/1.16.2/libexec/bin:$PATH
2021/03/23 07:23:53 go build with modules failed: error building go package in "/var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-324558004/src/bb.u-root.com/bb": main.go:14:2: module github.com/k14s/imgpkg provides package github.com/k14s/imgpkg/cmd/imgpkg and is replaced but not required; to add it:
    go get github.com/k14s/imgpkg
main.go:15:2: module github.com/k14s/ytt provides package github.com/k14s/ytt/cmd/ytt and is replaced but not required; to add it:
    go get github.com/k14s/ytt
, exit status 1
2021/03/23 07:23:53 Preserving bb generated source directory at /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-324558004 due to error. To debug build, `cd /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-324558004/src/bb.u-root.com/bb` and use `go build` to build, or `go mod [why|tidy|graph]` to debug dependencies, or `go list -m all` to list all dependency versions.
exit status 1

The generated bb source references the packages correctly:

$ cat /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-324558004/src/bb.u-root.com/bb/main.go
// Copyright 2018 the u-root Authors. All rights reserved
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

// Package main is the busybox main.go template.
package main

import (
    "log"
    "os"
    "path/filepath"

    "bb.u-root.com/bb/pkg/bbmain"
    _ "github.com/k14s/imgpkg/cmd/imgpkg"
    _ "github.com/k14s/ytt/cmd/ytt"
    // There MUST NOT be any other dependencies here.
    //
    // It is preferred to copy minimal code necessary into this file, as
    // dependency management for this main file is... hard.
)
...

The source for each command package looks correctly rewritten and calls bbmain.Register.

But the go.mod only contains the replacements, not the requires, which are are also necessary because the bb command is consuming vendor packages.

$ cat /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-324558004/src/bb.u-root.com/bb/go.mod
module bb.u-root.com/bb
replace github.com/k14s/imgpkg => ../../github.com/k14s/imgpkg

replace github.com/k14s/ytt => ../../github.com/k14s/ytt

I tried working around this with some makefile hackery (i.e. running go get manually to add the requires). This seems to work.

jpeach commented 3 years ago

I also tries this using local checkouts of the tools, with the same result:

$ git clone http://github.com/k14s/ytt
...
$ git clone https://github.com/k14s/imgpkg
...
$ makebb ./imgpkg/cmd/imgpkg ./ytt/cmd/ytt
2021/03/23 07:34:12 Disabling CGO for u-root...
2021/03/23 07:34:12 Build environment: GOARCH=amd64 GOOS=darwin GOPATH=/Users/jpeach/go CGO_ENABLED=0 GO111MODULE= GOROOT=/usr/local/Cellar/go/1.16.2/libexec PATH=/usr/local/Cellar/go/1.16.2/libexec/bin:$PATH
2021/03/23 07:34:18 go build with modules failed: error building go package in "/var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-893762319/src/bb.u-root.com/bb": main.go:14:2: module github.com/k14s/imgpkg provides package github.com/k14s/imgpkg/cmd/imgpkg and is replaced but not required; to add it:
    go get github.com/k14s/imgpkg
main.go:15:2: module github.com/k14s/ytt provides package github.com/k14s/ytt/cmd/ytt and is replaced but not required; to add it:
    go get github.com/k14s/ytt
, exit status 1
2021/03/23 07:34:18 Preserving bb generated source directory at /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-893762319 due to error. To debug build, `cd /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-893762319/src/bb.u-root.com/bb` and use `go build` to build, or `go mod [why|tidy|graph]` to debug dependencies, or `go list -m all` to list all dependency versions.
hugelgupf commented 3 years ago

What Go version are you using?

Can you try 1.15?

jpeach commented 3 years ago

This does work with Go 1.15:

$ GOROOT=$(go env GOROOT) makebb ./imgpkg/cmd/imgpkg ./ytt/cmd/ytt
2021/03/23 12:58:28 Disabling CGO for u-root...
2021/03/23 12:58:28 Build environment: GOARCH=amd64 GOOS=darwin GOPATH=/Users/jpeach/go CGO_ENABLED=0 GO111MODULE= GOROOT=/usr/local/Cellar/go@1.15/1.15.10/libexec PATH=/usr/local/Cellar/go@1.15/1.15.10/libexec/bin:$PATH
$ ./bb imgpkg --help
imgpkg stores files as Docker images (copy, pull, push, tag, version)
...
$ ./bb ytt --help
ytt performs YAML templating.
...

Verified that Go 1.16 doesn't:

$ GOROOT=$(/usr/local/bin/go env GOROOT) makebb ./imgpkg/cmd/imgpkg ./ytt/cmd/ytt
2021/03/23 13:00:52 Disabling CGO for u-root...
2021/03/23 13:00:52 Build environment: GOARCH=amd64 GOOS=darwin GOPATH=/Users/jpeach/go CGO_ENABLED=0 GO111MODULE= GOROOT=/usr/local/Cellar/go/1.16.2/libexec PATH=/usr/local/Cellar/go/1.16.2/libexec/bin:$PATH
2021/03/23 13:00:55 go build with modules failed: error building go package in "/var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-756218030/src/bb.u-root.com/bb": main.go:14:2: module github.com/k14s/imgpkg provides package github.com/k14s/imgpkg/cmd/imgpkg and is replaced but not required; to add it:
    go get github.com/k14s/imgpkg
main.go:15:2: module github.com/k14s/ytt provides package github.com/k14s/ytt/cmd/ytt and is replaced but not required; to add it:
    go get github.com/k14s/ytt
, exit status 1
2021/03/23 13:00:55 Preserving bb generated source directory at /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-756218030 due to error. To debug build, `cd /var/folders/d9/tpzrvxbs7s98zqy4c5yq8bk40000gr/T/bb-756218030/src/bb.u-root.com/bb` and use `go build` to build, or `go mod [why|tidy|graph]` to debug dependencies, or `go list -m all` to list all dependency versions.
hugelgupf commented 3 years ago

Ah neat! At least it's an explanation. Thx for checking.

jpeach commented 3 years ago

Lightly tested patch that works for one case I tried :)

diff --git src/go.mod src/go.mod
index 223bdb2..1dd5503 100644
--- src/go.mod
+++ src/go.mod
@@ -5,6 +5,7 @@ go 1.13
 require (
    github.com/google/goterm v0.0.0-20200907032337-555d40f16ae2
    github.com/u-root/u-root v7.0.0+incompatible
+   golang.org/x/mod v0.4.2 // indirect
    golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
    golang.org/x/tools v0.0.0-20200904185747-39188db58858
 )
diff --git src/go.sum src/go.sum
index 639e4a6..bfdf56e 100644
--- src/go.sum
+++ src/go.sum
@@ -8,6 +8,8 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=
+golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
diff --git src/pkg/bb/bb.go src/pkg/bb/bb.go
index e69d301..d5e5434 100644
--- src/pkg/bb/bb.go
+++ src/pkg/bb/bb.go
@@ -28,10 +28,12 @@ import (
    "io/ioutil"
    "log"
    "os"
+   "path"
    "path/filepath"
    "strings"

    "github.com/google/goterm/term"
+   "golang.org/x/mod/modfile"
    "golang.org/x/tools/go/ast/astutil"
    "golang.org/x/tools/go/packages"

@@ -199,6 +201,9 @@ func BuildBusybox(opts *Opts) (nerr error) {
    if opts.Env.GO111MODULE == "off" || numNoModule > 0 {
        opts.Env.GOPATH = tmpDir
    }
+   if err := opts.Env.TidyDir(bbDir); err != nil {
+       return err
+   }
    if err := opts.Env.BuildDir(bbDir, opts.BinaryPath, opts.GoBuildOpts); err != nil {
        if opts.Env.GO111MODULE == "off" || numNoModule > 0 {
            return &ErrGopathBuild{
@@ -479,16 +484,30 @@ func dealWithDeps(env golang.Environ, bbDir, tmpDir, pkgDir string, mainPkgs []*
        //
        // The module name is something that'll never be online, lest Go
        // decides to go on the internet.
-       content := `module bb.u-root.com/bb`
+       mod := modfile.File{}
+
+       mod.AddModuleStmt(`bb.u-root.com/bb`)
+
        for _, mpath := range localModules {
-           content += fmt.Sprintf("\nreplace %s => ../../%s\n", mpath, mpath)
+           if err := mod.AddRequire(mpath, "v1.0.0"); err != nil {
+               return err
+           }
+
+           if err := mod.AddReplace(mpath, "v1.0.0", path.Join("..", "..", mpath), ""); err != nil {
+               return err
+           }
+       }
+
+       content, err := mod.Format()
+       if err != nil {
+           return nil
        }

        // TODO(chrisko): add other go.mod files' replace and exclude
        // directives.
        //
        // Warn the user if they are potentially incompatible.
-       if err := ioutil.WriteFile(filepath.Join(bbDir, "go.mod"), []byte(content), 0755); err != nil {
+       if err := ioutil.WriteFile(filepath.Join(bbDir, "go.mod"), content, 0755); err != nil {
            return err
        }
        return nil
diff --git src/pkg/golang/build.go src/pkg/golang/build.go
index a527c0d..12451ef 100644
--- src/pkg/golang/build.go
+++ src/pkg/golang/build.go
@@ -154,6 +154,19 @@ func (b *BuildOpts) RegisterFlags(f *flag.FlagSet) {
    f.Var(&arg, "go-extra-args", "Extra args to `go build`")
 }

+func (c Environ) TidyDir(dirPath string) error {
+   args := []string{"mod", "tidy"}
+
+   cmd := c.GoCmd(args...)
+   cmd.Dir = dirPath
+
+   if o, err := cmd.CombinedOutput(); err != nil {
+       return fmt.Errorf("error tidying modules in go package in %q: %v, %v", dirPath, string(o), err)
+   }
+
+   return nil
+}
+
 // BuildDir compiles the package in the directory `dirPath`, writing the build
 // object to `binaryPath`.
 func (c Environ) BuildDir(dirPath string, binaryPath string, opts *BuildOpts) error {
hugelgupf commented 3 years ago

Hey @jpeach, I just merged support for Go 1.16.

At least 

$ ./src/cmd/makebb/makebb ./ytt/cmd/ytt ./imgpkg/cmd/imgpkg

works for me, let me know how it works for you.

jpeach commented 3 years ago

works for me, let me know how it works for you.

thanks a lot, I'll try to verify soon :)

hugelgupf commented 2 years ago

I'm gonna assume this is working for you at the moment :) Open a new issue if you have new problems!