search

kolide / fleet

A flexible control server for osquery fleets
https://kolide.com/fleet
MIT License
1.1k stars 264 forks source link

make build broken on 3.0.0 and higher #2287

Closed jalseth closed 3 years ago

jalseth commented 3 years ago

The builds fail on 3.0.0 and higher. This appears to be due to the addition to the CGO_ENABLED=0 environment variable added to all of the build commands. If I remove this, the build is successful.

What version of fleet are you using (fleet version --full)?

N/A

What operating system are you using?

Tested on macOS Catalina 10.15.6 with Golang 1.15.0 and Ubuntu 20.04 with Golang 1.14.2

What did you do?

$ make deps
$ make generate
$ make build

What did you expect to see?

Successful build.

What did you see instead?

$ make build
mkdir -p build/linux
mkdir -p build/darwin
CGO_ENABLED=0 go build -tags full -i -o build/fleet -ldflags " -X github.com/kolide/kit/version.appName=fleet -X github.com/kolide/kit/version.version=3.0.0 -X github.com/kolide/kit/version.branch=HEAD -X github.com/kolide/kit/version.revision=0058d459022df132df36556c40c342958639501c -X github.com/kolide/kit/version.buildDate=2020-08-19T19:23:52Z -X github.com/kolide/kit/version.buildUser=james -X github.com/kolide/kit/version.goVersion=go1.15" ./cmd/fleet
go build net: copying /Users/james/Library/Caches/go-build/0b/0bea6c310e5c2fb4021cd6e334c52f7b692fea5ff47dcb3af72b710843a8afc1-d: open /opt/local/lib/go/pkg/darwin_amd64/net.a: permission denied
go build os/user: copying /Users/james/Library/Caches/go-build/1f/1fc0c83ac80ef7f361f8f91ae0828b4b8a43f73a00b4dfb253c529074e30a990-d: open /opt/local/lib/go/pkg/darwin_amd64/os/user.a: permission denied
make: *** [fleet] Error 1
zwass commented 3 years ago

Is this with Go 1.15? I've not seen such issues before. Maybe because I have GOPATH set?

jalseth commented 3 years ago

As noted in the issue, I tried both Go 1.15 and 1.14 on macOS and Ubuntu respectively.

This is from a fresh Ubuntu 18.04 VM. I have skipped the output on the successful steps.

$ sudo apt update && sudo apt upgrade -y && sudo apt install -y git npm
$ sudo npm install -g yarn
$ curl -L -o go.tar.gz https://golang.org/dl/go1.14.7.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go.tar.gz
$ export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin
$ mkdir -p go/src/github.com/kolide
$ cd go/src/github.com/kolide
$ git clone https://github.com/kolide/fleet
$ cd fleet
$ git checkout tags/3.1.0
$ go version
go version go1.14.7 linux/amd64
$ make deps
$ make generate
$ make fleet
mkdir -p build/linux
mkdir -p build/darwin
CGO_ENABLED=0 go build -tags full -i -o build/fleet -ldflags " -X github.com/kolide/kit/version.appName=fleet -X github.com/kolide/kit/version.version=3.1.0 -X github.com/kolide/kit/version.branch=HEAD -X github.com/kolide/kit/version.revision=c6ce648fef3bb39b6e604333ec47cff0e625ff8e -X github.com/kolide/kit/version.buildDate=2020-08-20T16:11:05Z -X github.com/kolide/kit/version.buildUser=vagrant -X github.com/kolide/kit/version.goVersion=go1.14.7" ./cmd/fleet
go build net: copying /tmp/go-build824086985/b073/_pkg_.a: open /usr/local/go/pkg/linux_amd64/net.a: permission denied
go build os/user: copying /tmp/go-build824086985/b290/_pkg_.a: open /usr/local/go/pkg/linux_amd64/os/user.a: permission denied
Makefile:110: recipe for target 'fleet' failed
make: *** [fleet] Error 1

If I roll back to 2.6.0 where CGO_ENABLED is not specified, it works fine.

$ git checkout tags/2.6.0
$ make deps
$ make generate
$ make fleet
mkdir -p build/linux
mkdir -p build/darwin
go build -i -o build/fleet -ldflags " -X github.com/kolide/kit/version.appName=fleet -X github.com/kolide/kit/version.version=2.6.0 -X github.com/kolide/kit/version.branch=HEAD -X github.com/kolide/kit/version.revision=d3849d8fe1ea841d34a19e5cc06bfe62fc8dfd2d -X github.com/kolide/kit/version.buildDate=2020-08-20T16:14:37Z -X github.com/kolide/kit/version.buildUser=vagrant -X github.com/kolide/kit/version.goVersion=go1.14.7" ./cmd/fleet
$ file build/fleet
build/fleet: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, not stripped

Similarly, if I just remove that from the 3.1.0 build it works fine.

$ git checkout tags/3.1.0
$ sed -i 's/CGO_ENABLED=0 //g' Makefile
$ make deps
$ make generate
$ make fleet
$ echo $?
0

Edit: Setting export GOPATH=$HOME/go made no difference.

jalseth commented 3 years ago

Hmm, interestingly make xp-fleet still fails with a similar error where it's trying to write to system directories. And it also fails when I revert to 2.6.0. Output:

$ export GOPATH=$HOME/go
$ make xp-fleet
rm -rf build/binary-bundle
mkdir -p build/binary-bundle/linux
mkdir -p build/binary-bundle/darwin
mkdir -p build/linux
mkdir -p build/darwin
NODE_ENV=production webpack --progress --colors
 67% building 736/769 modules 33 active ...thub.com/kolide/fleet/node_modules/react-select/dist/react-select.cssBrowserslist: caniuse-lite is outdated. Please run next command `yarn upgrade`
Hash: 0a42143e4037a6959d07
Version: webpack 4.37.0
Time: 17535ms
Built at: 08/20/2020 6:23:32 PM
                                                          Asset       Size  Chunks                    Chunk Names
                               ../frontend/templates/react.tmpl   7.13 KiB          [emitted]
                       404@f3676fd679816c107348ed2737a4897c.svg     71 KiB          [emitted]
                       500@b18abd32b7900af901327c55495fb0c7.svg   21.9 KiB          [emitted]
               Oxygen-Bold@3d91dc525b7e4b7c8a8a2d94c9e12626.eot     48 KiB          [emitted]
              Oxygen-Bold@9a9cf40f37a1d18e192986b32f5df71d.woff   27.9 KiB          [emitted]
               Oxygen-Bold@cb22d418b93310fe79ea10552fed6995.ttf   47.7 KiB          [emitted]
              Oxygen-Light@3c857e0ee7f444de09e3b8058c02d052.eot   43.1 KiB          [emitted]
              Oxygen-Light@703fea81b1d774e3bce8d17dbe013e7f.ttf   42.8 KiB          [emitted]
             Oxygen-Light@cb562c374a247dc4adc24cb77d0c1f4b.woff   25.3 KiB          [emitted]
            Oxygen-Regular@6ec330b4917001db3f6c80c23ac12817.ttf     47 KiB          [emitted]
            Oxygen-Regular@984776b3bc41069562206a96f25b65ef.eot   47.3 KiB          [emitted]
           Oxygen-Regular@d672de176f1d1f5e44a7fb2f14583458.woff   27.6 KiB          [emitted]
SourceCodePro-Regular.ttf@b6ba243267725a84615cfaba137a6f55.woff   93.6 KiB          [emitted]
     SourceCodePro-Regular@1066e54d79f902a2ff4d864c9a38a183.ttf    208 KiB          [emitted]
     SourceCodePro-Regular@d5892edc24a4ec262052ebb4dcb3fec8.eot   87.9 KiB          [emitted]
            avatar-default@23086def317630fcdd4fddddd5ca37f5.png   13.3 KiB          [emitted]
                background@c54a7bdedd884842ffdbd97a0d5a9590.png    353 KiB          [emitted]  [big]
                                 bundle-0a42143e4037a6959d07.js   1.94 MiB       0  [emitted]  [big]  bundle
                                bundle-b7c4bf979c7f1906fbc9.css    153 KiB       0  [emitted]         bundle
               footer-logo@bd8b92e34e99f955afdd993acf667060.svg   2.53 KiB          [emitted]
     kolide-logo-condensed@60a5addbf42470fd54a117be483ed491.svg   3.48 KiB          [emitted]
      kolide-logo-vertical@ad7afdd9f66ec8a6bfda8c1db00c00c8.svg    2.3 KiB          [emitted]
               kolide-logo@311a81c6f65bf0fbf98caf5770e1499c.svg  872 bytes          [emitted]
               kolidecons@1a543b8d9f51de6682fba5fe895c0985.woff   12.3 KiB          [emitted]
              kolidecons@4959f6a6f76ce3d39fc83b0f1b1d4894.woff2     10 KiB          [emitted]
               laptop-plus@50e51c3e80307960656836f2eb591c12.svg  513 bytes          [emitted]
       osquery-certificate@8b14a15c345627a0faf7eb0648403fe8.svg   8.48 KiB          [emitted]
               swoop-arrow@5a3a2402459ad28ca1f8f1de0ed29808.svg   1.65 KiB          [emitted]
Entrypoint bundle [big] = bundle-b7c4bf979c7f1906fbc9.css bundle-0a42143e4037a6959d07.js
  [4] (webpack)/buildin/global.js 472 bytes {0} [built]
 [12] ./frontend/router/paths.js 1.39 KiB {0} [built]
 [48] ./node_modules/react-router/es/index.js + 32 modules 74.9 KiB {0} [built]
      |    33 modules
 [77] ./frontend/router/url_prefix.js 259 bytes {0} [built]
[367] ./frontend/index.jsx 537 bytes {0} [built]
[372] ./frontend/public-path.js 358 bytes {0} [built]
[373] ./frontend/router/index.jsx 7.57 KiB {0} [built]
[401] ./frontend/pages/admin/AppSettingsPage/index.js 372 bytes {0} [built]
[466] ./frontend/pages/admin/UserManagementPage/index.js 387 bytes {0} [built]
[553] ./frontend/pages/packs/AllPacksPage/index.js 357 bytes {0} [built]
[574] ./frontend/pages/ConfirmInvitePage/index.js 382 bytes {0} [built]
[582] ./frontend/pages/ConfirmSSOInvitePage/index.js 397 bytes {0} [built]
[587] ./frontend/layouts/CoreLayout/index.js 347 bytes {0} [built]
[601] ./frontend/pages/packs/EditPackPage/index.js 357 bytes {0} [built]
[855] ./frontend/index.scss 39 bytes {0} [built]
    + 846 hidden modules

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets:
  background@c54a7bdedd884842ffdbd97a0d5a9590.png (353 KiB)
  bundle-0a42143e4037a6959d07.js (1.94 MiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  bundle (2.09 MiB)
      bundle-b7c4bf979c7f1906fbc9.css
      bundle-0a42143e4037a6959d07.js

WARNING in webpack performance recommendations:
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/
Child html-webpack-plugin for "../frontend/templates/react.tmpl":
     1 asset
    Entrypoint undefined = ../frontend/templates/react.tmpl
    [0] ./node_modules/html-webpack-plugin/lib/loader.js!./frontend/templates/react.ejs 7.49 KiB {0} [built]
    [2] (webpack)/buildin/global.js 472 bytes {0} [built]
    [3] (webpack)/buildin/module.js 497 bytes {0} [built]
        + 1 hidden module
Child mini-css-extract-plugin node_modules/css-loader/index.js!node_modules/postcss-loader/src/index.js!frontend/components/KolideAce/theme.css:
    Entrypoint mini-css-extract-plugin = *
       2 modules
Child mini-css-extract-plugin node_modules/css-loader/index.js!node_modules/postcss-loader/src/index.js!node_modules/rc-pagination/assets/index.css:
    Entrypoint mini-css-extract-plugin = *
       2 modules
Child mini-css-extract-plugin node_modules/css-loader/index.js!node_modules/postcss-loader/src/index.js!node_modules/react-select/dist/react-select.css:
    Entrypoint mini-css-extract-plugin = *
       2 modules
Child mini-css-extract-plugin node_modules/css-loader/index.js!node_modules/postcss-loader/src/index.js!node_modules/sass-loader/lib/loader.js??ref--7-3!node_modules/import-glob-loader/index.js!frontend/index.scss:
    Entrypoint mini-css-extract-plugin = *
     [0] ./assets/fonts/oxygen/Oxygen-Light.eot 95 bytes {0} [built]
     [1] ./assets/fonts/oxygen/Oxygen-Bold.eot 94 bytes {0} [built]
     [2] ./assets/fonts/oxygen/Oxygen-Regular.eot 97 bytes {0} [built]
     [3] ./assets/fonts/source-code-pro/SourceCodePro-Regular.eot 104 bytes {0} [built]
     [4] ./node_modules/css-loader!./node_modules/postcss-loader/src!./node_modules/sass-loader/lib/loader.js??ref--7-3!./node_modules/import-glob-loader!./frontend/index.scss 140 KiB {0} [built]
     [7] ./assets/fonts/oxygen/Oxygen-Light.woff 96 bytes {0} [built]
     [8] ./assets/fonts/oxygen/Oxygen-Light.ttf 95 bytes {0} [built]
     [9] ./assets/fonts/oxygen/Oxygen-Bold.woff 95 bytes {0} [built]
    [10] ./assets/fonts/oxygen/Oxygen-Bold.ttf 94 bytes {0} [built]
    [11] ./assets/fonts/oxygen/Oxygen-Regular.woff 98 bytes {0} [built]
    [12] ./assets/fonts/oxygen/Oxygen-Regular.ttf 97 bytes {0} [built]
    [13] ./assets/fonts/kolidecons/kolidecons.woff2 95 bytes {0} [built]
    [14] ./assets/fonts/kolidecons/kolidecons.woff 94 bytes {0} [built]
    [15] ./assets/fonts/source-code-pro/SourceCodePro-Regular.ttf.woff 109 bytes {0} [built]
    [16] ./assets/fonts/source-code-pro/SourceCodePro-Regular.ttf 104 bytes {0} [built]
        + 4 hidden modules
go-bindata -pkg=service \
    -o=server/service/bindata.go \
    frontend/templates/ assets/...
go-bindata -pkg=kolide -o=server/kolide/bindata.go server/mail/templates
GOOS=linux go build -i -o build/binary-bundle/linux/fleet -ldflags " -X github.com/kolide/kit/version.appName=fleet -X github.com/kolide/kit/version.version=2.6.0 -X github.com/kolide/kit/version.branch=HEAD -X github.com/kolide/kit/version.revision=d3849d8fe1ea841d34a19e5cc06bfe62fc8dfd2d -X github.com/kolide/kit/version.buildDate=2020-08-20T18:23:32Z -X github.com/kolide/kit/version.buildUser=vagrant -X github.com/kolide/kit/version.goVersion=go1.14.7" ./cmd/fleet
GOOS=darwin go build -i -o build/binary-bundle/darwin/fleet -ldflags " -X github.com/kolide/kit/version.appName=fleet -X github.com/kolide/kit/version.version=2.6.0 -X github.com/kolide/kit/version.branch=HEAD -X github.com/kolide/kit/version.revision=d3849d8fe1ea841d34a19e5cc06bfe62fc8dfd2d -X github.com/kolide/kit/version.buildDate=2020-08-20T18:23:32Z -X github.com/kolide/kit/version.buildUser=vagrant -X github.com/kolide/kit/version.goVersion=go1.14.7" ./cmd/fleet
go build internal/cpu: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build runtime/internal/atomic: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build internal/race: copying /home/vagrant/.cache/go-build/77/776c73be4e2fe961fd3dcccb2f406f3dc3188c5a7f37add755107b51cf1d6c80-d: open /usr/local/go/pkg/darwin_amd64/internal/race.a: no such file or directory
go build runtime/internal/sys: copying /home/vagrant/.cache/go-build/01/01b47c05489549961edcf5a2b1ea1269d8a619fbf1a7927dfbd6de0de037760a-d: open /usr/local/go/pkg/darwin_amd64/runtime/internal/sys.a: no such file or directory
go build sync/atomic: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build unicode/utf8: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build container/list: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build math/bits: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build crypto/internal/subtle: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build crypto/subtle: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build unicode/utf16: copying /home/vagrant/.cache/go-build/96/96dc3112b46bf08056889d489d27f0f959b78fbaf5b7fbba2c6c44cedea1a783-d: open /usr/local/go/pkg/darwin_amd64/unicode/utf16.a: no such file or directory
go build unicode: mkdir /usr/local/go/pkg/darwin_amd64/: permission denied
go build vendor/golang.org/x/crypto/cryptobyte/asn1: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build internal/nettrace: copying /home/vagrant/.cache/go-build/5a/5ab853b24c3a8cf50bb132d36dafd24c41cccb2d65a0b675601d525e919d1fa0-d: open /usr/local/go/pkg/darwin_amd64/internal/nettrace.a: no such file or directory
go build vendor/golang.org/x/crypto/internal/subtle: mkdir /usr/local/go/pkg/darwin_amd64: permission denied
go build encoding: copying /home/vagrant/.cache/go-build/ee/ee07a7afaac4939080714bc90039d1c5126dc16fe24f5e2042aac31d51c8e70a-d: open /usr/local/go/pkg/darwin_amd64/encoding.a: no such file or directory
Makefile:222: recipe for target 'xp-fleet' failed
make: *** [xp-fleet] Error 1
zwass commented 3 years ago

I've just tried this on master from a ubuntu:20.04 Docker container and everything worked as expected. Can you provide step by step instructions for reproducing the issue?

My suspicion is that there is some issue with your Go install.

Here is the shell history from that test (mostly testing to figure out which dependencies were needed and adding those):

root@98573bdadd11:/fleet# history
    1  git
    2  apt-get update
    3  sudo add-apt-repository ppa:longsleep/golang-backports
    4  add-apt-repository ppa:longsleep/golang-backports
    5  wget https://golang.org/dl/go1.15.2.linux-amd64.tar.gz
    6  curl https://golang.org/dl/go1.15.2.linux-amd64.tar.gz > go.tgz
    7  apt-get install wget
    8  add-apt-repository ppa:longsleep/golang-backports
    9  wget https://golang.org/dl/go1.15.2.linux-amd64.tar.gz
   10  tar -C /usr/local -xzf go1.15.2.linux-amd64.tar.gz
   11  export PATH=$PATH:/usr/local/go/bin
   12  go version
   13  git clone https://github.com/kolide/fleet.git
   14  apt-get install git
   15  git clone https://github.com/kolide/fleet.git
   16  cd fleet
   17  make deps && make generate && make
   18  apt-get install build-essential
   19  make deps && make generate && make
   20  apt-get install node
   21  apt install nodejs
   22  make deps && make generate && make
   23  apt-get install npm
   24  make deps && make generate && make
   25  npm install -g yarn
   26  make deps && make generate && make
   27  make deps
   28  ls /usr/local/go/bin
   29  ls
   30  ls ..
   31  ls /usr/local/go/bin
   32  ls ~/go
   33  pwd
   34  ls ~
   35  cd ~
   36  ls
   37  pwd
   38  cd -
   39  export PATH=$PATH:/root/go/bin
   40  make generate && make
   41  history
jalseth commented 3 years ago

This is from a fresh Vagrant Ubuntu 18.04 VM. 

vagrant@ubuntu-bionic:~/go/src/github.com/kolide/fleet$ history
    1  sudo apt update && sudo apt upgrade -y
    2  sudo apt install -y git curl
    3  curl -L -o go.tar.gz https://golang.org/dl/go1.15.2.linux-amd64.tar.gz
    4  sudo tar -C /usr/local -xzf go.tar.gz
    5  export PATH=$PATH:/usr/local/go/bin
    6  mkdir -p go/src/github.com/kolide
    7  cd go/src/github.com/kolide/
    8  git clone https://github.com/kolide/fleet
    9  cd fleet
   10  export PATH=$PATH:$HOME/go/bin
   11  make deps
   12  sudo apt install make -y
   13  make deps
   14  sudo apt install -y npm
   15  sudo npm install -g yarn
   16  make deps
   17  make generate
   18  echo $?
   19  make
   20  history

Which errors with

vagrant@ubuntu-bionic:~/go/src/github.com/kolide/fleet$ make
mkdir -p build/linux
mkdir -p build/darwin
CGO_ENABLED=0 go build -tags full -i -o build/fleet -ldflags " -X github.com/kolide/kit/version.appName=fleet -X github.com/kolide/kit/version.version=3.1.0-9-gd959a4e4 -X github.com/kolide/kit/version.branch=master -X github.com/kolide/kit/version.revision=d959a4e4afbd266d8075799890940b7d5061f704 -X github.com/kolide/kit/version.buildDate=2020-09-23T16:47:20Z -X github.com/kolide/kit/version.buildUser=vagrant -X github.com/kolide/kit/version.goVersion=go1.15.2" ./cmd/fleet
go build net: copying /tmp/go-build662741959/b074/_pkg_.a: open /usr/local/go/pkg/linux_amd64/net.a: permission denied
go build os/user: copying /tmp/go-build662741959/b292/_pkg_.a: open /usr/local/go/pkg/linux_amd64/os/user.a: permission denied
Makefile:110: recipe for target 'fleet' failed
make: *** [fleet] Error 1

I see you're running as root, which wouldn't run into these permissions issues. I am also able to build as root.

After doing some digging, it may be due to the -i flag which installs the packages that are dependencies of the target which means it is attempting to write to $GOROOT which a regular user cannot in this scenario. Related issue: https://github.com/golang/go/issues/18981#issuecomment-278101364

zwass commented 3 years ago

Ah, thank you for that digging.

Good point that I was testing with root permissions, as that's the default in Docker containers. It also works on my macOS machine with an unprivileged user.

Nice find on the -i flag. I'll need to take a look and try to understand why we use that flag -- I'm not sure it's necessary.

jalseth commented 3 years ago

@zwass I did some testing and didn't have any issues building without the flag, so I have opened a PR to remove it.