search

cloudfoundry-attic / bosh-init

bosh-init is a tool used to create and update the Director VM
Apache License 2.0
31 stars 33 forks source link

bosh-init deploy loses contact with the director during compilation [macOS Sierra] #112

Closed dsharp-pivotal closed 7 years ago

dsharp-pivotal commented 7 years ago

bosh-init deploy hangs at "Compiling postgres/xxxxx" or another package. If you ssh into the director, and examine logs, you can see that it is no longer compiling, and the compilation was successful.

Running with BOSH_INIT_LOG_LEVEL=debug results in a go panic.

Logs from the VM: https://slack-files.com/T02FL4A1X-F2MMLCT98-d5004aee83 Snippet of local debug output: https://slack-files.com/T02FL4A1X-F2MKL0TMH-f3a96589cd

We tried various binaries from 0.81 through 0.97 on macOS, two stemcell versions (3215 and 3263.5), two bosh versions (255.7 and 257.15), and two bosh-aws-cpi releases (48 and 60). The earlier versions represent what was running on our original director.

When we ran the same command under a Linux Docker image using bosh-init-0.97-linux-amd64, it worked fine.

Thanks for the initial debugging @zaksoup and @DennisDenuto !

voelzmo commented 7 years ago

@dsharp-pivotal this happens on macOS sierra, because you're using a binary not built with golang 1.7.1 – which doesn't work on sierra.

Make sure to upgrade to 0.97, which is built with golang 1.7.1. See also https://github.com/cloudfoundry/bosh/pull/1447 which updates the docker image used at that time for building bosh-init. Now they switched to a different image with golang 1.7.1 with commit https://github.com/cloudfoundry/bosh-init/commit/031c606e87967e7e03834c8b24d1f5bc8b39b0fc.

voelzmo commented 7 years ago

sorry for the above comment, seems like bosh-init 0.97 wasn't actually built with golang 1.7.1:

$ docker run -i -t bosh/init
(...)
Digest: sha256:d6d4e7b14daabad647f1e65a4a38ca20d70f893109c383336c0963a490a0e8fb
Status: Downloaded newer image for bosh/init:latest

root@06332b2250c3:/# go version
go version go1.6.1 linux/amd64

root@e1bd20c9e00d:~# cat /usr/local/go/VERSION
go1.6.1

According to the build at https://main.bosh-ci.cf-app.com/teams/main/pipelines/bosh-init/jobs/build/builds/28 – which produces bosh-init 0.97 – the image used is exactly the one I'm looking at: Successfully pulled bosh/init@sha256:d6d4e7b14daabad647f1e65a4a38ca20d70f893109c383336c0963a490a0e8fb.

Anybody knows what happened here? Has the new image with golang 1.7.1 been pushed to dockerhub?

tylerschultz commented 7 years ago

We'll investigate today and ensure the new image is pushed.

voelzmo commented 7 years ago

After a local docker build with https://github.com/cloudfoundry/bosh-init/blob/master/ci/docker/Dockerfile golang seems to be at version 1.7.1. So my best guess is that no push happened to bosh/init on dockerhub.

dpb587-pivotal commented 7 years ago

Oh no! I should have pulled to double check that. Will fix...

dpb587-pivotal commented 7 years ago

Story: https://www.pivotaltracker.com/story/show/132166973

cppforlife commented 7 years ago

latest release (v0.0.98) should be fixed.