issue with concurrent update

[ghost]

1. glide install sporadically fails with following trace

cd src/voyager && /src/build/glide install; [WARN] Unable to locate home directory [INFO] Fetching updates for github.com/armon/consul-api. [INFO] Fetching updates for github.com/BurntSushi/toml. [INFO] Fetching updates for github.com/bluele/gcache. [INFO] Fetching updates for github.com/coreos/go-etcd. [INFO] Fetching updates for github.com/codegangsta/cli. [INFO] Fetching updates for github.com/go-sql-driver/mysql. [INFO] Fetching updates for github.com/gorilla/context. [INFO] Fetching updates for github.com/gorilla/handlers. [INFO] Fetching updates for github.com/gorilla/mux. [INFO] Fetching updates for github.com/hashicorp/hcl. [INFO] Fetching updates for github.com/jmoiron/sqlx. [INFO] Fetching updates for github.com/jtolds/gls. [INFO] Fetching updates for github.com/justinas/alice. [INFO] Fetching updates for github.com/kisielk/sqlstruct. [INFO] Fetching updates for github.com/kr/pretty. [INFO] Fetching updates for github.com/kr/pty. [INFO] Fetching updates for github.com/kr/text. [INFO] Fetching updates for github.com/magiconair/properties. [INFO] Fetching updates for github.com/mitchellh/mapstructure. [INFO] Fetching updates for github.com/ooyala/go-dogstatsd. [WARN] Unable to checkout github.com/gorilla/context [WARN] Update failed for github.com/gorilla/context: fatal: could not create leading directories of '/src/src/voyager/vendor/github.com/gorilla/context': File exists : exit status 128 [INFO] Fetching updates for github.com/Sirupsen/logrus. [INFO] Fetching updates for github.com/smartystreets/assertions. [INFO] Fetching updates for github.com/smartystreets/goconvey. [INFO] Fetching updates for github.com/spf13/cast. [INFO] Fetching updates for github.com/spf13/jwalterweatherman. [INFO] Fetching updates for github.com/spf13/pflag. [INFO] Fetching updates for github.com/spf13/viper. [INFO] Fetching updates for github.com/ugorji/go. [INFO] Fetching updates for github.com/xordataexchange/crypt. [INFO] Fetching updates for golang.org/x/crypto. [INFO] Fetching updates for golang.org/x/net. [INFO] Fetching updates for golang.org/x/text. [INFO] Fetching updates for gopkg.in/airbrake/gobrake.v2. [INFO] Fetching updates for gopkg.in/DATA-DOG/go-sqlmock.v1. [INFO] Fetching updates for gopkg.in/fsnotify.v1. [INFO] Fetching updates for gopkg.in/gemnasium/logrus-airbrake-hook.v2. [INFO] Fetching updates for gopkg.in/guregu/null.v3. [INFO] Fetching updates for gopkg.in/yaml.v2. [WARN] Failed to set version on github.com/gorilla/context to 1c83b3eabd45b6d76072b66b746c20815fb2872d: open /src/src/voyager/vendor/github.com/gorilla/context: no such file or directory [INFO] Setting version for github.com/jmoiron/sqlx to 398dd5876282499cdfd4cb8ea0f31a672abe9495. [INFO] Setting version for github.com/codegangsta/cli to 0ab42fd482c27cf2c95e7794ad3bb2082c2ab2d7. [INFO] Setting version for github.com/armon/consul-api to dcfedd50ed5334f96adee43fc88518a4f095e15c. [INFO] Setting version for github.com/kisielk/sqlstruct to 648daed35d49dac24a4bff253b190a80da3ab6a5. [INFO] Setting version for github.com/bluele/gcache to 1e0f672176a0d8f3709552241eb6516c6a503ed3. [INFO] Setting version for github.com/go-sql-driver/mysql to 267b128680c46286b9ca13475c3cca5de8f79bd7. [INFO] Setting version for github.com/BurntSushi/toml to 5c4df71dfe9ac89ef6287afc05e4c1b16ae65a1e. [INFO] Setting version for github.com/magiconair/properties to c81f9d71af8f8cba1466501d30326b99a4e56c19. [INFO] Setting version for github.com/kr/pretty to e6ac2fc51e89a3249e82157fa0bb7a18ef9dd5bb. [INFO] Setting version for github.com/coreos/go-etcd to 003851be7bb0694fe3cc457a49529a19388ee7cf. [INFO] Setting version for github.com/mitchellh/mapstructure to d2dd0262208475919e1a362f675cfc0e7c10e905. [INFO] Setting version for github.com/hashicorp/hcl to 1c284ec98f4b398443cbabb0d9197f7f4cc0077c. [INFO] Setting version for github.com/jtolds/gls to 8ddce2a84170772b95dd5d576c48d517b22cac63. [INFO] Setting version for github.com/justinas/alice to 78131a09b9e7bcccbb8a99d8e2f2168bb576e67a. [INFO] Setting version for github.com/kr/pty to f7ee69f31298ecbe5d2b349c711e2547a617d398. [INFO] Setting version for github.com/ooyala/go-dogstatsd to 23f2a1659b0203cd76910e56cd4e6394f14a5cbb. [INFO] Setting version for github.com/gorilla/mux to 26a6070f849969ba72b72256e9f14cf519751690. [INFO] Setting version for github.com/gorilla/handlers to b3aff83722cb2ae031a70cae984650e3a16cd20e. [INFO] Setting version for github.com/Sirupsen/logrus to 3455d89ac9652295c85db2a98ea32f1d61c380bc. [INFO] Setting version for github.com/kr/text to bb797dc4fb8320488f47bf11de07a733d7233e1f. [INFO] Setting version for github.com/smartystreets/assertions to 287b4346dc4e71a038c346375a9d572453bc469b. [INFO] Setting version for github.com/smartystreets/goconvey to bf58a9a1291224109919756b4dcc469c670cc7e4. [INFO] Setting version for github.com/ugorji/go to f4485b318aadd133842532f841dc205a8e339d74. [INFO] Setting version for github.com/spf13/pflag to 7f60f83a2c81bc3c3c0d5297f61ddfa68da9d3b7. [INFO] Setting version for github.com/spf13/jwalterweatherman to d00654080cddbd2b082acaa74007cb94a2b40866. [INFO] Setting version for github.com/xordataexchange/crypt to 749e360c8f236773f28fc6d3ddfce4a470795227. [INFO] Setting version for github.xxx.net/Platform/go-log to ff724a9ea71a68c36b1eb86365e7161c6ddeb15a. [INFO] Setting version for github.com/spf13/viper to c975dc1b4eacf4ec7fdbf0873638de5d090ba323. [INFO] Setting version for github.com/spf13/cast to ee7b3e0353166ab1f3a605294ac8cd2b77953778. [INFO] Setting version for github.xxx.net/Platform/go-magicnaming to d4ad98287925726811ee547b5744995f090e7497. [INFO] Setting version for github.xxx.net/Platform/go-headers to 517ef5fb50a31b9a124c47a7e3bf4afe813bf4e3. [INFO] Setting version for golang.org/x/text to 07b9a78963006a15c538ec5175243979025fa7a8. [INFO] Setting version for gopkg.in/airbrake/gobrake.v2 to c9d51adc624b5cc4c1bf8de730a09af4878ffe2d. [INFO] Setting version for golang.org/x/crypto to 1f22c0103821b9390939b6776727195525381532. [INFO] Setting version for golang.org/x/net to cbbbe2bc0f2efdd2afb318d93f1eadb19350e4a3. [INFO] Setting version for gopkg.in/DATA-DOG/go-sqlmock.v1 to febff80c09712044459f72847e56aaa69c9dad3b. [INFO] Setting version for gopkg.in/fsnotify.v1 to 8611c35ab31c1c28aa903d33cf8b6e44a399b09e. [INFO] Setting version for gopkg.in/gemnasium/logrus-airbrake-hook.v2 to 31e6fd4bd5a98d8ee7673d24bc54ec73c31810dd. [INFO] Setting version for gopkg.in/guregu/null.v3 to a9db3ac26fcd2d70230a8dc6286d0c3694700003. [INFO] Setting version for gopkg.in/yaml.v2 to f7716cbe52baa25d2e9b0d0da546fcf909fc16b4. ::build vendor/github.com/gorilla/mux/mux.go:14:2: no buildable Go source files in /src/src/voyager/vendor/github.xxx.net/Platform/go-log/vendor/src/github.com/gorilla/context make: * [build] Error 1**

Suspecting something is happening here https://github.com/Masterminds/glide/blob/b0616a2619e65023306b690bdf7062a0def7de60/repo/installer.go#L248 Not sure where "Unable to checkout" message is coming from.

[ghost]

Today I am seeing the same error on a different repo.

[DEBUG] Attempting to find current branch for https://github.com/kr/pretty [DEBUG] Unable to find current branch for https://github.com/kr/pretty, error: chdir /src/src/voyager/vendor/github.com/kr/pretty: no such file or directory [WARN] Unable to checkout github.com/kr/pretty [WARN] Update failed for github.com/kr/pretty: fatal: could not create leading directories of '/src/src/voyager/vendor/github.com/kr/pretty': File exists : exit status 128 .. .. .. ::build vendor/github.com/spf13/viper/viper.go:34:2: cannot find package "github.com/kr/pretty" in any of: /src/src/voyager/vendor/github.com/kr/pretty (vendor tree) /usr/local/go/src/github.com/kr/pretty (from $GOROOT) /src/src/github.com/kr/pretty (from $GOPATH) /src/src/voyager/vendor/src/github.com/kr/pretty /src/src/voyager/vendor/stash.corp.xxx.com/scm/pla/go-log/vendor/src/github.com/kr/pretty make: *\ [build] Error 1

[mattfarina]

@blackgold A fews things:

1. If you use the --no-color (e.g., glide --no-color install) it will remove the characters around WARN, etc. This may make your output easier to read.
2. What version of Glide are you using (glide --version) and how did you get it (binary, homebrew, go get, etc)?
3. Before running glide install what's at the location /src/src/voyager/vendor/github.com/gorilla/context? It is an empty directory? A file? symlink? Something else?

[ghost]

@mattfarina I am using glide version 0.8.3. I am downloading the tar from http://github.com/Masterminds/glide/releases/download/0.8.3/glide-0.8.3-linux-amd64.tar.gz and using the binary /src/src/voyager/vendor directory does not exist before running glide install

[sunzl1989]

Has this issue been looked at? We are also seeing the same issue.

[technosophos]

Yeah, we're still trying to reproduce. If you can reproduce, can you give us some details?

[sunzl1989]

The issue cannot be reproduced reliably. And the libraries failed to be installed are different each time.

As @ghost pointed out, the errors are always like:

fatal: could not create leading directories of XXX: File exists Failed to set version on XXX to YYY: open XXX: no such file or directory

[ascandella]

I'm also running into this issue on my build servers:

The error is coming from VcsUpdate: https://github.com/Masterminds/glide/blob/815d933fc6de40fb433c05b5e717716f0129013d/repo/vcs.go#L48-L51

Here is an excerpt of the build logs, running glide install with an empty vendor directory (fresh clone):

[INFO] Fetching updates for golang.org/x/crypto.
[INFO] Fetching updates for golang.org/x/net.
[INFO] Fetching updates for golang.org/x/sys.
[INFO] Fetching updates for gopkg.in/fsnotify.v1.
[WARN] Unable to checkout golang.org/x/sys
[ERROR] Update failed for golang.org/x/sys: fatal: could not create leading directories of '/var/cache/build/vendor/golang.org/x/sys': File exists
: exit status 128
[INFO] Fetching updates for gopkg.in/validator.v2.
[WARN] Unable to checkout golang.org/x/net
[ERROR] Update failed for golang.org/x/net: fatal: could not create leading directories of '/var/cache/build/vendor/golang.org/x/net': File exists
: exit status 128

Googling around for the "could not create leading directories" error, this seems to be coming from git itself. Which makes sense. Basically, it looks like a race where two simultaneous git clones are run, although I can't reproduce this (perhaps git is doing its own locking?)

[jacobpgallagher]

I upgraded my build servers from git 1.7 to git 2.x and that seemed to fix it.

[ascandella]

That sounds about right -- we're seeing issues on Ubuntu Precise servers, but not on Debian Jessie, which has a more recent git

[ascandella]

Confirmed that it's due to an old git. The following script will panic on git 1.7:

❯❯❯ cat test.go                                                                                                                                                                    ⏎
package main

import (
    "fmt"
    "os/exec"
    "sync"
)

type r struct {
    Source      string
    Destination string
}

func main() {
    var wg sync.WaitGroup
    repos := []r{
        {
            Source:      "https://github.com/golang/crypto",
            Destination: "vendor/golang.org/x/crypto",
        },
        {
            Source:      "https://github.com/golang/net",
            Destination: "vendor/golang.org/x/net",
        },
    }
    for _, repo := range repos {
        wg.Add(1)
        go func(repo r) {
            defer wg.Done()
            cmd := exec.Command("git", "clone", repo.Source, repo.Destination)
            bs, err := cmd.CombinedOutput()
            fmt.Println(string(bs))
            if err != nil {
                panic(err)
            }
        }(repo)
    }
    wg.Wait()
}
❯❯❯ go run test.go
fatal: could not create leading directories of 'vendor/golang.org/x/crypto': File exists

panic: exit status 128

goroutine 5 [running]:
main.main.func1(0xc82000a360, 0x54e280, 0x20, 0x54a140, 0x1a)
    /home/aiden/test.go:34 +0x2b0
created by main.main
    /home/aiden/test.go:36 +0x1af

goroutine 1 [semacquire]:
...
❯❯❯ git --version                                                                                                                                                                  ⏎
git version 1.7.9.5

Works fine on 2.3 (which is what's on our Debian Jessie servers)

[blackgold]

We have git 2.4 on build servers and we are seeing these errors sporadically.

[mattfarina]

Folks, what version of Glide are you using? Running glide --version will show it. The latest release of 0.10.2 should have this fixed. We need to know if this isn't fixed in the latest release.

[ascandella]

Just tested with 0.10.2, and the issue is still present:

[INFO] Downloading dependencies. Please wait...
[INFO] Fetching updates for github.com/BurntSushi/toml.
[INFO] Fetching updates for github.com/getsentry/raven-go.
[INFO] Fetching updates for github.com/codahale/hdrhistogram.
[INFO] Fetching updates for github.com/spf13/cobra.
[INFO] Fetching updates for github.com/kr/pretty.
[INFO] Fetching updates for github.com/spf13/cast.
[INFO] Fetching updates for github.com/pborman/uuid.
[INFO] Fetching updates for github.com/spf13/jwalterweatherman.
[INFO] Fetching updates for github.com/hashicorp/hcl.
[INFO] Fetching updates for github.com/spf13/pflag.
[INFO] Fetching updates for github.com/apache/thrift.
[INFO] Fetching updates for github.com/kr/text.
[INFO] Fetching updates for github.com/magiconair/properties.
[INFO] Fetching updates for github.com/mitchellh/mapstructure.
[INFO] Fetching updates for github.com/mgutz/ansi.
[INFO] Fetching updates for github.com/spf13/viper.
[INFO] Fetching updates for github.com/cactus/go-statsd-client.
[WARN] Unable to checkout github.com/spf13/jwalterweatherman
[ERROR] Update failed for github.com/spf13/jwalterweatherman: fatal: could not create leading directories of '/home/aiden/build/vendor/github.com/spf13/jwalterweatherman': File exists
: exit status 128

$ glide --version
glide version 0.10.2
$ git --version
git version 1.7.9.5

[sdboyer]

seems like this should be relatively easily addressed with some strategically-placed os.MkdirAll()

[avarabyeu]

+1 also can reproduce git version 2.14.1 glide version 0.12.3

Is it possible to make some lock-file or smth like that to at least make install sequential?