Add a vagrant file and provisioning script

[dlecocq]

In the course of helping @vadim-moz debug, I ended up creating a provisioning script and Vagrantfile. Helpful for those on a new machine, wishing to hack on it, etc. for getting up and running with a development environment.

[myronmarston]

This branch seems to cause problems for git/bundler for reasons I don't yet understand. I frequently get these sorts of errors from running bundler commands from projects that are using qless from git:

error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
error: some local refs could not be updated; try running
 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /Users/myron/moz/vanguard/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
If this error persists you could try removing the cache directory '/Users/myron/moz/vanguard/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
error: some local refs could not be updated; try running
 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /Users/myron/moz/vanguard/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
If this error persists you could try removing the cache directory '/Users/myron/moz/vanguard/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
error: some local refs could not be updated; try running
 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /Users/myron/moz/vanguard/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
If this error persists you could try removing the cache directory '/Users/myron/moz/vanguard/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'

I think it may have to do with the fact that there used to be a branch called dan and this branch is dan/vagrant and since git uses the file system to store its database, there's a directory in .git for dan (refs/heads/dan, apparently) and refs/heads/dan/vagrant would be nested in a way that git doesn't like.

@dlecocq -- do you mind deleting this branch and repushing it as dan-vagrant or something?

[dlecocq]

The branch on github has been deleted, so you just have to delete your local one (I imagine). Have you tried giving the command they suggest a shot? Specifically git remote prune ...?

[myronmarston]

Yes, I've given the command a shot. It fixes it temporarily and then it comes back some time later. This has happened multiple times over the course of months. I only got annoyed enough to comment here when it happened yet again yesterday.

[dlecocq]

I'd delete your local repo and re-clone. That's what I did when I encountered this a while back.

[myronmarston]

I'd delete your local repo and re-clone. That's what I did when I encountered this a while back.

Which repo are you talking about -- vanguard (the consuming repo referencing Qless via :git bundler dependency) or the qless clone bundler has made within Vanguard?

If it's the latter...I believe I've done that, a couple of times, and it's still come back. If it's the former...cloning vanguard can take quite some time, so I'm loath to do that but I may try that at some point. Although, this problem seems to go away for a while and then come back for some reason so it's hard to say if it's ever really fixed :(.

[myronmarston]

So we just ran into this error again in our prod servers (after not seeing it for a month or more):

 ** [out :: dalmavancollector04.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector04.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector04.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector04.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector04.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector04.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector04.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector04.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector04.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector04.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector04.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector04.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector04.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector04.dal.moz.com] Git error: command `git fetch --force --quiet --tags
 ** [out :: dalmavancollector04.dal.moz.com] 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory
 ** [out :: dalmavancollector04.dal.moz.com] /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60
 ** [out :: dalmavancollector04.dal.moz.com] has failed.
 ** [out :: dalmavancollector04.dal.moz.com] If this error persists you could try removing the cache directory
 ** [out :: dalmavancollector04.dal.moz.com] '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector02.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector02.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector02.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector02.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector02.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector02.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector02.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector02.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector02.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector02.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavanadmin02.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavanadmin02.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavanadmin02.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavanadmin02.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavanadmin02.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector02.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector02.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector02.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector02.dal.moz.com] Git error: command `git fetch --force --quiet --tags
 ** [out :: dalmavancollector02.dal.moz.com] 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory
 ** [out :: dalmavancollector02.dal.moz.com] /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60
 ** [out :: dalmavancollector02.dal.moz.com] has failed.
 ** [out :: dalmavancollector02.dal.moz.com] If this error persists you could try removing the cache directory
 ** [out :: dalmavancollector02.dal.moz.com] '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavanadmin02.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavanadmin02.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavanadmin02.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavanadmin02.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavanadmin02.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavanadmin02.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavanadmin02.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavanadmin02.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavanadmin02.dal.moz.com] Git error: command `git fetch --force --quiet --tags
 ** [out :: dalmavanadmin02.dal.moz.com] 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory
 ** [out :: dalmavanadmin02.dal.moz.com] /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60
 ** [out :: dalmavanadmin02.dal.moz.com] has failed.
 ** [out :: dalmavanadmin02.dal.moz.com] If this error persists you could try removing the cache directory
 ** [out :: dalmavanadmin02.dal.moz.com] '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector03.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector03.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector03.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector03.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector03.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector03.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector03.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector03.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector03.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector03.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector03.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector03.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector03.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector03.dal.moz.com] Git error: command `git fetch --force --quiet --tags
 ** [out :: dalmavancollector03.dal.moz.com] 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory
 ** [out :: dalmavancollector03.dal.moz.com] /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60
 ** [out :: dalmavancollector03.dal.moz.com] has failed.
 ** [out :: dalmavancollector03.dal.moz.com] If this error persists you could try removing the cache directory
 ** [out :: dalmavancollector03.dal.moz.com] '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector01.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector01.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector01.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector01.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector01.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector01.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector01.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector01.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector01.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavancollector01.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector01.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavancollector01.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavancollector01.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavancollector01.dal.moz.com] Git error: command `git fetch --force --quiet --tags
 ** [out :: dalmavancollector01.dal.moz.com] 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory
 ** [out :: dalmavancollector01.dal.moz.com] /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60
 ** [out :: dalmavancollector01.dal.moz.com] has failed.
 ** [out :: dalmavancollector01.dal.moz.com] If this error persists you could try removing the cache directory
 ** [out :: dalmavancollector01.dal.moz.com] '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavancollector01.dal.moz.com]
 ** [out :: dalmavanadmin01.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavanadmin01.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavanadmin01.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavanadmin01.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (2/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavanadmin01.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavanadmin01.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavanadmin01.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavanadmin01.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavanadmin01.dal.moz.com] Retrying git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*" due to error (3/3): Bundler::Source::Git::GitCommandError Git error: command `git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60 has failed.
 ** [out :: dalmavanadmin01.dal.moz.com] If this error persists you could try removing the cache directory '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavanadmin01.dal.moz.com] error: 'refs/heads/dan' exists; cannot create 'refs/heads/dan/vagrant'
 ** [out :: dalmavanadmin01.dal.moz.com] error: some local refs could not be updated; try running
 ** [out :: dalmavanadmin01.dal.moz.com] 'git remote prune git://github.com/seomoz/qless.git' to remove any old, conflicting branches
 ** [out :: dalmavanadmin01.dal.moz.com] Git error: command `git fetch --force --quiet --tags
 ** [out :: dalmavanadmin01.dal.moz.com]
 ** [out :: dalmavanadmin01.dal.moz.com] 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"` in directory
 ** [out :: dalmavanadmin01.dal.moz.com]
 ** [out :: dalmavanadmin01.dal.moz.com] /var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60
 ** [out :: dalmavanadmin01.dal.moz.com]
 ** [out :: dalmavanadmin01.dal.moz.com] has failed.
 ** [out :: dalmavanadmin01.dal.moz.com]
 ** [out :: dalmavanadmin01.dal.moz.com] If this error persists you could try removing the cache directory
 ** [out :: dalmavanadmin01.dal.moz.com]
 ** [out :: dalmavanadmin01.dal.moz.com] '/var/www/vhosts/vanguard/shared/bundle/ruby/2.1.0/cache/bundler/git/qless-deb3e4deff77b3f2bf7a32dd28b4e1f576731a60'
 ** [out :: dalmavanadmin01.dal.moz.com]

I'm going to work on fixing this in prod...but we've applied fixes in prod for this 3-4 times before. @dlecocq, would you reconsider blowing away this branch and pushing it under a different name to avoid this issue?

[b4hand]

I don't see why you think deleting this branch is going to solve the problem. The supposedly conflicting branch named dan doesn't exist, and yet it still seemingly conflicts. So if it comes back when the dan branch is already deleted, why would you think deleting this branch would solve the problem? dan/vagrant is a perfectly valid branch name. Is it possible your version of git on those local machines is really old and busted? The other alternative that I would see is to simply remove .git/refs/heads/dan from those machines. That should be harmless since the dan branch no longer exists. If the problem comes back after explicitly removing the local reference, then I don't know what is going on.

[vadim-ex]

do we do git remote prune origin on deploy?

/Vadim

On Fri, Apr 10, 2015 at 11:54 AM, Brandon Forehand <notifications@github.com

wrote:

I don't see why you think deleting this branch is going to solve the problem. The supposedly conflicting branch named dan doesn't exist, and yet it still seemingly conflicts. So if it comes back when the dan branch is already deleted, why would you think deleting this branch would solve the problem? dan/vagrant is a perfectly valid branch name. Is it possible your version of git on those local machines is really old and busted? The other alternative that I would see is to simply remove .git/refs/heads/dan from those machines. That should be harmless since the dan branch no longer exists. If the problem comes back after explicitly removing the local reference, then I don't know what is going on.

— Reply to this email directly or view it on GitHub https://github.com/seomoz/qless/pull/206#issuecomment-91650927.

[myronmarston]

I don't see why you think deleting this branch is going to solve the problem.

It certainly might not solve the problem. But it certainly might. The error message always names this branch as causing a problem, and given that, and the fact that we've already wasted many hours trying to solve this problem on our end over the last 6 months, why not spend the couple of minutes it would take to move this PR to a different branch name? We've already tried many things. Admittedly, I don't fully understand why this problem occurs given that we delete the local ref and it later comes back, but yet the problem continues to periodically return.

If the problem comes back after explicitly removing the local reference, then I don't know what is going on.

I definitely don't know what's going on. We've repeatedly removed the local references and yet this problem resurfaces. Usually after a few weeks. So yes, it's very odd.

Does Doesn't seem like much to ask for this branch to be renamed, though. (Or is there a bunch of worked involved in doing that that I'm not thinking of?)

[vadim-ex]

If the problem comes back after explicitly removing the local reference, then I don't know what is going on.

I definitely don't know what's going on. We've repeatedly removed the local references and yet this problem resurfaces. Usually after a few weeks. So yes, it's very odd.

Does seem like much to ask for this branch to be renamed, though. (Or is there a bunch of worked involved in doing that that I'm not thinking of?)

If branch resurrects, the renaming of dead branch would not help, would it?

[b4hand]

That should not be necessary. git remote prune origin removes remote branch references, but the error message indicates this is a local reference. Notice that remote branch references are stored in .git/refs/remotes/origin/*, so if this were a remote issue, the error would be on .git/refs/remotes/origin/dan.

The issue appears to be that once upon a time there was a branch called dan. That branch got deleted long ago, but then dan created a new branch called dan/vagrant. Some machines appear to still have a local branach named dan which conflicts with this newer branch. One way to clean that reference up is to use git branch -D dan on each of the machines. That should delete any stale branch references. Alternatively, you can just do the rm manually.

[myronmarston]

I believe we've done the rm manually many many times.

@GreyKn -- can you speak to what you've done on the prod servers to deal with this when it periodically occurs?

I can understand why you think that my suggestion may not work. Is anyone opposed to me trying it, though? Maybe I will waste a few minutes, but that's my perogative to waste my own time. I don't understand the resistance to trying a different technique than what we've already tried.

[b4hand]

I don't mind you wasting your time Myron. ;) I just want to help you find the root cause.

Are you doing anything weird like pushing branches directly to these machines? It sounds like someone is explicitly reviving the dan branch because that shouldn't happen automatically. It might be the case that someone isn't running git push in single mode and thus they are pushing all local branches including this dan branch.

[myronmarston]

Are you doing anything weird like pushing branches directly to these machines?

Nope.

It might be the case that someone isn't running git push in single mode and thus they are pushing all local branches including this dan branch.

I don't believe that can be the problem. On deploy we don't run any git commands to fetch qless. It's bundler that does that. It does it based on the git reference in our Gemfile.lock.

I supposed bundler could be doing something weird...

[GreyKn]

Our cap deploys will fail with a fatal error when this occurs. To fix it, I need to ssh into the affected machines and rm the bundler shared cached copy of the git repo to allow deploys to succeed.

Location: /var/www/vhosts/<project>/shared/bundle/ruby/<version>/cache/bundler/git/qless-<somesha>

[myronmarston]

FWIW, here are the git commands bundler is using:

https://github.com/bundler/bundler/blob/v1.6.1/lib/bundler/source/git/git_proxy.rb#L63-L75

It does fetch all the refs, but given that the ref has been deleted from github, that shouldn't cause this to resurface.

...and yet it does. Makes me wonder if there's some latent GitHub bug where a deleted branch can later temporarily be included in the list of refs again.

[vadim-ex]

Do we do this issue right now? I'd like to telnet to Mordor and explore that a bit.

[b4hand]

By the way, I wasn't saying that your deploy code was issuing the git push, it's quite likely that this is some developer that is issuing it during their normal development who still has a local dan branch on their machine. Regardless, this does seem mysterious. I've also confirmed that the dan branch is dead according to GitHub and there is also no tag named dan either. Also, my local repos for qless contain no such file.

I also ran these exact commands on my local machine:

mkdir /tmp/blah/
cd /tmp/blah
git init --bare
git fetch --force --quiet --tags 'git://github.com/seomoz/qless.git' "refs/heads/*:refs/heads/*"

And the dan file doesn't reappear. Even after issuing the fetch multiple times.

If there's no other way to resolve this, I guess Dan can try renaming the branch. I don't see how that will help, but I guess it can't hurt.

[myronmarston]

Do we do this issue right now? I'd like to telnet to Mordor and explore that a bit.

No, @GreyKn rm'd the directory so we could deploy. Next time it happens, we'll let you poke around before cleaning up if you're around at the time.

[vadim-ex]

it would take long: I will just copy .git and bundle :-) Than I will enjoy my time :-)

/Vadim

On Fri, Apr 10, 2015 at 1:09 PM, Myron Marston notifications@github.com wrote:

Do we do this issue right now? I'd like to telnet to Mordor and explore that a bit.

No, @GreyKn https://github.com/GreyKn rm'd the directory so we could deploy. Next time it happens, we'll let you poke around before cleaning up if you're around at the time.

— Reply to this email directly or view it on GitHub https://github.com/seomoz/qless/pull/206#issuecomment-91669326.

[vadim-ex]

*would not​

[dlecocq]

I've updated this branch because I needed an environment for #246 -- it took some fiddling with dependencies to make it happen.

[dlecocq]

I destroyed my vagrant and reprovisioned it and successfully ran tests. Big caveat, the headless tests don't all pass (similar to the flakiness in Travis), and so I needed to run with TRAVIS=true rake spec. I hope to address this in a subsequent PR, but this provides some benefit now.