Mysterious crashes when starting up

[offby1]

I have no idea what this means:

2015/01/27 17:13:05 Searcher started for inventory
panic: exit status 128

goroutine 1 [running]:
main.main()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/cmds/houndd/main.go:304 +0x5a6

goroutine 6 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b
exit status 2

I found that deleting the relevant entries from my config file allowed it to continue. (There were maybe half a dozen URLs that causes crashes like this).

[kellegous]

@offby1 Man, we really need to fix some of these error messages. It looks like Hound is having trouble cloning the inventory repository. Does the data/inventory directory exist? If it does, can you do the following: cd data/inventory && git pull

That should give you some kind of error.

[winkler1]

Similar error here. Can't nail it down to one repo, there are so many it's hitting. I keep restarting, and it makes progress but croaks quickly on different repos. Have 200+ repos in my config. Worked fine with just a couple repos. Memory? Github anti-DOS throttling?

$ go run src/hound/cmds/houndd/main.go
2015/01/28 10:28:46 Searcher started for react-disqus-thread
2015/01/28 10:28:47 Searcher started for wow-realm-status-react
2015/01/28 10:28:47 Searcher started for guessign
2015/01/28 10:28:47 Searcher started for literallycanvas
2015/01/28 10:28:49 merge 0 files + mem
2015/01/28 10:28:49 957552 data bytes, 420335 index bytes
2015/01/28 10:28:49 Searcher started for isomorphic-lab
2015/01/28 10:28:49 Searcher started for react-sortablejs
panic: exit status 128

goroutine 1 [running]:
main.main()
    /Users/jeff.winkler/repos/Hound/src/hound/cmds/houndd/main.go:304 +0x5a6

goroutine 5 [sleep]:
hound/searcher.func·001()
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 6 [sleep]:
hound/searcher.func·001()
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 7 [sleep]:
hound/searcher.func·001()
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 9 [sleep]:
hound/searcher.func·001()
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 10 [sleep]:
hound/searcher.func·001()
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/jeff.winkler/repos/Hound/src/hound/searcher/searcher.go:183 +0x54b
exit status 2

[adstephenson]

I had this error too. In my case it was caused by empty repositories. When cloning them via command line I got a warning: warning: You appear to have cloned an empty repository. and when running git pull: Your configuration specifies to merge with the ref 'master' from the remote, but no such ref was fetched.

I have just removed the repos from the config file for now.

[offby1]

@kellegous it's sho' 'nuff empty; behold:

:) 09:10:23 [erichanchrow@Eric-Hanchrows-iMac Hound] (master)$ cd data/inventory/
:) 09:10:27 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ ll
total 0
drwxr-xr-x 9 erichanchrow staff 306 2015-01-28T01:13:06Z .git/
:) 09:10:29 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ git show
fatal: bad default revision 'HEAD'
:( 128 09:10:33 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ git remote -v
origin  git@github.com:Internetworkexpert/inventory.git (fetch)
origin  git@github.com:Internetworkexpert/inventory.git (push)
:( 130 09:10:47 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ git log
fatal: bad default revision 'HEAD'
:( 128 09:10:48 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ git branch -avv
:) 09:10:56 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ find . -ls
54982061    0 drwxr-xr-x   3 erichanchrow staff         102 Jan 27 17:13 .
54982062    0 drwxr-xr-x   9 erichanchrow staff         306 Jan 27 17:13 ./.git
54982095    4 -rw-r--r--   1 erichanchrow staff         317 Jan 27 17:13 ./.git/config
54982066    4 -rw-r--r--   1 erichanchrow staff          73 Jan 27 17:13 ./.git/description
54982079    4 -rw-r--r--   1 erichanchrow staff          23 Jan 27 17:13 ./.git/HEAD
54982067    0 drwxr-xr-x  11 erichanchrow staff         374 Jan 27 17:13 ./.git/hooks
54982068    4 -rwxr-xr-x   1 erichanchrow staff         452 Jan 27 17:13 ./.git/hooks/applypatch-msg.sample
54982069    4 -rwxr-xr-x   1 erichanchrow staff         896 Jan 27 17:13 ./.git/hooks/commit-msg.sample
54982070    4 -rwxr-xr-x   1 erichanchrow staff         189 Jan 27 17:13 ./.git/hooks/post-update.sample
54982071    4 -rwxr-xr-x   1 erichanchrow staff         398 Jan 27 17:13 ./.git/hooks/pre-applypatch.sample
54982072    4 -rwxr-xr-x   1 erichanchrow staff        1642 Jan 27 17:13 ./.git/hooks/pre-commit.sample
54982073    4 -rwxr-xr-x   1 erichanchrow staff        1356 Jan 27 17:13 ./.git/hooks/pre-push.sample
54982074    8 -rwxr-xr-x   1 erichanchrow staff        4951 Jan 27 17:13 ./.git/hooks/pre-rebase.sample
54982075    4 -rwxr-xr-x   1 erichanchrow staff        1239 Jan 27 17:13 ./.git/hooks/prepare-commit-msg.sample
54982076    4 -rwxr-xr-x   1 erichanchrow staff        3611 Jan 27 17:13 ./.git/hooks/update.sample
54982077    0 drwxr-xr-x   3 erichanchrow staff         102 Jan 27 17:13 ./.git/info
54982078    4 -rw-r--r--   1 erichanchrow staff         240 Jan 27 17:13 ./.git/info/exclude
54982089    0 drwxr-xr-x   4 erichanchrow staff         136 Jan 27 17:13 ./.git/objects
54982091    0 drwxr-xr-x   2 erichanchrow staff          68 Jan 27 17:13 ./.git/objects/info
54982090    0 drwxr-xr-x   2 erichanchrow staff          68 Jan 27 17:13 ./.git/objects/pack
54982063    0 drwxr-xr-x   4 erichanchrow staff         136 Jan 27 17:13 ./.git/refs
54982064    0 drwxr-xr-x   2 erichanchrow staff          68 Jan 27 17:13 ./.git/refs/heads
54982065    0 drwxr-xr-x   2 erichanchrow staff          68 Jan 27 17:13 ./.git/refs/tags
:) 09:11:02 [erichanchrow@Eric-Hanchrows-iMac inventory] (master #)$ 

[jklein]

Cool, @offby1 my PR #36 should fix this. Hopefully we'll merge today.

[offby1]

I just noticed a similar-but-perhaps-different problem with a different repo (tl;dr: I think I force-pushed, and that made Hound's head asplode):

:) 09:14:13 [erichanchrow@Eric-Hanchrows-iMac Hound] (master)$ ~/git-repositories/3rd-party/go/bin/go run src/hound/cmds/houndd/main.go
2015/01/30 09:14:15 Searcher started for gateone-virtual-machine-image
2015/01/30 09:14:16 Searcher started for ine-admin
2015/01/30 09:14:18 Searcher started for work-notes
2015/01/30 09:14:21 Searcher started for ansible-roles
2015/01/30 09:14:22 Searcher started for members-site
2015/01/30 09:14:23 Searcher started for guacamole-virtual-machine-image
2015/01/30 09:14:24 Searcher started for ine-async
2015/01/30 09:14:25 Searcher started for ine-rackcontrol
2015/01/30 09:14:26 Searcher started for ine-rest
2015/01/30 09:14:27 Searcher started for experimental-scheduler
panic: exit status 1

goroutine 1 [running]:
main.main()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/cmds/houndd/main.go:304 +0x5a6

goroutine 5 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 6 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 7 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 8 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 9 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 10 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 11 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 12 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b

goroutine 13 [sleep]:
hound/searcher.func·001()
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:145 +0x6f
created by hound/searcher.New
    /Users/erichanchrow/git-repositories/3rd-party/Hound/src/hound/searcher/searcher.go:183 +0x54b
exit status 2
:( 1 09:14:27 [erichanchrow@Eric-Hanchrows-iMac Hound] (master)$ cd data/experimental-scheduler
:) 09:15:02 [erichanchrow@Eric-Hanchrows-iMac experimental-scheduler] (master *+|MERGING)$ ls
.git/  .gitignore  README.md  assign_racks.py*  queries.py  requirements.txt  session.py  stubs.py
:) 09:15:03 [erichanchrow@Eric-Hanchrows-iMac experimental-scheduler] (master *+|MERGING)$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 2 and 3 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)
You have unmerged paths.
  (fix conflicts and run "git commit")

Changes to be committed:

    modified:   assign_racks.py

Unmerged paths:
  (use "git add <file>..." to mark resolution)

    both modified:   stubs.py

:) 09:17:46 [erichanchrow@Eric-Hanchrows-iMac experimental-scheduler] (master *+|MERGING)$ git reset --hard @{u}
HEAD is now at c84f84d Only log a few returned rows.
:) 09:17:58 [erichanchrow@Eric-Hanchrows-iMac experimental-scheduler] (master)$ cd ../..
:) 09:18:01 [erichanchrow@Eric-Hanchrows-iMac Hound] (master)$ ~/git-repositories/3rd-party/go/bin/go run src/hound/cmds/houndd/main.go
2015/01/30 09:18:03 Searcher started for ine-rest
2015/01/30 09:18:05 Searcher started for work-notes
2015/01/30 09:18:10 Searcher started for members-site
2015/01/30 09:18:12 Searcher started for gateone-virtual-machine-image
2015/01/30 09:18:13 Searcher started for guacamole-virtual-machine-image
2015/01/30 09:18:14 Searcher started for ine-rackcontrol
2015/01/30 09:18:15 Searcher started for experimental-scheduler
2015/01/30 09:18:17 merge 0 files + mem
2015/01/30 09:18:17 19893 data bytes, 65527 index bytes
2015/01/30 09:18:17 Searcher started for ExpectUtils
2015/01/30 09:18:18 Searcher started for ansible-roles
2015/01/30 09:18:19 Searcher started for ine-admin
2015/01/30 09:18:20 Searcher started for ine-async
2015/01/30 09:18:21 All indexes built, running server at http://localhost:6080...

[jklein]

Yup, if Hound can't git pull or git clone cleanly it's going to puke. Once my PR from #36 is merged Hound will continue with the rest of the indexes and just skip the repo that died, which should be an improvement.

[offby1]

I've got to ask -- why are you git pulling in the first place? Wouldn't it make more sense to get fetch && git reset --hard @{upstream}?

[jklein]

Well in the typical case there are never going to be changes in these repos, so git pull works well enough. If there are changes I feel like it might be better to communicate that and skip the repo instead of just blowing them away.

[offby1]

I guess the communication would want to get the following across, without perhaps being so verbose:

Yikes! I couldn't "pull" into this repo. Either you've done some work in there (which you shouldn't do; this repo is essentially just a cache for Hound), or else somebody force-pushed to the upstream. You can fix this repo by first saving any work you've done in it (perhaps by pushing it upstream), and then doing "git reset --hard @{upstream}".

[jklein]

The fix for empty repos has been merged, and future errors should be much less mysterious since we are passing through the git messages. I'm going to close this issue for now, and feel free to open new ones if you keep having issues.

[offby1]

Great work -- the error message is very clear now. Thanks!