search

ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.54k stars 20.13k forks source link

Geth stops syncing archive node #17746

Closed rmulhol closed 4 years ago

rmulhol commented 6 years ago

Hi there,

please note that this is an issue tracker reserved for bug reports and feature requests.

For general questions please use the gitter channel or the Ethereum stack exchange at https://ethereum.stackexchange.com.

System information

Geth version: geth version 1/8/13-stable OS & Version: Windows/Linux/OSX OSX Commit hash : (if develop) 225171a4bfcc16bd12a1906b1e0d43d0b18c353b

Expected behaviour

Geth would continue to sync to the head of the chain and continue running.

Actual behaviour

Geth stops syncing new blocks. Needs a force quit and restart.

Steps to reproduce the behaviour

Not sure how to reproduce, not sure why this would happen in the first place. The stack trace below shows the node syncing regularly until block 6368257, doing ~3 blocks ever ~19 seconds. Then, there is no further activity for ~18 hours, at which point I attempt a force quit. This continues for ~3 days, until I quit via LoudPanic (repeated force quit).

Backtrace

INFO [09-20|15:06:01.355] Imported new chain segment               blocks=2 txs=123 mgas=15.960 elapsed=8.151s   mgasps=1.958 number=6368238 hash=710980…5f42a2 cache=0.00B
INFO [09-20|15:06:10.815] Imported new chain segment               blocks=2 txs=179 mgas=15.019 elapsed=9.460s   mgasps=1.588 number=6368240 hash=de6f93…afae9f cache=0.00B
INFO [09-20|15:06:20.576] Imported new chain segment               blocks=3 txs=196 mgas=23.921 elapsed=9.760s   mgasps=2.451 number=6368243 hash=662eea…093dea cache=0.00B
INFO [09-20|15:06:30.209] Imported new chain segment               blocks=2 txs=209 mgas=15.959 elapsed=9.633s   mgasps=1.657 number=6368245 hash=1cc9c3…e4bd20 cache=0.00B
INFO [09-20|15:06:43.731] Imported new chain segment               blocks=3 txs=312 mgas=23.953 elapsed=13.521s  mgasps=1.771 number=6368248 hash=a68659…936d0f cache=0.00B
INFO [09-20|15:06:54.319] Imported new chain segment               blocks=4 txs=194 mgas=23.950 elapsed=10.587s  mgasps=2.262 number=6368252 hash=aba940…49387d cache=0.00B
INFO [09-20|15:07:04.438] Imported new chain segment               blocks=2 txs=384 mgas=15.741 elapsed=10.119s  mgasps=1.556 number=6368254 hash=ba0ec8…041667 cache=0.00B
INFO [09-20|15:07:12.723] Imported new chain segment               blocks=3 txs=105 mgas=23.859 elapsed=8.285s   mgasps=2.880 number=6368257 hash=a56127…b40240 cache=0.00B
^CINFO [09-21|08:52:50.396] Got interrupt, shutting down...
^CWARN [09-21|13:47:32.428] Already shutting down, interrupt more to panic. times=9
^CWARN [09-21|13:50:40.700] Already shutting down, interrupt more to panic. times=8
^CWARN [09-24|09:14:47.577] Already shutting down, interrupt more to panic. times=7
^CWARN [09-24|09:14:48.033] Already shutting down, interrupt more to panic. times=6
^CWARN [09-24|09:14:48.278] Already shutting down, interrupt more to panic. times=5
^CWARN [09-24|09:14:48.477] Already shutting down, interrupt more to panic. times=4
^CWARN [09-24|09:14:48.825] Already shutting down, interrupt more to panic. times=3
^CWARN [09-24|09:14:49.038] Already shutting down, interrupt more to panic. times=2
^CWARN [09-24|09:14:49.233] Already shutting down, interrupt more to panic. times=1
^Cpanic: boom

goroutine 124 [running]:
github.com/ethereum/go-ethereum/internal/debug.LoudPanic(0x49cdc60, 0x4ce5600)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/internal/debug/loudpanic.go:26 +0x4e
github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1(0xc42010c280)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:84 +0x233
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:70 +0xad

goroutine 1 [chan receive, 5751 minutes]:
github.com/ethereum/go-ethereum/node.(*Node).Wait(0xc42010c280)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:464 +0x7f
main.geth(0xc42021a2c0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:259 +0xf1
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0x49e7a20, 0x4bfb178, 0xc42021a2c0, 0xc42021a2c0, 0xc4204f5f48)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:490 +0xc8
github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc420032340, 0xc4200ba000, 0x6, 0x6, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/gopkg.in/urfave/cli.v1/app.go:264 +0x5ee
main.main()
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:244 +0x55

goroutine 20 [chan receive, 5752 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*recursiveTree).dispatch(0xc4200b82d0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_recursive.go:125 +0xc7
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newRecursiveTree
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_recursive.go:119 +0x149

goroutine 21 [syscall, 5752 minutes, locked to thread]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify._Cfunc_CFRunLoopRun()
    _cgo_gotypes.go:214 +0x41
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.init.1.func1()
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:72 +0x62
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.init.1
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:65 +0x4e

goroutine 22 [syscall]:
os/signal.signal_recv(0x4cf5760)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sigqueue.go:139 +0xa7
os/signal.loop()
    /usr/local/Cellar/go/1.10.2/libexec/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
    /usr/local/Cellar/go/1.10.2/libexec/src/os/signal/signal_unix.go:28 +0x41

goroutine 23 [chan receive, 5416 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc4200a76a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 24 [chan receive, 5416 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc4200a76a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 25 [chan receive, 5416 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc4200a76a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 26 [chan receive, 5416 minutes]:
github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0xc4200a76a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:63 +0x97
created by github.com/ethereum/go-ethereum/core.newTxSenderCacher
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_cacher.go:55 +0xa4

goroutine 31 [select, locked to thread]:
runtime.gopark(0x4bfbab8, 0x0, 0x4b6c512, 0x6, 0x18, 0x1)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420179750, 0xc4201fc060)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/signal_unix.go:549 +0x1c6
runtime.goexit()
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/asm_amd64.s:2361 +0x1

goroutine 33 [select, 5752 minutes]:
github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).loop(0xc42000cae0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:94 +0x4a0
created by github.com/ethereum/go-ethereum/accounts/keystore.(*watcher).start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/watch.go:52 +0x5d

goroutine 50 [select]:
github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).updater(0xc4200920f0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:203 +0xf0
created by github.com/ethereum/go-ethereum/accounts/keystore.(*KeyStore).Subscribe
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/keystore/keystore.go:190 +0x122

goroutine 51 [sleep]:
time.Sleep(0x3b9aca00)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/time.go:102 +0x166
github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc4203aa500)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:224 +0x3e
created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:213 +0x122

goroutine 52 [sleep]:
time.Sleep(0x3b9aca00)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/time.go:102 +0x166
github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).updater(0xc4203aa640)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:224 +0x3e
created by github.com/ethereum/go-ethereum/accounts/usbwallet.(*Hub).Subscribe
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/usbwallet/hub.go:213 +0x122

goroutine 53 [select, 5752 minutes]:
github.com/ethereum/go-ethereum/accounts.(*Manager).update(0xc4201f4ea0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:95 +0x1c6
created by github.com/ethereum/go-ethereum/accounts.NewManager
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/accounts/manager.go:68 +0x54c

goroutine 54 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4203047e0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x152
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x171

goroutine 55 [select, 5407 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc42029ab60)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0xff
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x49d

goroutine 56 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc42029ab60)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x127
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x4bf

goroutine 57 [semacquire, 5407 minutes]:
sync.runtime_SemacquireMutex(0xc420258a98, 0xc4772d7300)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420258a94)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).Log(0xc420258a80, 0xc458be8320, 0x92)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:225 +0x77
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).logf(0xc4202befd0, 0x4b894b3, 0x21, 0xc470853270, 0x4, 0x4)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session_util.go:33 +0x81
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).computeCompaction(0xc48760c320)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:395 +0x62f
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*versionStaging).finish(0xc470853510, 0xc481d33f40)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:510 +0x893
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).spawn(0xc49a4e7db0, 0xc481d33f40, 0xc49a4e7db0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:279 +0x63
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*session).commit(0xc4202befd0, 0xc481d33f40, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/session.go:195 +0x7e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionCommit.func1(0xc459a0ba80, 0xc433202201, 0xc459a0ba80)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:263 +0x37
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*compactionTransactFunc).run(0xc4267c6400, 0xc459a0ba80, 0x4cf3460, 0xc4267c6400)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:241 +0x33
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc42029ab60, 0xc45461fa80, 0x11, 0x4cf3460, 0xc4267c6400)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:185 +0x162
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransactFunc(0xc42029ab60, 0xc45461fa80, 0x11, 0xc489862bc0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:252 +0x88
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionCommit(0xc42029ab60, 0x4b70e75, 0xa, 0xc481d33f40)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:262 +0x11e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tableCompaction(0xc42029ab60, 0xc444127200, 0xc49a4e7d00)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:551 +0xc7a
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tableAutoCompaction(0xc42029ab60)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:633 +0x56
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc42029ab60)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:858 +0x460
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x627

goroutine 58 [select, 5431 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc42029ab60)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762 +0x15d
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x649

goroutine 59 [select]:
github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).meter(0xc4202bf1e0, 0xb2d05e00)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:338 +0x1783
created by github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Meter
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:175 +0x1b8

goroutine 63 [semacquire, 5407 minutes]:
sync.runtime_Semacquire(0xc420255ef4)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*RWMutex).RLock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:50 +0x49
github.com/ethereum/go-ethereum/core.(*TxPool).loop(0xc420255dc0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:300 +0xa02
created by github.com/ethereum/go-ethereum/core.NewTxPool
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:253 +0x4d1

goroutine 64 [select]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).qosTuner(0xc42029b380)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1584 +0x33e
created by github.com/ethereum/go-ethereum/eth/downloader.New
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:231 +0x45c

goroutine 65 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).stateFetcher(0xc42029b380)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/statesync.go:76 +0x146
created by github.com/ethereum/go-ethereum/eth/downloader.New
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:232 +0x47e

goroutine 83 [chan receive, 5751 minutes]:
github.com/ethereum/go-ethereum/miner.(*worker).wait(0xc42021ab00)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:300 +0x362
created by github.com/ethereum/go-ethereum/miner.newWorker
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:160 +0x4d0

goroutine 85 [select, 7 minutes]:
github.com/ethereum/go-ethereum/p2p/nat.Map(0x4cfb0a0, 0xc4202ba040, 0xc42b5f3c80, 0x4b69697, 0x3, 0x765f, 0x765f, 0x4b78152, 0x12)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:114 +0x4b9
created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:436 +0xeda

goroutine 89 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4203055e0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x152
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x171

goroutine 90 [select, 19 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc42029b520)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0xff
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x49d

goroutine 91 [select]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc42029b520)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x127
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x4bf

goroutine 92 [select, 19 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc42029b520)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:834 +0x369
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x627

goroutine 93 [select, 19 minutes]:
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc42029b520)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762 +0x15d
created by github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.openDB
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x649

goroutine 94 [select, 51 minutes]:
github.com/ethereum/go-ethereum/p2p/discover.(*nodeDB).expirer(0xc4200bb860)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/database.go:233 +0x114
created by github.com/ethereum/go-ethereum/p2p/discover.(*nodeDB).ensureExpirer.func1
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/database.go:224 +0x3e

goroutine 95 [select]:
github.com/ethereum/go-ethereum/p2p/discover.(*Table).loop(0xc4200da8c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:370 +0x3d2
created by github.com/ethereum/go-ethereum/p2p/discover.newTable
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/table.go:132 +0x450

goroutine 96 [select]:
github.com/ethereum/go-ethereum/p2p/discover.(*udp).loop(0xc420259c00)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:401 +0x543
created by github.com/ethereum/go-ethereum/p2p/discover.newUDP
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:260 +0x306

goroutine 97 [IO wait]:
internal/poll.runtime_pollWait(0x9cd81b8, 0x72, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420215718, 0x72, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc420215718, 0xc42226c000, 0x500, 0x500)
    /usr/local/Cellar/go/1.10.2/libexec/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).ReadFrom(0xc420215700, 0xc42226c000, 0x500, 0x500, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/internal/poll/fd_unix.go:207 +0x17d
net.(*netFD).readFrom(0xc420215700, 0xc42226c000, 0x500, 0x500, 0xfdff60727c7ddd6, 0x1ef487b71f96aaa9, 0x96e2cbd354ba87e5, 0x4b3a4e8df6c53412, 0xd66703cb3fbbb7ff)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/fd_unix.go:208 +0x5b
net.(*UDPConn).readFrom(0xc4200b74a0, 0xc42226c000, 0x500, 0x500, 0xc422261ed8, 0x43ffa51, 0x4a1c340, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFromUDP(0xc4200b74a0, 0xc42226c000, 0x500, 0x500, 0x500, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/udpsock.go:106 +0x6f
github.com/ethereum/go-ethereum/p2p/discover.(*udp).readLoop(0xc420259c00, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:536 +0xec
created by github.com/ethereum/go-ethereum/p2p/discover.newUDP
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/discover/udp.go:261 +0x338

goroutine 98 [IO wait, 5751 minutes]:
internal/poll.runtime_pollWait(0x866f5e8, 0x72, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420c6e998, 0x72, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc420c6e998, 0xffffffffffffff00, 0x0, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420c6e980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/internal/poll/fd_unix.go:372 +0x1a8
net.(*netFD).accept(0xc420c6e980, 0xc42035a7c0, 0xc4440b4358, 0xc4440b4358)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc420bfe9d0, 0x4006f6b, 0xc4440b4300, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc420bfe9d0, 0x0, 0xc4440b4300, 0xc42035a7a0, 0x2)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/tcpsock.go:259 +0x49
github.com/ethereum/go-ethereum/p2p.(*Server).listenLoop(0xc42010ca00)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:763 +0x23c
created by github.com/ethereum/go-ethereum/p2p.(*Server).startListening
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:520 +0x159

goroutine 99 [select, 7 minutes]:
github.com/ethereum/go-ethereum/p2p/nat.Map(0x4cfb0a0, 0xc4202ba040, 0xc42b5f3c80, 0x4b6968b, 0x3, 0x765f, 0x765f, 0x4b72731, 0xc)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/nat/nat.go:114 +0x4b9
github.com/ethereum/go-ethereum/p2p.(*Server).startListening.func1(0xc42010ca00, 0xc420c81320)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:525 +0x8b
created by github.com/ethereum/go-ethereum/p2p.(*Server).startListening
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:524 +0x1ee

goroutine 100 [select, 5397 minutes]:
github.com/ethereum/go-ethereum/p2p.(*Server).run(0xc42010ca00, 0x4cfb020, 0xc420cd0000)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:590 +0x59f
created by github.com/ethereum/go-ethereum/p2p.(*Server).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:505 +0x78c

goroutine 101 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 102 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 103 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 104 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 105 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 106 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 107 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 108 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 109 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 110 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 111 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 112 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 113 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 114 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 115 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 116 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1(0xc4203048c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:56 +0x37e
created by github.com/ethereum/go-ethereum/eth.(*Ethereum).startBloomHandlers
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/bloombits.go:54 +0x48

goroutine 117 [select, 5407 minutes]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).txBroadcastLoop(0xc420226300)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:756 +0x134
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:209 +0xcb

goroutine 118 [chan receive, 5751 minutes]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).minedBroadcastLoop(0xc420226300)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:746 +0x59
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:213 +0x155

goroutine 119 [select]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer(0xc420226300)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:145 +0x1ca
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:216 +0x177

goroutine 120 [select, 5409 minutes]:
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).txsyncLoop(0xc420226300)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:109 +0x25c
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:217 +0x199

goroutine 121 [select, 5416 minutes]:
github.com/ethereum/go-ethereum/eth/downloader.(*PublicDownloaderAPI).eventLoop(0xc442863f80)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/api.go:63 +0x2f7
created by github.com/ethereum/go-ethereum/eth/downloader.NewPublicDownloaderAPI
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/api.go:49 +0xd7

goroutine 123 [chan receive, 3 minutes]:
github.com/ethereum/go-ethereum/eth/filters.(*PublicFilterAPI).timeoutLoop(0xc4428805f0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/api.go:83 +0x124
created by github.com/ethereum/go-ethereum/eth/filters.NewPublicFilterAPI
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/filters/api.go:73 +0x15f

goroutine 125 [chan receive, 5751 minutes]:
main.startNode.func1(0xc42010c280, 0xc420d01da0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:301 +0x30e
created by main.startNode
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/geth/main.go:286 +0x2c9

goroutine 126 [select, 5751 minutes]:
net.(*pipe).read(0xc422226200, 0xc4222fc000, 0x200, 0x200, 0xc4222dda70, 0x40449c7, 0x200)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/pipe.go:164 +0x1d7
net.(*pipe).Read(0xc422226200, 0xc4222fc000, 0x200, 0x200, 0x0, 0x200, 0xce496ddab19cfbe8)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/pipe.go:147 +0x4d
encoding/json.(*Decoder).refill(0xc4443c03c0, 0x61, 0xc4222ddba8)
    /usr/local/Cellar/go/1.10.2/libexec/src/encoding/json/stream.go:159 +0x132
encoding/json.(*Decoder).readValue(0xc4443c03c0, 0x0, 0x0, 0x88a2bb8e48ae39e7)
    /usr/local/Cellar/go/1.10.2/libexec/src/encoding/json/stream.go:134 +0x23d
encoding/json.(*Decoder).Decode(0xc4443c03c0, 0x4a2c320, 0xc42035ac20, 0x5b07cb3395841d01, 0xc42035ac20)
    /usr/local/Cellar/go/1.10.2/libexec/src/encoding/json/stream.go:63 +0x78
encoding/json.(*Decoder).Decode-fm(0x4a2c320, 0xc42035ac20, 0xc4428808d8, 0x38)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/json.go:119 +0x3e
github.com/ethereum/go-ethereum/rpc.(*jsonCodec).ReadRequestHeaders(0xc4428808c0, 0x0, 0x0, 0x0, 0xc4222fa000, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/json.go:144 +0xbe
github.com/ethereum/go-ethereum/rpc.(*Server).readRequest(0xc420c81440, 0x4cffc80, 0xc4428808c0, 0x55a2701, 0x4a7ed80, 0xc444116800, 0x4cfa920, 0xc42f85a7b0, 0x8ab33b78fb4c1e42)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:375 +0x5f
github.com/ethereum/go-ethereum/rpc.(*Server).serveRequest(0xc420c81440, 0x4cfa8a0, 0xc4200b0008, 0x4cffc80, 0xc4428808c0, 0x6f46c84800, 0x3, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:160 +0x2c3
github.com/ethereum/go-ethereum/rpc.(*Server).ServeCodec(0xc420c81440, 0x4cffc80, 0xc4428808c0, 0x3)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/server.go:216 +0x94
created by github.com/ethereum/go-ethereum/rpc.DialInProc.func1
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/inproc.go:29 +0xbd

goroutine 127 [select, 5751 minutes]:
github.com/ethereum/go-ethereum/rpc.(*Client).dispatch(0xc420c837a0, 0x4cff080, 0xc422226280)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:541 +0x3df
created by github.com/ethereum/go-ethereum/rpc.newClient
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:244 +0x2d0

goroutine 128 [select, 5751 minutes]:
net.(*pipe).read(0xc422226280, 0xc420120c00, 0x200, 0x200, 0xc420cde580, 0x40449c7, 0x200)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/pipe.go:164 +0x1d7
net.(*pipe).Read(0xc422226280, 0xc420120c00, 0x200, 0x200, 0x0, 0x200, 0xc4222210a0)
    /usr/local/Cellar/go/1.10.2/libexec/src/net/pipe.go:147 +0x4d
encoding/json.(*Decoder).refill(0xc4443c05a0, 0xc420cde640, 0x4020da6)
    /usr/local/Cellar/go/1.10.2/libexec/src/encoding/json/stream.go:159 +0x132
encoding/json.(*Decoder).readValue(0xc4443c05a0, 0x0, 0x0, 0x49b24e0)
    /usr/local/Cellar/go/1.10.2/libexec/src/encoding/json/stream.go:134 +0x23d
encoding/json.(*Decoder).Decode(0xc4443c05a0, 0x4a2c320, 0xc422221060, 0xbe549f6cd10c9f0c, 0xc420cde710)
    /usr/local/Cellar/go/1.10.2/libexec/src/encoding/json/stream.go:63 +0x78
github.com/ethereum/go-ethereum/rpc.(*Client).read.func1(0x4a86a80, 0xc422221080, 0xc422226280, 0x9cdaf30, 0xc422226280)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:682 +0x79
github.com/ethereum/go-ethereum/rpc.(*Client).read(0xc420c837a0, 0x4cff080, 0xc422226280, 0xf1010892f3ad00fb, 0x747d7a6be8a686d4)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:695 +0x14d
created by github.com/ethereum/go-ethereum/rpc.(*Client).dispatch
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/rpc/client.go:517 +0x71

goroutine 10 [select, 5408 minutes]:
github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).loop(0xc4201f4820)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:316 +0x7d7
created by github.com/ethereum/go-ethereum/eth/fetcher.(*Fetcher).Start
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/fetcher/fetcher.go:178 +0x3f

goroutine 121796 [semacquire, 5407 minutes]:
sync.runtime_SemacquireMutex(0xc420258a98, 0x0)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420258a94)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).Open(0xc420258a80, 0x4, 0x201c547, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:471 +0xc3
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*iStorage).Open(0xc42000cd00, 0x4, 0x201c547, 0xc422277980, 0xc420962600, 0x28, 0x5a53d00)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.go:15 +0x4b
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).open.func1(0xc46d1be8e0, 0xc420083400, 0xc425446980)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:349 +0x6e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Cache).Get(0xc420083400, 0x0, 0x201c547, 0xc46bc4a9b8, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:389 +0x1ec
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).open(0xc4201de810, 0xc48def7950, 0x28, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:347 +0x7c
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).find(0xc4201de810, 0xc48def7950, 0xc42c291b30, 0x28, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:377 +0x75
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).get.func1(0x2, 0xc48def7950, 0x20)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:176 +0x44e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).walkOverlapping(0xc49a4e7db0, 0x0, 0x0, 0x0, 0xc42c291b30, 0x28, 0x28, 0xc46bc4ace8, 0xc46bc4acb8)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:125 +0x457
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).get(0xc49a4e7db0, 0x0, 0x0, 0x0, 0xc42c291b30, 0x28, 0x28, 0x0, 0x7a5f900, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:160 +0x2ae
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).get(0xc42029ab60, 0x0, 0x0, 0x0, 0x0, 0xc48cfb62c0, 0x20, 0x20, 0x13f2f28e3, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:779 +0x397
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Get(0xc42029ab60, 0xc48cfb62c0, 0x20, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:845 +0x12f
github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Get(0xc4202bf1e0, 0xc48cfb62c0, 0x20, 0x20, 0xc48cfb62a0, 0x191, 0x0, 0x0, 0xc46b1e57a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:110 +0x5a
github.com/ethereum/go-ethereum/trie.(*Database).Node(0xc43f9385a0, 0x44c4acae0073980b, 0x4e713150f0c1e867, 0x40d22fde8ed5a5d8, 0x56988cff7a2a92d5, 0x4013ac8, 0x20, 0x4adfd80, 0x1, 0xc48cfb62a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:364 +0x170
github.com/ethereum/go-ethereum/core.(*BlockChain).TrieNode(0xc43f956000, 0x44c4acae0073980b, 0x4e713150f0c1e867, 0x40d22fde8ed5a5d8, 0x56988cff7a2a92d5, 0xc46d5b8cc0, 0xbee11d6ae9b690b8, 0x12dccaf8dcaa, 0x557a3c0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:644 +0x69
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handleMsg(0xc420226300, 0xc4200d3740, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:549 +0x2065
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc4200d3740, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:309 +0x846
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc453d55800, 0x4cf2c60, 0xc46b6917a0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc46b6917a0, 0xc453d55800, 0x4cf2c60, 0xc46b6917a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 131951 [select, 5407 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc49f5f7680)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 132029 [select, 5405 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc450b7e0c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 121769 [semacquire, 5407 minutes]:
sync.runtime_Semacquire(0xc453d55834)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc453d55828)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc453d55800, 0x4999940, 0xc457609b00, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc453d55800)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 132142 [semacquire, 5405 minutes]:
sync.runtime_Semacquire(0xc453d554d4)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc453d554c8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc453d554a0, 0x4999940, 0xc437abcb00, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc453d554a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 132048 [semacquire, 5405 minutes]:
sync.runtime_SemacquireMutex(0xc420255eec, 0xc49bd94b00)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
sync.(*RWMutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:93 +0x2d
github.com/ethereum/go-ethereum/core.(*TxPool).Pending(0xc420255dc0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:527 +0x65
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncTransactions(0xc420226300, 0xc49f5f60c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:48 +0x4c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc49f5f60c0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:286 +0x712
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc453d54fc0, 0x4cf2c60, 0xc498f23b90, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc498f23b90, 0xc453d54fc0, 0x4cf2c60, 0xc498f23b90)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 132028 [semacquire, 5405 minutes]:
sync.runtime_Semacquire(0xc4389eac94)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4389eac88)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc4389eac60, 0x4999940, 0xc4373fa400, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc4389eac60)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 132139 [select, 5405 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc49f5f60c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 132236 [semacquire, 5405 minutes]:
sync.runtime_Semacquire(0xc4b1e2aed4)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4b1e2aec8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc4b1e2aea0, 0x4999940, 0xc43b55ab80, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc4b1e2aea0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 132261 [select, 5405 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc450b7ed80)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 132154 [select, 5405 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc49f5f6180)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 131942 [semacquire, 5407 minutes]:
sync.runtime_SemacquireMutex(0xc420255eec, 0xc42ae81500)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
sync.(*RWMutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:93 +0x2d
github.com/ethereum/go-ethereum/core.(*TxPool).Pending(0xc420255dc0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:527 +0x65
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncTransactions(0xc420226300, 0xc49f5f7680)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:48 +0x4c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc49f5f7680, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:286 +0x712
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc48ca8b0e0, 0x4cf2c60, 0xc46d4a6a80, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc46d4a6a80, 0xc48ca8b0e0, 0x4cf2c60, 0xc46d4a6a80)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 132166 [semacquire, 5405 minutes]:
sync.runtime_SemacquireMutex(0xc420255eec, 0xc4444ea500)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
sync.(*RWMutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:93 +0x2d
github.com/ethereum/go-ethereum/core.(*TxPool).Pending(0xc420255dc0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:527 +0x65
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncTransactions(0xc420226300, 0xc49f5f6180)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:48 +0x4c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc49f5f6180, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:286 +0x712
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc453d554a0, 0x4cf2c60, 0xc453084000, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc453084000, 0xc453d554a0, 0x4cf2c60, 0xc453084000)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 132844 [semacquire, 5397 minutes]:
sync.runtime_SemacquireMutex(0xc420255eec, 0xc497f53200)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
sync.(*RWMutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:93 +0x2d
github.com/ethereum/go-ethereum/core.(*TxPool).Pending(0xc420255dc0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:527 +0x65
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncTransactions(0xc420226300, 0xc450b7e240)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:48 +0x4c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc450b7e240, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:286 +0x712
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc4b1f966c0, 0x4cf2c60, 0xc4c0e74230, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc4c0e74230, 0xc4b1f966c0, 0x4cf2c60, 0xc4c0e74230)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 132869 [semacquire, 5397 minutes]:
sync.runtime_Semacquire(0xc4b1f966f4)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4b1f966e8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc4b1f966c0, 0x4999940, 0xc433a6dd80, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc4b1f966c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 121815 [semacquire, 5407 minutes]:
sync.runtime_SemacquireMutex(0xc420258a98, 0xc4c1d97100)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420258a94)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileWrap).Close(0xc43ac48ff0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:600 +0x47
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Reader).Release(0xc42faf84e0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table/reader.go:1001 +0x1f0
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*mBucket).delete(0xc442909e60, 0xc420083400, 0xc425446980, 0xc42139b475, 0x0, 0x2016ce6, 0xc4706f11c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:171 +0x578
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Cache).delete(0xc420083400, 0xc452942620, 0xc452942620)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:329 +0x6d
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Node).unrefLocked(0xc452942620)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:632 +0x78
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Handle).Release(0xc4706f11c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:658 +0x67
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*lru).Promote(0xc4200832c0, 0xc46afbbd50)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/lru.go:114 +0xa7
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Cache).Get(0xc420083400, 0x0, 0x2016f91, 0xc4c1d97548, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:400 +0x1d4
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).open(0xc4201de810, 0xc49bfc20f0, 0x28, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:347 +0x7c
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).find(0xc4201de810, 0xc49bfc20f0, 0xc42c291500, 0x28, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:377 +0x75
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).get.func1(0x3, 0xc49bfc20f0, 0x20)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:176 +0x44e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).walkOverlapping(0xc49a4e7db0, 0x0, 0x0, 0x0, 0xc42c291500, 0x28, 0x28, 0xc4c1d97878, 0xc4c1d97848)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:125 +0x457
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).get(0xc49a4e7db0, 0x0, 0x0, 0x0, 0xc42c291500, 0x28, 0x28, 0x0, 0x9190000, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:160 +0x2ae
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).get(0xc42029ab60, 0x0, 0x0, 0x0, 0x0, 0xc4886da880, 0x20, 0x20, 0x13f2f28e3, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:779 +0x397
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Get(0xc42029ab60, 0xc4886da880, 0x20, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:845 +0x12f
github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Get(0xc4202bf1e0, 0xc4886da880, 0x20, 0x20, 0x0, 0x2c, 0xc4c1cd2454, 0x0, 0x2c)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:110 +0x5a
github.com/ethereum/go-ethereum/trie.(*Database).node(0xc43f9385a0, 0x21b459bc033e434c, 0xe779ee8e41cbf951, 0xc7a697766834fbc9, 0x50a2ce512a202cf0, 0x0, 0x2, 0xf)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:345 +0x129
github.com/ethereum/go-ethereum/trie.(*Trie).resolveHash(0xc46aeab810, 0xc4886da680, 0x20, 0x20, 0xc4545fc820, 0x4, 0x41, 0xc4c1cd2243, 0x211, 0x23d, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:436 +0x159
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf96a0, 0xc47da2e900, 0xc4545fc820, 0x41, 0x41, 0x4, 0xc4c1d97ec0, 0x428ab1e, 0xc4886da580, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:169 +0xef
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf77e0, 0xc48cbba500, 0xc4545fc820, 0x41, 0x41, 0x3, 0x4cf77e0, 0xc48cbba500, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf96a0, 0xc47da2e6e0, 0xc4545fc820, 0x41, 0x41, 0x3, 0xc4c1d981e0, 0x428ab1e, 0xc4886da300, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:173 +0x221
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf77e0, 0xc48cbba3c0, 0xc4545fc820, 0x41, 0x41, 0x2, 0x4cf77e0, 0xc48cbba3c0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf96a0, 0xc42b35e5e0, 0xc4545fc820, 0x41, 0x41, 0x2, 0xc48b48e0e0, 0x20, 0xc4886da020, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:173 +0x221
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf77e0, 0xc48b75a8c0, 0xc4545fc820, 0x41, 0x41, 0x1, 0x410f4a6574b29ec3, 0xc4c1d98638, 0x415602d, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc46aeab810, 0x4cf77e0, 0xc48cbba280, 0xc4545fc820, 0x41, 0x41, 0x0, 0x20, 0xc46aeab850, 0x20, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).TryGet(0xc46aeab810, 0xc46aeab850, 0x20, 0x20, 0xc46aeab850, 0x20, 0x20, 0x4943160, 0xc432bb9120)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:135 +0x7b
github.com/ethereum/go-ethereum/trie.(*SecureTrie).TryGet(0xc46aeab810, 0xc4886da2a0, 0x20, 0x20, 0xc46aeab810, 0x0, 0xc459899c00, 0x20, 0x5a53d90)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/secure_trie.go:80 +0x7b
github.com/ethereum/go-ethereum/core/state.(*stateObject).GetState(0xc448f09960, 0x4cfd9a0, 0xc4238b0ff0, 0xea9017470268b7fd, 0x32682ab1be4dcf24, 0xf3bcb9597baa0b80, 0x718512760b12d66d, 0x0, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/state_object.go:169 +0x179
github.com/ethereum/go-ethereum/core/state.(*StateDB).GetState(0xc4a7d70700, 0x93eaf7f8b6d69fdd, 0xbbea67be2b999729, 0x268b7fd3c6f313e, 0xbe4dcf24ea901747, 0x7baa0b8032682ab1, 0xb12d66df3bcb959, 0x71851276, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:242 +0xb2
github.com/ethereum/go-ethereum/core/vm.opSload(0xc450897ec8, 0xc4b7b42000, 0xc44e25e9c0, 0xc4814a5040, 0xc4814a5060, 0x2bc, 0x1388, 0x32, 0x61a8, 0xc434c75500)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/vm/instructions.go:607 +0x104
github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc4b7b42000, 0xc44e25e9c0, 0xc45aa4a370, 0x44, 0x44, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/vm/interpreter.go:235 +0x582
github.com/ethereum/go-ethereum/core/vm.run(0xc434c75500, 0xc44e25e9c0, 0xc45aa4a370, 0x44, 0x44, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/vm/evm.go:64 +0x1e6
github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc434c75500, 0x4cf08e0, 0xc47ed1f3a0, 0x93eaf7f8b6d69fdd, 0xbbea67be2b999729, 0x3c6f313e, 0xc45aa4a370, 0x44, 0x44, 0x417f8, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/vm/evm.go:213 +0x4f6
github.com/ethereum/go-ethereum/core.(*StateTransition).TransitionDb(0xc46aeab420, 0x4cff140, 0xc49991b2c0, 0xc45a06a7d0, 0xc46aeab420, 0xc49991b2c0, 0xc49991b2c0, 0x4b30a40)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_transition.go:213 +0x3c1
github.com/ethereum/go-ethereum/core.ApplyMessage(0xc434c75500, 0x4cff140, 0xc49991b2c0, 0xc45a06a7d0, 0x9505bb159546381c, 0xc7420aee, 0xc4814a4dc0, 0xf234a84350b93b2, 0xfe96aeac3114e500, 0x47033a54, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_transition.go:132 +0x5b
github.com/ethereum/go-ethereum/core.ApplyTransaction(0x554ef00, 0x4cf2560, 0xc43f956000, 0x0, 0xc45a06a7d0, 0xc4a7d70700, 0xc4229feb40, 0xc42fd20c60, 0xc4c1d9e728, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_processor.go:99 +0x29f
github.com/ethereum/go-ethereum/core.(*StateProcessor).Process(0xc4201f2da0, 0xc4207ba750, 0xc4a7d70700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_processor.go:71 +0x4d6
github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain(0xc43f956000, 0xc47c8fa580, 0xa1, 0xa1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1151 +0x16ad
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertChain(0xc43f956000, 0xc47c8fa580, 0xa1, 0xa1, 0x55a2740, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:1003 +0x67
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).importBlockResults(0xc42029b380, 0xc47c8fa000, 0xa1, 0xa1, 0xa1, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1361 +0x431
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processFullSyncContent(0xc42029b380, 0xc054ec32b7d8805b, 0xc4a8746798)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1335 +0x79
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).(github.com/ethereum/go-ethereum/eth/downloader.processFullSyncContent)-fm(0x8, 0x4bfbc10)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:472 +0x2a
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc42029b380, 0xc4a0752120, 0xc47e81a4a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0x59
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:484 +0xaf

goroutine 132137 [semacquire, 5405 minutes]:
sync.runtime_Semacquire(0xc453d54ff4)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc453d54fe8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc453d54fc0, 0x4999940, 0xc437abca80, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc453d54fc0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 122655 [semacquire, 5407 minutes]:
sync.runtime_Semacquire(0xc4282b14d4)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4282b14c8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc4282b14a0, 0x4999940, 0xc45ab96500, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc4282b14a0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 121779 [select, 5407 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc4200d3740)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 122641 [select, 5408 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc450b7e300)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 156134 [semacquire, 4343 minutes]:
sync.runtime_Semacquire(0xc43f956344)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc43f956338)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/core.(*BlockChain).Stop(0xc43f956000)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/blockchain.go:658 +0xaa
github.com/ethereum/go-ethereum/eth.(*Ethereum).Stop(0xc4203048c0, 0xc4202ab740, 0xc420cdcf58)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/backend.go:413 +0x45
github.com/ethereum/go-ethereum/node.(*Node).Stop(0xc42010c280, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/node/node.go:419 +0x215
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/cmd/utils/cmd.go:76 +0x13d

goroutine 121826 [semacquire, 5416 minutes]:
sync.runtime_Semacquire(0xc42029b4cc)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc42029b4c0)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Cancel(0xc42029b380)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:525 +0x41
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync(0xc42029b380, 0xc4244ed7c0, 0x5, 0x8, 0x5, 0xc4244ed7c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:500 +0x156
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc42029b380, 0xc4a0977680, 0x90da688a53a386b8, 0x7de1c29b0b3cc3f, 0x9f3749da42011d78, 0xd3437bcde99a8ff5, 0xc47c8f5d80, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:474 +0x666
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc42029b380, 0xc442fea8a0, 0x10, 0x90da688a53a386b8, 0x7de1c29b0b3cc3f, 0x9f3749da42011d78, 0xd3437bcde99a8ff5, 0xc47c8f5d80, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:399 +0x38b
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc42029b380, 0xc442fea8a0, 0x10, 0x90da688a53a386b8, 0x7de1c29b0b3cc3f, 0x9f3749da42011d78, 0xd3437bcde99a8ff5, 0xc47c8f5d80, 0x0, 0x4115eef, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:317 +0x97
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).synchronise(0xc420226300, 0xc4200d3740)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:200 +0x358
created by github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncer
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:155 +0x21e

goroutine 132015 [semacquire, 5405 minutes]:
sync.runtime_SemacquireMutex(0xc420255eec, 0xc4504e4e00)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
sync.(*RWMutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:93 +0x2d
github.com/ethereum/go-ethereum/core.(*TxPool).Pending(0xc420255dc0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:527 +0x65
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncTransactions(0xc420226300, 0xc450b7e0c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:48 +0x4c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc450b7e0c0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:286 +0x712
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc4389eac60, 0x4cf2c60, 0xc49450c230, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc49450c230, 0xc4389eac60, 0x4cf2c60, 0xc49450c230)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 132785 [select, 5397 minutes]:
github.com/ethereum/go-ethereum/eth.(*peer).broadcast(0xc450b7e240)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:116 +0x1d2
created by github.com/ethereum/go-ethereum/eth.(*peerSet).Register
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/peer.go:428 +0x121

goroutine 132220 [semacquire, 5405 minutes]:
sync.runtime_SemacquireMutex(0xc420255eec, 0xc45bb6ca00)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/mutex.go:134 +0x108
sync.(*RWMutex).Lock(0xc420255ee8)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/rwmutex.go:93 +0x2d
github.com/ethereum/go-ethereum/core.(*TxPool).Pending(0xc420255dc0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:527 +0x65
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).syncTransactions(0xc420226300, 0xc450b7ed80)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/sync.go:48 +0x4c
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc450b7ed80, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:286 +0x712
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc4b1e2aea0, 0x4cf2c60, 0xc4c24ca150, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc4c24ca150, 0xc4b1e2aea0, 0x4cf2c60, 0xc4c24ca150)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201

goroutine 131884 [semacquire, 5405 minutes]:
sync.runtime_Semacquire(0xc48ca8b114)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc48ca8b108)
    /usr/local/Cellar/go/1.10.2/libexec/src/sync/waitgroup.go:129 +0x72
github.com/ethereum/go-ethereum/p2p.(*Peer).run(0xc48ca8b0e0, 0x4999940, 0xc46e242680, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:235 +0x3aa
github.com/ethereum/go-ethereum/p2p.(*Server).runPeer(0xc42010ca00, 0xc48ca8b0e0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:895 +0x12b
created by github.com/ethereum/go-ethereum/p2p.(*Server).run
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:647 +0x12bf

goroutine 122669 [syscall, 5407 minutes]:
syscall.Syscall(0x5, 0xc43c0bf740, 0x1000000, 0x0, 0x0, 0x0, 0xc4793cd008)
    /usr/local/Cellar/go/1.10.2/libexec/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.Open(0xc43c0bf700, 0x36, 0x1000000, 0x0, 0xc4793cd058, 0x40e966e, 0xc43c0bf700)
    /usr/local/Cellar/go/1.10.2/libexec/src/syscall/zsyscall_darwin_amd64.go:880 +0x90
os.openFileNolog(0xc43c0bf700, 0x36, 0x0, 0x0, 0x2, 0xc43c0bf700, 0x36)
    /usr/local/Cellar/go/1.10.2/libexec/src/os/file_unix.go:168 +0xad
os.OpenFile(0xc43c0bf700, 0x36, 0x0, 0xc400000000, 0x36, 0x31a3, 0xc42c25fc80)
    /usr/local/Cellar/go/1.10.2/libexec/src/os/file.go:269 +0x5f
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).Open(0xc420258a80, 0x4, 0x200db4d, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:476 +0x1af
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*iStorage).Open(0xc42000cd00, 0x4, 0x200db4d, 0xc49902fe00, 0xc458e39880, 0x28, 0x5a53000)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/storage.go:15 +0x4b
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).open.func1(0xc46c88a5d0, 0xc420083400, 0xc425446980)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:349 +0x6e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache.(*Cache).Get(0xc420083400, 0x0, 0x200db4d, 0xc4793cd2f0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/cache/cache.go:389 +0x1ec
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).open(0xc4201de810, 0xc426f88050, 0x28, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:347 +0x7c
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).find(0xc4201de810, 0xc426f88050, 0xc42c25fc80, 0x28, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/table.go:377 +0x75
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).get.func1(0x5, 0xc426f88050, 0x20)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:176 +0x44e
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).walkOverlapping(0xc49a4e7db0, 0x0, 0x0, 0x0, 0xc42c25fc80, 0x28, 0x28, 0xc4793cd620, 0xc4793cd5f0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:125 +0x457
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*version).get(0xc49a4e7db0, 0x0, 0x0, 0x0, 0xc42c25fc80, 0x28, 0x28, 0x0, 0x8929000, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/version.go:160 +0x2ae
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).get(0xc42029ab60, 0x0, 0x0, 0x0, 0x0, 0xc4883ef660, 0x20, 0x20, 0x13f2f28e3, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:779 +0x397
github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).Get(0xc42029ab60, 0xc4883ef660, 0x20, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/vendor/github.com/syndtr/goleveldb/leveldb/db.go:845 +0x12f
github.com/ethereum/go-ethereum/ethdb.(*LDBDatabase).Get(0xc4202bf1e0, 0xc4883ef660, 0x20, 0x20, 0x0, 0x2c, 0xc4249128d4, 0x0, 0x2c)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/ethdb/database.go:110 +0x5a
github.com/ethereum/go-ethereum/trie.(*Database).node(0xc43f9385a0, 0x8c5fef865a08576f, 0xcd7b27cef888aca2, 0xd74b38736139e508, 0x32c78326d3eaaf68, 0x1181, 0x2, 0xf)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/database.go:345 +0x129
github.com/ethereum/go-ethereum/trie.(*Trie).resolveHash(0xc47093e380, 0xc4883ef560, 0x20, 0x20, 0xc4813bc7d0, 0x5, 0x41, 0xc4249126c3, 0x211, 0x23d, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:436 +0x159
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf96a0, 0xc480662a40, 0xc4813bc7d0, 0x41, 0x41, 0x5, 0xc4793cdc68, 0x428ab1e, 0xc4883ef440, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:169 +0xef
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf77e0, 0xc453d33680, 0xc4813bc7d0, 0x41, 0x41, 0x4, 0x4cf77e0, 0xc453d33680, 0x0, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf96a0, 0xc431b91ec0, 0xc4813bc7d0, 0x41, 0x41, 0x4, 0xc4813bc460, 0x4d, 0x50, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:173 +0x221
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf77e0, 0xc473e91b80, 0xc4813bc7d0, 0x41, 0x41, 0x3, 0xc42017c000, 0x5a53000, 0x5a53000, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf77e0, 0xc48f415400, 0xc4813bc7d0, 0x41, 0x41, 0x2, 0x200e5bd681334667, 0x1ee800bc104a3cbb, 0x4013279, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf77e0, 0xc4228f5180, 0xc4813bc7d0, 0x41, 0x41, 0x1, 0x5df0884c459b3eb4, 0xc4793ce3e0, 0x415602d, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc47093e380, 0x4cf77e0, 0xc453d33540, 0xc4813bc7d0, 0x41, 0x41, 0x0, 0x20, 0xc47093e3c0, 0x20, ...)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:161 +0x6d7
github.com/ethereum/go-ethereum/trie.(*Trie).TryGet(0xc47093e380, 0xc47093e3c0, 0x20, 0x20, 0xc47093e3c0, 0x20, 0x20, 0x4942e60, 0xc4b2e2d4c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:135 +0x7b
github.com/ethereum/go-ethereum/trie.(*SecureTrie).TryGet(0xc47093e380, 0xc4883ef400, 0x14, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/secure_trie.go:80 +0x7b
github.com/ethereum/go-ethereum/core/state.(*StateDB).getStateObject(0xc456959180, 0x5f78c320e07c8b92, 0xda041d4443c147dd, 0x8fb6d3a2, 0xffffffffffffffff)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:372 +0x130
github.com/ethereum/go-ethereum/core/state.(*StateDB).GetNonce(0xc456959180, 0x5f78c320e07c8b92, 0xda041d4443c147dd, 0xc48fb6d3a2, 0x1)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:200 +0x3f
github.com/ethereum/go-ethereum/core.(*TxPool).validateTx(0xc420255dc0, 0xc45a745dd0, 0xc8f9e4607b206f00, 0x884a984b09726b64, 0x2780114ae8c3948f)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:580 +0x231
github.com/ethereum/go-ethereum/core.(*TxPool).add(0xc420255dc0, 0xc45a745dd0, 0x0, 0xc4807a5180, 0x1, 0x1)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:614 +0xde
github.com/ethereum/go-ethereum/core.(*TxPool).addTxsLocked(0xc420255dc0, 0xc4806627c0, 0x1, 0x4, 0xc4883ef300, 0x4adfd80, 0xc4883ef300, 0xc4793cf070)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:818 +0x12e
github.com/ethereum/go-ethereum/core.(*TxPool).addTxs(0xc420255dc0, 0xc4806627c0, 0x1, 0x4, 0xc8f9e4607b206f00, 0x0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:806 +0xaf
github.com/ethereum/go-ethereum/core.(*TxPool).AddRemotes(0xc420255dc0, 0xc4806627c0, 0x1, 0x4, 0x2780114ae8c3948f, 0x12cfc674ecbf, 0x557a3c0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/tx_pool.go:780 +0x4e
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handleMsg(0xc420226300, 0xc450b7e300, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:683 +0x491
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc420226300, 0xc450b7e300, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:309 +0x846
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func1(0xc4282b14a0, 0x4cf2c60, 0xc46ed8fdc0, 0x0, 0x0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/eth/handler.go:142 +0x1db
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc46ed8fdc0, 0xc4282b14a0, 0x4cf2c60, 0xc46ed8fdc0)
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:353 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
    /Users/robmulholand/go/src/github.com/ethereum/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:352 +0x201
holiman commented 6 years ago

It looks like there's a syscall trying to create a file, which has blocked other readers for 5407 minutes, that is, 90 hours or almost four days. This is odd -- are you using some particular filesystem? As in, is this a normal unix environment, or is there some non-standard filesystem used for the data directory?

rmulhol commented 6 years ago

Thanks for the insight! Yes, using ZFS (running on OS X) to manage external drives.

AFDudley commented 6 years ago

@holiman can we get some pointers for debugging this issue? It may be an issues with the osx implementation of ZFS, while running a node on Linux I don't think I ran into this issue.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.