search

libp2p / go-libp2p

libp2p implementation in Go
MIT License
6.05k stars 1.07k forks source link

Deadlock on host.Close() #933

Closed hsanjuan closed 4 years ago

hsanjuan commented 4 years ago

Cluster has suffered a deadlock on host.Close() which looks a libp2p issue. I can't reproduce this consistently :/

The deadlock happens on https://github.com/ipfs/ipfs-cluster/blob/583270729017205d48dd0a0af5f1266016a77871/cmdutils/cmdutils.go#L131 . The the dht is Closed. But the host.Close() call hangs. The datastore should still be open.

I am on libp2p v0.8.3 (all should be very recent versions).

Stack trace (the first one is very relevant).. as you see it is relatively short because there is not much left to shutdown at this point:

SIGABRT: abort
PC=0x46b7a1 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x229b7c8, 0x80, 0x0, 0x0, 0x7ffc00000000, 0x46b3c5, 0xc3812, 0x19023cfd, 0x7ffc8c28b828, 0x40cf6f, ...)
        /usr/lib64/go/1.14/src/runtime/sys_linux_amd64.s:567 +0x21
runtime.futexsleep(0x229b7c8, 0x0, 0xffffffffffffffff)
        /usr/lib64/go/1.14/src/runtime/os_linux.go:45 +0x46
runtime.notesleep(0x229b7c8)
        /usr/lib64/go/1.14/src/runtime/lock_futex.go:151 +0x9f
runtime.stopm()
        /usr/lib64/go/1.14/src/runtime/proc.go:1828 +0xc0
runtime.findrunnable(0xc00004c800, 0x0)
        /usr/lib64/go/1.14/src/runtime/proc.go:2360 +0xa0d
runtime.schedule()
        /usr/lib64/go/1.14/src/runtime/proc.go:2520 +0x2fc
runtime.park_m(0xc000102c00)
        /usr/lib64/go/1.14/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x0)
        /usr/lib64/go/1.14/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [semacquire, 16 minutes]:
sync.runtime_SemacquireMutex(0xc0003bae28, 0xc00039c900, 0x0)
        /usr/lib64/go/1.14/src/runtime/sema.go:71 +0x47
sync.(*RWMutex).Lock(0xc0003bae20)
        /usr/lib64/go/1.14/src/sync/rwmutex.go:103 +0x88
github.com/libp2p/go-libp2p-swarm.(*Swarm).StopNotify(0xc0003bad80, 0x17b37c0, 0xc00017f040)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm.go:491 +0x3d
github.com/libp2p/go-libp2p/p2p/host/basic.newNatManager.func1(0xc00027cf48, 0x0)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/basic/natmgr.go:59 +0x47
github.com/jbenet/goprocess.(*process).doClose(0xc0004528a0)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:235 +0x2db
github.com/jbenet/goprocess.(*process).Close(0xc0004528a0, 0x1783020, 0xc00006e0c0)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:177 +0xd7
github.com/libp2p/go-libp2p/p2p/host/basic.(*natManager).Close(0xc00017f040, 0xc00017f8b0, 0xc00017f8b0)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/basic/natmgr.go:71 +0x34
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).Close.func1()
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/basic/basic_host.go:780 +0xf9
sync.(*Once).doSlow(0xc00012ede8, 0xc00027d0a0)
        /usr/lib64/go/1.14/src/sync/once.go:66 +0xec
sync.(*Once).Do(...)
        /usr/lib64/go/1.14/src/sync/once.go:57
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).Close(0xc00012edd0, 0x0, 0x0)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/basic/basic_host.go:777 +0x6a
github.com/libp2p/go-libp2p/p2p/host/routed.(*RoutedHost).Close(0xc0003ae180, 0x0, 0x0)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/routed/routed.go:189 +0x33
github.com/ipfs/ipfs-cluster/cmdutils.HandleSignals(0x17a9480, 0xc00011d540, 0xc00019c6c0, 0xc0004d58c0, 0x17c3b20, 0xc0003ae180, 0xc00019d8f0, 0x17b9b80, 0xc0004522a0, 0xc0004522a0, ...)
        src/github.com/ipfs/ipfs-cluster/cmdutils/cmdutils.go:131 +0x20b
main.daemon(0xc0004d4580, 0x0, 0x0)
        src/github.com/ipfs/ipfs-cluster/cmd/ipfs-cluster-service/daemon.go:105 +0x61d
github.com/urfave/cli.HandleAction(0x121a000, 0x15d7ae0, 0xc0004d4580, 0xc0004d4580, 0x0)
        pkg/mod/github.com/urfave/cli@v1.22.4/app.go:526 +0x11a
github.com/urfave/cli.Command.Run(0x144fd3f, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x147aa07, 0x24, 0x0, ...)
        pkg/mod/github.com/urfave/cli@v1.22.4/command.go:174 +0x57a
github.com/urfave/cli.(*App).Run(0xc000139880, 0xc00000e0a0, 0x2, 0x2, 0x0, 0x0)
        pkg/mod/github.com/urfave/cli@v1.22.4/app.go:279 +0x7c7
main.main()
        src/github.com/ipfs/ipfs-cluster/cmd/ipfs-cluster-service/main.go:571 +0x1189

goroutine 6 [select, 16 minutes]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc00022c060)
        pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:71 +0x128
created by github.com/ipfs/go-log/writer.NewMirrorWriter
        pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:36 +0xb9

goroutine 8 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00008e820)
        pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x57

goroutine 21 [select, 2 minutes]:
github.com/libp2p/go-libp2p-connmgr.(*BasicConnMgr).background(0xc000157b00)
        pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.1/connmgr.go:200 +0x163
created by github.com/libp2p/go-libp2p-connmgr.NewConnManager
        pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.1/connmgr.go:116 +0x1e5

goroutine 22 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc0004221e0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 23 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422240)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 24 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc0004222a0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 25 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422300)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 26 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422360)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 27 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc0004223c0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 28 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422420)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 29 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422480)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 30 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc0004224e0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 31 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422540)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 32 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc0004225a0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 33 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc00012df80, 0xc000422600)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 50 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422660)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 51 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc0004226c0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 52 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422720)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 53 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422780)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 54 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc0004227e0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 55 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422840)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 56 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc0004228a0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 57 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422900)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 58 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422960)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 59 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc0004229c0)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 60 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422a20)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 61 [select]:
github.com/ipfs/ipfs-cluster/config.(*Manager).watchSave(0xc000188080, 0xc000422a80)
        src/github.com/ipfs/ipfs-cluster/config/config.go:172 +0x12b
created by github.com/ipfs/ipfs-cluster/config.(*Manager).RegisterComponent
        src/github.com/ipfs/ipfs-cluster/config/config.go:287 +0x7e

goroutine 62 [select, 16 minutes]:
github.com/dgraph-io/badger/y.(*WaterMark).process(0xc00050edc0, 0xc0002e8400)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/y/watermark.go:229 +0x2d7
created by github.com/dgraph-io/badger/y.(*WaterMark).Init
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/y/watermark.go:78 +0xa6

goroutine 63 [select, 16 minutes]:
github.com/dgraph-io/badger/y.(*WaterMark).process(0xc00050ee00, 0xc0002e8400)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/y/watermark.go:229 +0x2d7
created by github.com/dgraph-io/badger/y.(*WaterMark).Init
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/y/watermark.go:78 +0xa6

goroutine 64 [select, 2 minutes]:
github.com/dgraph-io/badger.(*DB).updateSize(0xc0000b3500, 0xc0002e8460)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:1007 +0x134
created by github.com/dgraph-io/badger.Open
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:289 +0x82f

goroutine 67 [select]:
github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00017c460, 0xc00050a020)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/levels.go:347 +0x20c
created by github.com/dgraph-io/badger.(*levelsController).startCompact
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/levels.go:328 +0x88

goroutine 68 [select]:
github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00017c460, 0xc00050a020)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/levels.go:347 +0x20c
created by github.com/dgraph-io/badger.(*levelsController).startCompact
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/levels.go:328 +0x88

goroutine 69 [chan receive, 16 minutes]:
github.com/dgraph-io/badger.(*DB).flushMemtable(0xc0000b3500, 0xc00050a040, 0x0, 0x0)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:921 +0xb7
github.com/dgraph-io/badger.Open.func4(0xc000116e30)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:306 +0x37
created by github.com/dgraph-io/badger.Open
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:305 +0x1290

goroutine 72 [select, 16 minutes]:
github.com/dgraph-io/badger.(*DB).doWrites(0xc0000b3500, 0xc00050a200)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:712 +0x3fa
created by github.com/dgraph-io/badger.Open
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:340 +0xd9c

goroutine 71 [select, 16 minutes]:
github.com/dgraph-io/badger.(*valueLog).flushDiscardStats(0xc0000b3670)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/value.go:1567 +0x175
created by github.com/dgraph-io/badger.(*valueLog).open
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/value.go:832 +0x57

goroutine 73 [chan receive, 16 minutes]:
github.com/dgraph-io/badger.(*valueLog).waitOnGC(0xc0000b3670, 0xc00050a220)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/value.go:1474 +0x63
created by github.com/dgraph-io/badger.Open
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:343 +0xe63

goroutine 74 [select, 16 minutes]:
github.com/dgraph-io/badger.(*publisher).listenForUpdates(0xc000505590, 0xc00050a240)
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/publisher.go:67 +0x161
created by github.com/dgraph-io/badger.Open
        pkg/mod/github.com/dgraph-io/badger@v1.6.1/db.go:346 +0xf2a

goroutine 75 [select, 2 minutes]:
github.com/libp2p/go-libp2p-connmgr.(*BasicConnMgr).background(0xc000158d00)
        pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.1/connmgr.go:200 +0x163
created by github.com/libp2p/go-libp2p-connmgr.NewConnManager
        pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.1/connmgr.go:116 +0x1e5

goroutine 76 [select, 16 minutes]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0xc000159600)
        pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.4/pstoremem/addr_book.go:93 +0x126
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
        pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.4/pstoremem/addr_book.go:83 +0x18e

goroutine 77 [semacquire, 16 minutes]:
sync.runtime_Semacquire(0xc0003bad88)
        /usr/lib64/go/1.14/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0003bad80)
        /usr/lib64/go/1.14/src/sync/waitgroup.go:130 +0x64
github.com/libp2p/go-libp2p-swarm.(*Swarm).teardown(0xc0003bad80, 0xc0003baf00, 0xc00017e190)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm.go:166 +0x261
github.com/jbenet/goprocess.(*process).doClose(0xc0004523c0)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:235 +0x2db
github.com/jbenet/goprocess.(*process).Close(0xc0004523c0, 0xc000020750, 0x2)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:177 +0xd7
github.com/jbenet/goprocess/context.CloseAfterContext.func1(0x17a9480, 0xc00011d540, 0x17c06a0, 0xc0004523c0)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:67 +0x112
created by github.com/jbenet/goprocess/context.CloseAfterContext
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:64 +0xa7

goroutine 82 [select, 16 minutes]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).handleEvents(0xc00033c480)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/protocol/identify/id.go:167 +0x1c6
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewIDService
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/protocol/identify/id.go:129 +0x98b

goroutine 96 [semacquire, 16 minutes]:
sync.runtime_Semacquire(0xc00050dc68)
        /usr/lib64/go/1.14/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc00050dc60)
        /usr/lib64/go/1.14/src/sync/waitgroup.go:130 +0x64
github.com/libp2p/go-libp2p-swarm.(*Swarm).notifyAll(0xc0003bad80, 0xc006c96dc0)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm.go:478 +0x13e
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2.1(0x17a9f00, 0xc000452c60, 0xc0003bad80, 0x17c3c20, 0xc0002e80e0)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm_listen.go:78 +0x11c
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2(0x17a9f00, 0xc000452c60, 0xc0003bad80, 0x17c3c20, 0xc0002e80e0)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm_listen.go:90 +0x247
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm_listen.go:69 +0x257

goroutine 166 [select, 16 minutes]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0xc000237b00)
        pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.4/pstoremem/addr_book.go:93 +0x126
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
        pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.4/pstoremem/addr_book.go:83 +0x18e

goroutine 1459 [semacquire, 16 minutes]:
sync.runtime_SemacquireMutex(0xc0004528f4, 0xc00062d500, 0x1)
        /usr/lib64/go/1.14/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc0004528f0)
        /usr/lib64/go/1.14/src/sync/mutex.go:138 +0xfc
sync.(*Mutex).Lock(...)
        /usr/lib64/go/1.14/src/sync/mutex.go:81
github.com/jbenet/goprocess.(*process).AddChild(0xc0004528a0, 0x17c06a0, 0xc005b33440)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:90 +0x27f
github.com/jbenet/goprocess.(*process).Go(0xc0004528a0, 0xc005b21c80, 0x1, 0x1)
        pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:132 +0x1bd
github.com/libp2p/go-libp2p/p2p/host/basic.(*natManager).sync(0xc00017f040)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/basic/natmgr.go:127 +0x93
github.com/libp2p/go-libp2p/p2p/host/basic.(*nmgrNetNotifiee).ListenClose(0xc00017f040, 0x17c65e0, 0xc0003bad80, 0x17c3c20, 0xc0002e80e0)
        pkg/mod/github.com/libp2p/go-libp2p@v0.8.3/p2p/host/basic/natmgr.go:237 +0x2b
github.com/libp2p/go-libp2p-swarm.(*Swarm).AddListenAddr.func2.1.1(0x17b37c0, 0xc00017f040)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm_listen.go:79 +0x58
github.com/libp2p/go-libp2p-swarm.(*Swarm).notifyAll.func1(0xc00050dc60, 0xc006c96dc0, 0x17b37c0, 0xc00017f040)
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm.go:474 +0x62
created by github.com/libp2p/go-libp2p-swarm.(*Swarm).notifyAll
        pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.3/swarm.go:472 +0x118

goroutine 726 [IO wait]:
internal/poll.runtime_pollWait(0x7f22acdb8928, 0x72, 0x0)
        /usr/lib64/go/1.14/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005b14a18, 0x72, 0x10000, 0x10000, 0x0)
        /usr/lib64/go/1.14/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib64/go/1.14/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadFrom(0xc005b14a00, 0xc005a2e000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib64/go/1.14/src/internal/poll/fd_unix.go:219 +0x1a3
net.(*netFD).readFrom(0xc005b14a00, 0xc005a2e000, 0x10000, 0x10000, 0xc005a1aec8, 0x17974e0, 0xc0003aa1b0, 0x0, 0x0)
        /usr/lib64/go/1.14/src/net/fd_unix.go:208 +0x5b
net.(*UDPConn).readFrom(0xc0002e6a88, 0xc005a2e000, 0x10000, 0x10000, 0x0, 0xc005a1af50, 0xc02000, 0xc005b181a8)
        /usr/lib64/go/1.14/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFrom(0xc0002e6a88, 0xc005a2e000, 0x10000, 0x10000, 0x17974e0, 0xc0003aa1b0, 0x0, 0x0, 0x0)
        /usr/lib64/go/1.14/src/net/udpsock.go:118 +0x5d
github.com/whyrusleeping/mdns.(*Server).recv(0xc005b18180, 0xc0002e6a88)
        pkg/mod/github.com/whyrusleeping/mdns@v0.0.0-20190826153040-b9b60ed33aa9/server.go:116 +0x9e
created by github.com/whyrusleeping/mdns.NewServer
        pkg/mod/github.com/whyrusleeping/mdns@v0.0.0-20190826153040-b9b60ed33aa9/server.go:73 +0x186

goroutine 727 [IO wait]:
internal/poll.runtime_pollWait(0x7f22acdb8848, 0x72, 0x0)
        /usr/lib64/go/1.14/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005b14a98, 0x72, 0x10000, 0x10000, 0x0)
        /usr/lib64/go/1.14/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib64/go/1.14/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadFrom(0xc005b14a80, 0xc005a3e000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib64/go/1.14/src/internal/poll/fd_unix.go:219 +0x1a3
net.(*netFD).readFrom(0xc005b14a80, 0xc005a3e000, 0x10000, 0x10000, 0xc005a1cec8, 0x17974e0, 0xc00051e5d0, 0x0, 0x0)
        /usr/lib64/go/1.14/src/net/fd_unix.go:208 +0x5b
net.(*UDPConn).readFrom(0xc0002e6a90, 0xc005a3e000, 0x10000, 0x10000, 0x0, 0xc005a1cf50, 0xc02000, 0xc005b181a8)
        /usr/lib64/go/1.14/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFrom(0xc0002e6a90, 0xc005a3e000, 0x10000, 0x10000, 0x17974e0, 0xc00051e5d0, 0x0, 0x0, 0x0)
        /usr/lib64/go/1.14/src/net/udpsock.go:118 +0x5d
github.com/whyrusleeping/mdns.(*Server).recv(0xc005b18180, 0xc0002e6a90)
        pkg/mod/github.com/whyrusleeping/mdns@v0.0.0-20190826153040-b9b60ed33aa9/server.go:116 +0x9e
created by github.com/whyrusleeping/mdns.NewServer
        pkg/mod/github.com/whyrusleeping/mdns@v0.0.0-20190826153040-b9b60ed33aa9/server.go:77 +0x15d

goroutine 819 [syscall]:
os/signal.signal_recv(0x17992e0)
        /usr/lib64/go/1.14/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        /usr/lib64/go/1.14/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.Notify.func1
        /usr/lib64/go/1.14/src/os/signal/signal.go:127 +0x44

rax    0xca
rbx    0x229b680
rcx    0x46b7a3
rdx    0x0
rdi    0x229b7c8
rsi    0x80
rbp    0x7ffc8c28b7f0
rsp    0x7ffc8c28b7a8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc000422360
r15    0x80
Version Information 
github.com/ipfs/ipfs-cluster
cloud.google.com/go v0.37.0
contrib.go.opencensus.io/exporter/jaeger v0.2.0
contrib.go.opencensus.io/exporter/prometheus v0.1.0
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9
github.com/BurntSushi/toml v0.3.1
github.com/DataDog/datadog-go v2.2.0+incompatible
github.com/Kubuxu/go-os-helper v0.0.1
github.com/OneOfOne/xxhash v1.2.2
github.com/Stebalien/go-bitfield v0.0.1
github.com/aead/siphash v1.0.1
github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af
github.com/alangpierce/go-forceexport v0.0.0-20160317203124-8f1d6941cd75
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6
github.com/armon/go-metrics v0.0.0-20190430140413-ec5e00d3c878
github.com/awalterschulze/gographviz v0.0.0-20190522210029-fa59802746ab
github.com/beorn7/perks v1.0.1
github.com/blang/semver v3.5.1+incompatible
github.com/boltdb/bolt v1.3.1
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
github.com/btcsuite/btcd v0.20.1-beta
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd
github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
github.com/btcsuite/winsvc v1.0.0
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23
github.com/cenkalti/backoff v2.2.1+incompatible
github.com/cespare/xxhash v1.1.0
github.com/cespare/xxhash/v2 v2.1.1
github.com/cheekybits/genny v1.0.0
github.com/cheekybits/is v0.0.0-20150225183255-68e9c0620927
github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible
github.com/circonus-labs/circonusllhist v0.1.3
github.com/client9/misspell v0.3.4
github.com/coreos/etcd v3.3.10+incompatible
github.com/coreos/go-etcd v2.0.0+incompatible
github.com/coreos/go-semver v0.3.0
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
github.com/cpuguy83/go-md2man v1.0.10
github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3
github.com/cskr/pubsub v1.0.2
github.com/davecgh/go-spew v1.1.1
github.com/davidlazar/go-crypto v0.0.0-20170701192655-dcfb0a7ac018
github.com/dgraph-io/badger v1.6.1
github.com/dgraph-io/ristretto v0.0.2
github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2
github.com/dustin/go-humanize v1.0.0
github.com/facebookgo/atomicfile v0.0.0-20151019160806-2de1f203e7d5
github.com/fatih/color v1.7.0
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90
github.com/francoispqt/gojay v1.2.13
github.com/fsnotify/fsnotify v1.4.7
github.com/ghodss/yaml v1.0.0
github.com/gliderlabs/ssh v0.1.1
github.com/go-check/check v0.0.0-20180628173108-788fd7840127
github.com/go-errors/errors v1.0.1
github.com/go-kit/kit v0.9.0
github.com/go-logfmt/logfmt v0.4.0
github.com/go-stack/stack v1.8.0
github.com/gogo/protobuf v1.3.1
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7
github.com/golang/mock v1.4.0
github.com/golang/protobuf v1.4.1
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c
github.com/google/go-cmp v0.4.0
github.com/google/go-github v17.0.0+incompatible
github.com/google/go-querystring v1.0.0
github.com/google/gofuzz v1.0.0
github.com/google/gopacket v1.1.17
github.com/google/martian v2.1.0+incompatible
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57
github.com/google/renameio v0.1.0
github.com/google/uuid v1.1.1
github.com/googleapis/gax-go v2.0.0+incompatible
github.com/googleapis/gax-go/v2 v2.0.3
github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c
github.com/gorilla/handlers v1.4.2
github.com/gorilla/mux v1.7.4
github.com/gorilla/websocket v1.4.2
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
github.com/grpc-ecosystem/grpc-gateway v1.5.0
github.com/gxed/hashland/keccakpg v0.0.1
github.com/gxed/hashland/murmur3 v0.0.1
github.com/hashicorp/errwrap v1.0.0
github.com/hashicorp/go-cleanhttp v0.5.0
github.com/hashicorp/go-hclog v0.13.0
github.com/hashicorp/go-immutable-radix v1.0.0
github.com/hashicorp/go-msgpack v0.5.5
github.com/hashicorp/go-multierror v1.1.0
github.com/hashicorp/go-retryablehttp v0.5.3
github.com/hashicorp/go-uuid v1.0.0
github.com/hashicorp/golang-lru v0.5.4
github.com/hashicorp/hcl v1.0.0
github.com/hashicorp/raft v1.1.1
github.com/hashicorp/raft-boltdb v0.0.0-20190605210249-ef2e128ed477
github.com/hpcloud/tail v1.0.0
github.com/hsanjuan/ipfs-lite v1.1.13
github.com/huin/goupnp v1.0.0
github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150
github.com/imdario/mergo v0.3.9
github.com/inconshreveable/mousetrap v1.0.0
github.com/ipfs/bbloom v0.0.4
github.com/ipfs/go-bitswap v0.2.14
github.com/ipfs/go-block-format v0.0.2
github.com/ipfs/go-blockservice v0.1.3
github.com/ipfs/go-cid v0.0.5
github.com/ipfs/go-cidutil v0.0.2
github.com/ipfs/go-datastore v0.4.4
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.2.4
github.com/ipfs/go-ds-crdt v0.1.12
github.com/ipfs/go-ds-leveldb v0.4.2
github.com/ipfs/go-fs-lock v0.0.4
github.com/ipfs/go-ipfs-api v0.0.3
github.com/ipfs/go-ipfs-blockstore v1.0.0
github.com/ipfs/go-ipfs-blocksutil v0.0.1
github.com/ipfs/go-ipfs-chunker v0.0.5
github.com/ipfs/go-ipfs-config v0.5.3
github.com/ipfs/go-ipfs-delay v0.0.1
github.com/ipfs/go-ipfs-ds-help v1.0.0
github.com/ipfs/go-ipfs-exchange-interface v0.0.1
github.com/ipfs/go-ipfs-exchange-offline v0.0.1
github.com/ipfs/go-ipfs-files v0.0.8
github.com/ipfs/go-ipfs-pinner v0.0.4
github.com/ipfs/go-ipfs-posinfo v0.0.1
github.com/ipfs/go-ipfs-pq v0.0.2
github.com/ipfs/go-ipfs-provider v0.4.3
github.com/ipfs/go-ipfs-routing v0.1.0
github.com/ipfs/go-ipfs-util v0.0.1
github.com/ipfs/go-ipld-cbor v0.0.4
github.com/ipfs/go-ipld-format v0.2.0
github.com/ipfs/go-ipns v0.0.2
github.com/ipfs/go-log v1.0.4
github.com/ipfs/go-log/v2 v2.0.8
github.com/ipfs/go-merkledag v0.3.2
github.com/ipfs/go-metrics-interface v0.0.1
github.com/ipfs/go-mfs v0.1.1
github.com/ipfs/go-path v0.0.7
github.com/ipfs/go-peertaskqueue v0.2.0
github.com/ipfs/go-unixfs v0.2.4
github.com/ipfs/go-verifcid v0.0.1
github.com/jackpal/gateway v1.0.5
github.com/jackpal/go-nat-pmp v1.0.2
github.com/jbenet/go-cienv v0.1.0
github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2
github.com/jbenet/goprocess v0.1.4
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
github.com/jessevdk/go-flags v1.4.0
github.com/jrick/logrotate v1.0.0
github.com/json-iterator/go v1.1.9
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
github.com/jtolds/gls v4.20.0+incompatible
github.com/julienschmidt/httprouter v1.2.0
github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5
github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
github.com/kelseyhightower/envconfig v1.4.0
github.com/kishansagathiya/go-dot v0.1.0
github.com/kisielk/errcheck v1.2.0
github.com/kisielk/gotool v1.0.0
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23
github.com/konsorten/go-windows-terminal-sequences v1.0.1
github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
github.com/kr/pretty v0.2.0
github.com/kr/pty v1.1.3
github.com/kr/text v0.1.0
github.com/lanzafame/go-libp2p-ocgorpc v0.1.1
github.com/libp2p/go-addr-util v0.0.1
github.com/libp2p/go-buffer-pool v0.0.2
github.com/libp2p/go-conn-security v0.0.1
github.com/libp2p/go-conn-security-multistream v0.2.0
github.com/libp2p/go-eventbus v0.1.0
github.com/libp2p/go-flow-metrics v0.0.3
github.com/libp2p/go-libp2p v0.8.3
github.com/libp2p/go-libp2p-autonat v0.2.2
github.com/libp2p/go-libp2p-blankhost v0.1.4
github.com/libp2p/go-libp2p-circuit v0.2.2
github.com/libp2p/go-libp2p-connmgr v0.2.1
github.com/libp2p/go-libp2p-consensus v0.0.1
github.com/libp2p/go-libp2p-core v0.5.4
github.com/libp2p/go-libp2p-crypto v0.1.0
github.com/libp2p/go-libp2p-discovery v0.4.0
github.com/libp2p/go-libp2p-gorpc v0.1.0
github.com/libp2p/go-libp2p-gostream v0.2.1
github.com/libp2p/go-libp2p-host v0.0.3
github.com/libp2p/go-libp2p-http v0.1.5
github.com/libp2p/go-libp2p-interface-connmgr v0.0.5
github.com/libp2p/go-libp2p-interface-pnet v0.0.1
github.com/libp2p/go-libp2p-kad-dht v0.7.11
github.com/libp2p/go-libp2p-kbucket v0.4.1
github.com/libp2p/go-libp2p-loggables v0.1.0
github.com/libp2p/go-libp2p-metrics v0.1.0
github.com/libp2p/go-libp2p-mplex v0.2.3
github.com/libp2p/go-libp2p-nat v0.0.6
github.com/libp2p/go-libp2p-net v0.0.2
github.com/libp2p/go-libp2p-netutil v0.1.0
github.com/libp2p/go-libp2p-peer v0.2.0
github.com/libp2p/go-libp2p-peerstore v0.2.4
github.com/libp2p/go-libp2p-pnet v0.2.0
github.com/libp2p/go-libp2p-protocol v0.1.0
github.com/libp2p/go-libp2p-pubsub v0.2.7
github.com/libp2p/go-libp2p-quic-transport v0.3.7
github.com/libp2p/go-libp2p-raft v0.1.5
github.com/libp2p/go-libp2p-record v0.1.3
github.com/libp2p/go-libp2p-routing v0.0.1
github.com/libp2p/go-libp2p-routing-helpers v0.2.3
github.com/libp2p/go-libp2p-secio v0.2.2
github.com/libp2p/go-libp2p-swarm v0.2.3
github.com/libp2p/go-libp2p-testing v0.1.1
github.com/libp2p/go-libp2p-tls v0.1.3
github.com/libp2p/go-libp2p-transport v0.0.5
github.com/libp2p/go-libp2p-transport-upgrader v0.2.0
github.com/libp2p/go-libp2p-yamux v0.2.7
github.com/libp2p/go-maddr-filter v0.0.5
github.com/libp2p/go-mplex v0.1.2
github.com/libp2p/go-msgio v0.0.4
github.com/libp2p/go-nat v0.0.5
github.com/libp2p/go-netroute v0.1.2
github.com/libp2p/go-openssl v0.0.4
github.com/libp2p/go-reuseport v0.0.1
github.com/libp2p/go-reuseport-transport v0.0.3
github.com/libp2p/go-sockaddr v0.0.2
github.com/libp2p/go-stream-muxer v0.1.0
github.com/libp2p/go-stream-muxer-multistream v0.3.0
github.com/libp2p/go-tcp-transport v0.2.0
github.com/libp2p/go-testutil v0.1.0
github.com/libp2p/go-ws-transport v0.3.1
github.com/libp2p/go-yamux v1.3.5
github.com/lucas-clemente/quic-go v0.15.7
github.com/lunixbochs/vtclean v1.0.0
github.com/magiconair/properties v1.8.0
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe
github.com/marten-seemann/qpack v0.1.0
github.com/marten-seemann/qtls v0.9.1
github.com/mattn/go-colorable v0.1.4
github.com/mattn/go-isatty v0.0.10
github.com/matttproud/golang_protobuf_extensions v1.0.1
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b
github.com/microcosm-cc/bluemonday v1.0.1
github.com/miekg/dns v1.1.28
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
github.com/minio/sha256-simd v0.1.1
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/mapstructure v1.1.2
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
github.com/modern-go/reflect2 v1.0.1
github.com/mr-tron/base58 v1.1.3
github.com/multiformats/go-base32 v0.0.3
github.com/multiformats/go-multiaddr v0.2.1
github.com/multiformats/go-multiaddr-dns v0.2.0
github.com/multiformats/go-multiaddr-fmt v0.1.0
github.com/multiformats/go-multiaddr-net v0.1.5
github.com/multiformats/go-multibase v0.0.1
github.com/multiformats/go-multicodec v0.1.6
github.com/multiformats/go-multihash v0.0.13
github.com/multiformats/go-multistream v0.1.1
github.com/multiformats/go-varint v0.0.5
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab
github.com/onsi/ginkgo v1.12.0
github.com/onsi/gomega v1.9.0
github.com/opentracing/opentracing-go v1.1.0
github.com/openzipkin/zipkin-go v0.1.1
github.com/pascaldekloe/goe v0.1.0
github.com/pelletier/go-toml v1.2.0
github.com/pkg/errors v0.9.1
github.com/pmezard/go-difflib v1.0.0
github.com/polydawn/refmt v0.0.0-20190408063855-01bf1e26dd14
github.com/prometheus/client_golang v1.6.0
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.9.1
github.com/prometheus/procfs v0.0.11
github.com/rogpeppe/go-internal v1.3.0
github.com/rs/cors v1.7.0
github.com/russross/blackfriday v1.5.2
github.com/russross/blackfriday/v2 v2.0.1
github.com/sergi/go-diff v1.0.0
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82
github.com/shurcooL/sanitized_anchor_name v1.0.0
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
github.com/sirupsen/logrus v1.4.2
github.com/smartystreets/assertions v1.0.0
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a
github.com/smola/gocompat v0.2.0
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572
github.com/spaolacci/murmur3 v1.1.0
github.com/spf13/afero v1.1.2
github.com/spf13/cast v1.3.0
github.com/spf13/cobra v0.0.5
github.com/spf13/jwalterweatherman v1.0.0
github.com/spf13/pflag v1.0.3
github.com/spf13/viper v1.3.2
github.com/src-d/envconfig v1.0.0
github.com/stretchr/objx v0.1.1
github.com/stretchr/testify v1.5.1
github.com/syndtr/goleveldb v1.0.0
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926
github.com/uber/jaeger-client-go v2.15.0+incompatible
github.com/ugorji/go v1.1.7
github.com/ugorji/go/codec v1.1.7
github.com/urfave/cli v1.22.4
github.com/urfave/cli/v2 v2.2.0
github.com/viant/assertly v0.4.8
github.com/viant/toolbox v0.24.0
github.com/wangjia184/sortedset v0.0.0-20160527075905-f5d03557ba30
github.com/warpfork/go-wish v0.0.0-20190328234359-8b3e70f8e830
github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158
github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1
github.com/whyrusleeping/go-logging v0.0.1
github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f
github.com/whyrusleeping/mafmt v1.2.8
github.com/whyrusleeping/mdns v0.0.0-20190826153040-b9b60ed33aa9
github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7
github.com/whyrusleeping/tar-utils v0.0.0-20180509141711-8c6c8ba81d5c
github.com/whyrusleeping/timecache v0.0.0-20160911033111-cfcb2f1abfee
github.com/x-cray/logrus-prefixed-formatter v0.5.2
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77
go.opencensus.io v0.22.3
go.uber.org/atomic v1.6.0
go.uber.org/goleak v1.0.0
go.uber.org/multierr v1.5.0
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go.uber.org/zap v1.14.1
go4.org v0.0.0-20200104003542-c7e774b10ea0
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d
golang.org/x/crypto v0.0.0-20200423211502-4bdfaf469ed5
golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2
golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81
golang.org/x/lint v0.0.0-20190930215403-16217165b5de
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee
golang.org/x/net v0.0.0-20190923162816-aa69164e4478
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
golang.org/x/sys v0.0.0-20200420163511-1957bb5e6d1f
golang.org/x/text v0.3.2
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c
golang.org/x/tools v0.0.0-20191216052735-49a3e744a425
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
gonum.org/v1/gonum v0.0.0-20190926113837-94b2bbd8ac13
gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0
gonum.org/v1/plot v0.0.0-20190615073203-9aa86143727f
google.golang.org/api v0.4.0
google.golang.org/appengine v1.4.0
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb
google.golang.org/grpc v1.22.0
google.golang.org/protobuf v1.22.0
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
gopkg.in/errgo.v2 v2.1.0
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/inf.v0 v0.9.1
gopkg.in/src-d/go-cli.v0 v0.0.0-20181105080154-d492247bbc0d
gopkg.in/src-d/go-log.v1 v1.0.1
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
gopkg.in/yaml.v2 v2.2.5
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
honnef.co/go/tools v0.0.1-2019.2.3
rsc.io/pdf v0.1.1
rsc.io/quote/v3 v3.1.0
rsc.io/sampler v1.3.0
sourcegraph.com/sourcegraph/go-diff v0.5.0
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
iulianpascalau commented 4 years ago

Happened to us, also, using

        github.com/libp2p/go-libp2p v0.9.2
        github.com/libp2p/go-libp2p-core v0.5.6
        github.com/libp2p/go-libp2p-kad-dht v0.8.0

Trying to mitigate this, I have switch the instructions between themselves, calling host.Close() first (afterwards cancelling contexts on kad dht and other clean-ups). It didn't solve the problem as the process hanged (not deadlocked) on host.Close().

Stebalien commented 4 years ago

This looks like a deadlock in goprocess.

Stebalien commented 4 years ago

Hm. Looks like we're trying to add a subprocess from a process teardown. We're not supposed to do that.

It really looks like we can't use goprocess for this use-case.