search

couchbaselabs / cbft

*THIS PROJECT HAS MOVED* from couchbaselabs TO: https://github.com/couchbase/cbft -- no further development will be done here on couchbaselabs/cbft
Other
27 stars 5 forks source link

segfault in leveldb cgo NewSnapshot() #49

Closed steveyen closed 9 years ago

steveyen commented 9 years ago

fatal error: unexpected signal during runtime execution [signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x4c60eda]

runtime stack: runtime.gothrow(0x47d0370, 0x2a) /usr/local/go/src/runtime/panic.go:503 +0x8e runtime.sigpanic() /usr/local/go/src/runtime/sigpanic_unix.go:14 +0x5e

goroutine 261 [syscall, locked to thread]: runtime.cgocall_errno(0x4001a00, 0xc20867b548, 0xc200000000) /usr/local/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20867b528 sp=0xc20867b500 github.com/jmhodges/levigo._Cfunc_leveldb_create_snapshot(0x58298f0, 0x0) github.com/jmhodges/levigo/_obj/_cgo_gotypes.go:152 +0x44 fp=0xc20867b548 sp=0xc20867b528 github.com/jmhodges/levigo.(_DB).NewSnapshot(0xc24f2e4698, 0xb078ef2c3) /Users/steveyen/go/src/github.com/jmhodges/levigo/db.go:299 +0x2b fp=0xc20867b570 sp=0xc20867b548 github.com/blevesearch/bleve/index/store/leveldb.newReader(0xc21f552ae0, 0x4b40120, 0x0, 0x0) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/store/leveldb/reader.go:27 +0x3e fp=0xc20867b598 sp=0xc20867b570 github.com/blevesearch/bleve/index/store/leveldb.(_Store).Reader(0xc21f552ae0, 0x0, 0x0, 0x0, 0x0) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/store/leveldb/store.go:103 +0x4c fp=0xc20867b5d8 sp=0xc20867b598 github.com/blevesearch/bleve/index/store/metrics.(_Store).Reader(0xc20a9461a0, 0x0, 0x0, 0x0, 0x0) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/store/metrics/metrics.go:139 +0x77 fp=0xc20867b658 sp=0xc20867b5d8 github.com/blevesearch/bleve/index/upside_down.(_UpsideDownCouch).Reader(0xc238496320, 0x0, 0x0, 0x0, 0x0) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/upside_down.go:648 +0x80 fp=0xc20867b708 sp=0xc20867b658 github.com/blevesearch/bleve.(_indexImpl).GetInternal(0xc232c8aee0, 0xc23482a630, 0x3, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/steveyen/go/src/github.com/blevesearch/bleve/index_impl.go:671 +0x11a fp=0xc20867b790 sp=0xc20867b708 github.com/couchbaselabs/cbft.(_BleveDestPartition).applyBatchUnlocked(0xc213e46a80, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/pindex_impl_bleve.go:765 +0x899 fp=0xc20867b9a8 sp=0xc20867b790 github.com/couchbaselabs/cbft.(_BleveDestPartition).updateSeqUnlocked(0xc213e46a80, 0x1637, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/pindex_impl_bleve.go:724 +0x222 fp=0xc20867ba30 sp=0xc20867b9a8 github.com/couchbaselabs/cbft.(_BleveDestPartition).OnDataUpdate(0xc213e46a80, 0xc20802ade8, 0x1, 0xc21332801f, 0x2e, 0x2e9d, 0x1637, 0xc21332804d, 0x2e6f, 0x2e6f, ...) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/pindex_impl_bleve.go:559 +0x1bc fp=0xc20867bb08 sp=0xc20867ba30 github.com/couchbaselabs/cbft.(_DestForwarder).OnDataUpdate(0xc235469600, 0xc20802ade8, 0x1, 0xc21332801f, 0x2e, 0x2e9d, 0x1637, 0xc21332804d, 0x2e6f, 0x2e6f, ...) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/dest_forwarder.go:48 +0x14c fp=0xc20867bba0 sp=0xc20867bb08 github.com/couchbaselabs/cbft.func·004(0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/feed_dcp.go:270 +0x15c fp=0xc20867bd00 sp=0xc20867bba0 github.com/couchbaselabs/cbft.func·027() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/misc.go:198 +0x2f fp=0xc20867bd38 sp=0xc20867bd00 github.com/rcrowley/go-metrics.(_StandardTimer).Time(0xc21f553b90, 0xc208a7f6e0) /Users/steveyen/go/src/github.com/rcrowley/go-metrics/timer.go:212 +0x44 fp=0xc20867bd78 sp=0xc20867bd38 github.com/couchbaselabs/cbft.Timer(0xc2116ae960, 0x4e15ab0, 0xc21f553b90, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/misc.go:199 +0xf0 fp=0xc20867bda8 sp=0xc20867bd78 github.com/couchbaselabs/cbft.(_DCPFeed).DataUpdate(0xc2498ddad0, 0x2, 0xc21332801f, 0x2e, 0x2e9d, 0x1637, 0xc20bb16960, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/feed_dcp.go:277 +0x1fc fp=0xc20867be08 sp=0xc20867bda8 github.com/couchbase/go-couchbase/cbdatasource.func·007() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:880 +0xfb0 fp=0xc20867bfe0 sp=0xc20867be08 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20867bfe8 sp=0xc20867bfe0 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).worker /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:930 +0x10c2

goroutine 1 [IO wait]: net.(_pollDesc).Wait(0xc2080caca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc2080caca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).accept(0xc2080cac40, 0x0, 0x4e12070, 0xc21e302d48) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(_TCPListener).AcceptTCP(0xc20802c4b0, 0x40a1d9e, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net/http.tcpKeepAliveListener.Accept(0xc20802c4b0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1976 +0x4c net/http.(_Server).Serve(0xc2081640c0, 0x4e1d8f8, 0xc20802c4b0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1728 +0x92 net/http.(_Server).ListenAndServe(0xc2081640c0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1718 +0x154 net/http.ListenAndServe(0x7fff5fbffa15, 0xe, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1808 +0xba main.main() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cmd/cbft/main.go:118 +0xb92

goroutine 17 [syscall, 1711 minutes, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 5 [chan receive]: github.com/blevesearch/bleve/index/upside_down.AnalysisWorker(0xc208032120) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:40 +0x73 created by github.com/blevesearch/bleve/index/upside_down.NewAnalysisQueue /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:32 +0x66

goroutine 6 [chan receive]: github.com/blevesearch/bleve/index/upside_down.AnalysisWorker(0xc208032120) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:40 +0x73 created by github.com/blevesearch/bleve/index/upside_down.NewAnalysisQueue /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:32 +0x66

goroutine 7 [chan receive]: github.com/blevesearch/bleve/index/upside_down.AnalysisWorker(0xc208032120) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:40 +0x73 created by github.com/blevesearch/bleve/index/upside_down.NewAnalysisQueue /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:32 +0x66

goroutine 8 [chan receive]: github.com/blevesearch/bleve/index/upside_down.AnalysisWorker(0xc208032120) /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:40 +0x73 created by github.com/blevesearch/bleve/index/upside_down.NewAnalysisQueue /Users/steveyen/go/src/github.com/blevesearch/bleve/index/upside_down/analysis_pool.go:32 +0x66

goroutine 10 [syscall, 1711 minutes]: os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1 /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 242 [select]: net/http.(_persistConn).writeLoop(0xc248f05d90) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(_Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 12 [chan receive, 1711 minutes]: main.dumpOnSignal(0xc208016fb0, 0x1, 0x1) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cmd/cbft/main.go:200 +0xa7 main.dumpOnSignalForPlatform() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cmd/cbft/dump.go:22 +0xe3 created by main.main /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cmd/cbft/main.go:80 +0x26c

goroutine 16 [chan receive, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).refreshCluster(0xc208094d80, 0xc20801cfc8) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:527 +0x9ab github.com/couchbase/go-couchbase/cbdatasource.func·001(0xc20801cf30) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:454 +0x2a github.com/couchbase/go-couchbase/cbdatasource.ExponentialBackoffLoop(0x4776470, 0x1b, 0xc208017fc8, 0x64, 0xc23fc00000, 0x4e20) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1545 +0x2e github.com/couchbase/go-couchbase/cbdatasource.func·002() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:455 +0xbd created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).Start /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:460 +0x3f0

goroutine 241 [IO wait]: net.(_pollDesc).Wait(0xc23f176bc0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc23f176bc0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc23f176b60, 0xc239809000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc234b42368) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc24f2e42d0, 0xc239809000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x4e13b28, 0xc24f2e42d0, 0xc248f05de8, 0xc239809000, 0x1000, 0x1000, 0x4595a00, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(_noteEOFReader).Read(0xc239c0f2a0, 0xc239809000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)

:125 +0xd4 bufio.(_Reader).fill(0xc22aa50a80) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).Peek(0xc22aa50a80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(_persistConn).readLoop(0xc248f05d90) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 18 [chan receive, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).refreshWorkers(0xc208094d80) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:563 +0xa1 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).Start /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:462 +0x40d goroutine 257 [IO wait]: net.(_pollDesc).Wait(0xc232c8bd40, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc232c8bd40, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc232c8bce0, 0xc20830f000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc26186b8c0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc24f2e47b8, 0xc20830f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x4e13b28, 0xc24f2e47b8, 0xc2498ddde8, 0xc20830f000, 0x1000, 0x1000, 0x4595a00, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(_noteEOFReader).Read(0xc2386f0c80, 0xc20830f000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0) :125 +0xd4 bufio.(_Reader).fill(0xc21a392d20) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).Peek(0xc21a392d20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(_persistConn).readLoop(0xc2498ddd90) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 28 [chan receive, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.func·006() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:778 +0xd1 created by github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).worker /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:789 +0xf67 goroutine 27 [select, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).worker(0xc208094d80, 0xc208166930, 0xf, 0xc208164420, 0x0) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:936 +0x157f github.com/couchbase/go-couchbase/cbdatasource.func·003(0x2) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:681 +0x44 github.com/couchbase/go-couchbase/cbdatasource.ExponentialBackoffLoop(0xc20815c5a0, 0x23, 0xc208059fb8, 0x64, 0xc23fc00000, 0x4e20) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1545 +0x2e github.com/couchbase/go-couchbase/cbdatasource.func·004() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:682 +0x185 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).workerStart /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:685 +0x270 goroutine 258 [select]: net/http.(_persistConn).writeLoop(0xc2498ddd90) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(_Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 29 [IO wait]: net.(_pollDesc).Wait(0xc208010760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc208010760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc208010700, 0xc208172c00, 0x18, 0x18, 0x0, 0x4e12070, 0xc26190f0b0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc20802c0d8, 0xc208172c00, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc io.ReadAtLeast(0x4e13b28, 0xc20802c0d8, 0xc208172c00, 0x18, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:298 +0xf1 io.ReadFull(0x4e13b28, 0xc20802c0d8, 0xc208172c00, 0x18, 0x18, 0xc200000000, 0x0, 0x0) /usr/local/go/src/io/io.go:316 +0x6d github.com/couchbase/gomemcached.(_MCRequest).Receive(0xc208164900, 0x4e13b28, 0xc20802c0d8, 0xc208172c00, 0x18, 0x18, 0xc208164900, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbase/gomemcached/mc_req.go:130 +0xee github.com/couchbase/go-couchbase/cbdatasource.func·007() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:808 +0x239 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).worker /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:930 +0x10c2 goroutine 32 [chan receive]: github.com/rcrowley/go-metrics.(*meterArbiter).tick(0x4b37a00) /Users/steveyen/go/src/github.com/rcrowley/go-metrics/meter.go:221 +0x52 created by github.com/rcrowley/go-metrics.NewMeter /Users/steveyen/go/src/github.com/rcrowley/go-metrics/meter.go:40 +0x1f7 goroutine 33 [chan send]: github.com/couchbaselabs/cbft.(_Manager).JanitorKick(0xc208098fc0, 0x4769b30, 0x16) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:45 +0x2ab github.com/couchbaselabs/cbft.(_Manager).PlannerLoop(0xc208098fc0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_planner.go:74 +0x3e8 created by github.com/couchbaselabs/cbft.(*Manager).Start /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager.go:139 +0x388 goroutine 35 [syscall]: syscall.Syscall(0xa, 0xc2357123c0, 0x0, 0x0, 0x0, 0x0, 0x3e) /usr/local/go/src/syscall/asm_darwin_amd64.s:20 +0x5 syscall.Unlink(0xc235712380, 0x3e, 0x0, 0x0) /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:1325 +0x91 os.Remove(0xc235712380, 0x3e, 0x0, 0x0) /usr/local/go/src/os/file_unix.go:266 +0x4d os.RemoveAll(0xc235712400, 0x3e, 0x0, 0x0) /usr/local/go/src/os/path.go:68 +0x5b os.RemoveAll(0xc24da9b3c0, 0x33, 0x0, 0x0) /usr/local/go/src/os/path.go:102 +0x45f os.RemoveAll(0xc224f91bf0, 0x2d, 0x0, 0x0) /usr/local/go/src/os/path.go:102 +0x45f github.com/couchbaselabs/cbft.(_PIndex).Close(0xc239bff3b0, 0xc224f91901, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/pindex.go:55 +0xb2 github.com/couchbaselabs/cbft.(_Manager).stopPIndex(0xc208098fc0, 0xc239bff3b0, 0xc215e65c01, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:390 +0x880 github.com/couchbaselabs/cbft.(_Manager).JanitorOnce(0xc208098fc0, 0xc2520545e0, 0x1e, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:136 +0x8a7 github.com/couchbaselabs/cbft.(_Manager).JanitorLoop(0xc208098fc0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:69 +0x2c6 created by github.com/couchbaselabs/cbft.(*Manager).Start /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager.go:144 +0x251 goroutine 37 [chan receive]: github.com/couchbaselabs/cbft.func·020() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager.go:152 +0xd3 created by github.com/couchbaselabs/cbft.(*Manager).Start /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager.go:155 +0x2e7 goroutine 38 [chan receive]: github.com/couchbaselabs/cbft.func·021() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager.go:159 +0xd3 created by github.com/couchbaselabs/cbft.(*Manager).Start /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager.go:162 +0x339 goroutine 39 [chan receive]: github.com/couchbaselabs/cbft.(_Manager).PlannerKick(0xc208098fc0, 0xc255320180, 0x1b) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_planner.go:44 +0x28c github.com/couchbaselabs/cbft.func·024() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_planner.go:57 +0x245 created by github.com/couchbaselabs/cbft.(_Manager).PlannerLoop /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_planner.go:59 +0xa5 goroutine 40 [chan receive]: github.com/couchbaselabs/cbft.(_Manager).JanitorKick(0xc208098fc0, 0xc2520545e0, 0x1e) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:45 +0x2ea github.com/couchbaselabs/cbft.func·022() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:58 +0x245 created by github.com/couchbaselabs/cbft.(_Manager).JanitorLoop /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_janitor.go:60 +0xa5 goroutine 55 [chan receive, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.func·006() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:778 +0xd1 created by github.com/couchbase/go-couchbase/cbdatasource.(*bucketDataSource).worker /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:789 +0xf67 goroutine 52 [chan receive, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).refreshCluster(0xc20822ed80, 0xc20802d778) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:527 +0x9ab github.com/couchbase/go-couchbase/cbdatasource.func·001(0x4362beb) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:454 +0x2a github.com/couchbase/go-couchbase/cbdatasource.ExponentialBackoffLoop(0x4776470, 0x1b, 0xc2080f9fc8, 0x64, 0x3fc00000, 0x4e20) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1545 +0x2e github.com/couchbase/go-couchbase/cbdatasource.func·002() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:455 +0xbd created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).Start /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:460 +0x3f0 goroutine 53 [chan receive, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).refreshWorkers(0xc20822ed80) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:563 +0xa1 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).Start /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:462 +0x40d goroutine 56 [IO wait, 1705 minutes]: net.(_pollDesc).Wait(0xc2080cb800, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc2080cb800, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc2080cb7a0, 0xc2083b7500, 0x18, 0x18, 0x0, 0x4e12070, 0xc20c4f0d00) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc208514458, 0xc2083b7500, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc io.ReadAtLeast(0x4e13b28, 0xc208514458, 0xc2083b7500, 0x18, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:298 +0xf1 io.ReadFull(0x4e13b28, 0xc208514458, 0xc2083b7500, 0x18, 0x18, 0x402cf39, 0x0, 0x0) /usr/local/go/src/io/io.go:316 +0x6d github.com/couchbase/gomemcached.(_MCRequest).Receive(0xc208289f20, 0x4e13b28, 0xc208514458, 0xc2083b7500, 0x18, 0x18, 0xc208289f20, 0x0, 0x0) /Users/steveyen/go/src/github.com/couchbase/gomemcached/mc_req.go:130 +0xee github.com/couchbase/go-couchbase/cbdatasource.func·007() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:808 +0x239 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).worker /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:930 +0x10c2 goroutine 54 [select, 1711 minutes]: github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).worker(0xc20822ed80, 0xc2083be500, 0xf, 0xc208289860, 0x0) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:936 +0x157f github.com/couchbase/go-couchbase/cbdatasource.func·003(0x2) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:681 +0x44 github.com/couchbase/go-couchbase/cbdatasource.ExponentialBackoffLoop(0xc2084d40c0, 0x23, 0xc2083aefb8, 0x64, 0xc23fc00000, 0x4e20) /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:1545 +0x2e github.com/couchbase/go-couchbase/cbdatasource.func·004() /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:682 +0x185 created by github.com/couchbase/go-couchbase/cbdatasource.(_bucketDataSource).workerStart /Users/steveyen/go/src/github.com/couchbase/go-couchbase/cbdatasource/cbdatasource.go:685 +0x270 goroutine 277 [IO wait]: net.(_pollDesc).Wait(0xc235358140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc235358140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc2353580e0, 0xc2218a2000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc215441978) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc20b8e9a80, 0xc2218a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(_liveSwitchReader).Read(0xc20b3642c8, 0xc2218a2000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(_LimitedReader).Read(0xc2095aaf80, 0xc2218a2000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(_Reader).fill(0xc22acb3bc0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).ReadSlice(0xc22acb3bc0, 0xc208679b0a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(_Reader).ReadLine(0xc22acb3bc0, 0x0, 0x0, 0x0, 0x4e1d100, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(_Reader).readLineSlice(0xc20875fc50, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(_Reader).ReadLine(0xc20875fc50, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc22acb3bc0, 0xc25e687ad0, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(_conn).readRequest(0xc20b364280, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(_conn).serve(0xc20b364280) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(_Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 363 [semacquire]: sync.(_Mutex).Lock(0xc213e46ac0) /usr/local/go/src/sync/mutex.go:66 +0xd3 github.com/couchbaselabs/cbft.func·038() /Users/steveyen/go/src/github.com/couchbaselabs/cbft/pindex_impl_bleve.go:340 +0xf9 created by github.com/couchbaselabs/cbft.(_BleveDest).closeUnlocked /Users/steveyen/go/src/github.com/couchbaselabs/cbft/pindex_impl_bleve.go:347 +0x15f goroutine 228 [IO wait]: net.(_pollDesc).Wait(0xc23f177b80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc23f177b80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc23f177b20, 0xc2081cf000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc2394d6f88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc24f2e48c8, 0xc2081cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(_liveSwitchReader).Read(0xc2150931c8, 0xc2081cf000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(_LimitedReader).Read(0xc239a71580, 0xc2081cf000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(_Reader).fill(0xc20891e000) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).ReadSlice(0xc20891e000, 0xc233057b0a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(_Reader).ReadLine(0xc20891e000, 0x0, 0x0, 0x0, 0x4e1d100, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(_Reader).readLineSlice(0xc241ee5800, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(_Reader).ReadLine(0xc241ee5800, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc20891e000, 0xc25e6872b0, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(_conn).readRequest(0xc215093180, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(_conn).serve(0xc215093180) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(_Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 280 [IO wait]: net.(_pollDesc).Wait(0xc235358290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc235358290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc235358230, 0xc20c019000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc261641920) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc20b8e9aa0, 0xc20c019000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(_liveSwitchReader).Read(0xc20b364548, 0xc20c019000, 0x1000, 0x1000, 0x3, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(_LimitedReader).Read(0xc2095ab0a0, 0xc20c019000, 0x1000, 0x1000, 0x1, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(_Reader).fill(0xc208631800) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).ReadSlice(0xc208631800, 0x409e00a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(_Reader).ReadLine(0xc208631800, 0x0, 0x0, 0x0, 0xc231d70800, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(_Reader).readLineSlice(0xc20875fd10, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(_Reader).ReadLine(0xc20875fd10, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc208631800, 0xc21b7391e0, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(_conn).readRequest(0xc20b364500, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(_conn).serve(0xc20b364500) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(_Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 279 [chan send]: github.com/couchbaselabs/cbft.(_Manager).PlannerKick(0xc208098fc0, 0xc2594cc5a0, 0x23) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_planner.go:44 +0x24d github.com/couchbaselabs/cbft.(_Manager).CreateIndex(0xc208098fc0, 0xc2250ff061, 0x9, 0xc2250ff040, 0x7, 0xc2250ff06b, 0x0, 0xc2250ff048, 0x0, 0xc2250fef60, ...) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/manager_api.go:105 +0xf8b github.com/couchbaselabs/cbft.(_CreateIndexHandler).ServeHTTP(0xc20802cdf8, 0xd49e788, 0xc23f6b0140, 0xc208f949c0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/rest_create_index.go:77 +0x6fb github.com/gorilla/mux.(_Router).ServeHTTP(0xc2081e8b90, 0xd49e788, 0xc23f6b0140, 0xc208f949c0) /Users/steveyen/go/src/github.com/gorilla/mux/mux.go:98 +0x297 net/http.(_ServeMux).ServeHTTP(0xc20803cc90, 0xd49e788, 0xc23f6b0140, 0xc208f949c0) /usr/local/go/src/net/http/server.go:1541 +0x17d net/http.serverHandler.ServeHTTP(0xc2081640c0, 0xd49e788, 0xc23f6b0140, 0xc208f949c0) /usr/local/go/src/net/http/server.go:1703 +0x19a net/http.(_conn).serve(0xc20b3643c0) /usr/local/go/src/net/http/server.go:1204 +0xb57 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 276 [IO wait]: net.(_pollDesc).Wait(0xc2391e4680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc2391e4680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc2391e4620, 0xc214435000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc2394d6d88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc24f682fd8, 0xc214435000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(_liveSwitchReader).Read(0xc232a267c8, 0xc214435000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(_LimitedReader).Read(0xc225d412a0, 0xc214435000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(_Reader).fill(0xc20b9843c0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).ReadSlice(0xc20b9843c0, 0xc233053b0a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(_Reader).ReadLine(0xc20b9843c0, 0x0, 0x0, 0x0, 0x4e1d100, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(_Reader).readLineSlice(0xc2192cb710, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(_Reader).ReadLine(0xc2192cb710, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc20b9843c0, 0xc25e687110, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(_conn).readRequest(0xc232a26780, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(_conn).serve(0xc232a26780) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(_Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 278 [IO wait]: net.(_pollDesc).Wait(0xc2353581b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc2353581b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).Read(0xc235358150, 0xc24e909000, 0x1000, 0x1000, 0x0, 0x4e12070, 0xc2394d6e68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(_conn).Read(0xc20b8e9a88, 0xc24e909000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(_liveSwitchReader).Read(0xc20b364368, 0xc24e909000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(_LimitedReader).Read(0xc2095aafe0, 0xc24e909000, 0x1000, 0x1000, 0x2, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(_Reader).fill(0xc22acb3f20) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(_Reader).ReadSlice(0xc22acb3f20, 0xc231ff3b0a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(_Reader).ReadLine(0xc22acb3f20, 0x0, 0x0, 0x0, 0x4e1d100, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(_Reader).readLineSlice(0xc20875fcb0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(_Reader).ReadLine(0xc20875fcb0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc22acb3f20, 0xc25e6871e0, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(_conn).readRequest(0xc20b364320, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(_conn).serve(0xc20b364320) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(_Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 358 [chan send]: github.com/couchbaselabs/cbft.func·003(0xc2081643c0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cfg_mem.go:114 +0x76 created by github.com/couchbaselabs/cbft.(*CfgMem).fireEvent /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cfg_mem.go:114 +0x1a0 goroutine 355 [chan send]: github.com/couchbaselabs/cbft.func·003(0xc2081643c0) /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cfg_mem.go:114 +0x76 created by github.com/couchbaselabs/cbft.(*CfgMem).fireEvent /Users/steveyen/go/src/github.com/couchbaselabs/cbft/cfg_mem.go:114 +0x1a0
steveyen commented 9 years ago

Possible steps to reproduce (crash only happened twice in 5 attempts)...

This was with cbft 0.0.0-718-g3158f68, go version go1.4 darwin/amd64, and with jmhodges/levigo of...

commit 2c43dde93d0e056173706534afd514fcbc1dd578 Date: Wed Oct 15 03:16:00 2014 -0700 correct example in doc.go

bleve was...

commit dbf50b7f2907b4a08ebbbe75deb6fa4a7bcbd97b Date: Thu Mar 26 16:40:18 2015 -0700 KVStore gtreap allows only 1 writer at a time

Steps...

steveyen commented 9 years ago

From chatting with Marty Schoch, thinking the manager_janitor order of operations should be...

Instead of the current order...

mschoch commented 9 years ago

Woops, didn't mean to close it, just refer to it....

steveyen commented 9 years ago

Yep, that bleve fix to check for open'ness seems to have done the trick, via manual testing repeated index definition updates and no crashes...

commit 11262c7 Author: Marty Schoch marty.schoch@gmail.com Date: Fri Apr 03 2015 15:05:24 GMT-0700 (PDT) Subject: fix bug, internal ops must check that index is open possibly fixes https://github.com/couchbaselabs/cbft/issues/49

Gonna close this one then, and create a new issue for the related discovery of janitor order of operations to not forget about that.