search

couchbaselabs / cbft

*THIS PROJECT HAS MOVED* from couchbaselabs TO: https://github.com/couchbase/cbft -- no further development will be done here on couchbaselabs/cbft
Other
27 stars 5 forks source link

handle bucket flush #28

Open steveyen opened 9 years ago

steveyen commented 9 years ago

Saw this hint from Aliaksey K (alk) from some related email thread...

"Any client ... can and should (as far as I'm aware) detect flush via vbucket uuid change. I.e. for well behaved DCP client it means it will work automatically. Because flush is same as data-loss-full failover followed by rebalance (i.e. when some or all vbuckets are lost during failover and has to be re-created during rebalance)."

Not sure if cbft client does that right, so this issue is a reminder to double-check.

steveyen commented 9 years ago

flushing a bucket seems to bring cbft to a state where it closes the index, but cbft doesn't seem to reopen it.

steveyen commented 8 years ago

Following up on Feb 13, 2015 comment, this is reconfirmed even in latest cbft/cbgt circa Dec 2015.

After a FLUSH, a cbft node can get into a looping state of...

2015/12/10 15:20:06 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: sendStreamReq, err: bleve: BleveDest already closed 2015/12/10 15:20:06 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: pkt.Receive, err: read tcp 127.0.0.1:11210: use of closed network connection 2015/12/10 15:20:26 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: sendStreamReq, err: bleve: BleveDest already closed 2015/12/10 15:20:26 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: pkt.Receive, err: read tcp 127.0.0.1:11210: use of closed network connection 2015/12/10 15:20:46 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: sendStreamReq, err: bleve: BleveDest already closed 2015/12/10 15:20:46 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: pkt.Receive, err: read tcp 127.0.0.1:11210: use of closed network connection 2015/12/10 15:21:06 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: sendStreamReq, err: bleve: BleveDest already closed 2015/12/10 15:21:06 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: pkt.Receive, err: read tcp 127.0.0.1:11210: use of closed network connection 2015/12/10 15:21:26 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: sendStreamReq, err: bleve: BleveDest already closed 2015/12/10 15:21:26 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: pkt.Receive, err: read tcp 127.0.0.1:11210: use of closed network connection 2015/12/10 15:21:46 feed_dcp: OnError, name: bs_be8575242aa17215: bucketName: beer-sample, bucketUUID: , err: sendStreamReq, err: bleve: BleveDest already closed

The latest theory is that the feed_dcp (and it's datasource) is just holding onto its old receiver; or the feed_dcp instance isn't getting stopped / shutdown and restarted with fresh, brand new pindex instance, but perhaps is just holding onto the old (unfortunately, already closed) pindex.

steveyen commented 8 years ago

Proposed fix in cbdatasource here: http://review.couchbase.org/57740