search

couchbaselabs / cbfs

Distributed Blobstore using Couchbase Server
http://dustin.github.com/2012/09/27/cbfs.html
Apache License 2.0
188 stars 40 forks source link

CBFS-Couchbase-XDCR Replication issue #142

Closed chengchengmu closed 9 years ago

chengchengmu commented 9 years ago

Hello,

I have an architecture of 2 clusters (name them A and B) of 1 node in each. Each node has 1 couchbase node and 1 cbfs daemon. And there is one unidirectional XDCR replication link from cluster A to B.

First I upload foo.txt on cluster A's cbfs. On cluster B I see foo.txt with 'cbfsclient ls' command. However I can't retrieve this file, neither looking for it in the cbfs root folder or by trying to download it with 'cbfsclient download' command from cluster A.

Any suggestions ?

Is cbfs supposed to automatically replicate files uploaded on cluster A when there is a XDCR link from A to B ?

Best regards, CM

mschoch commented 9 years ago

Unfortunately CBFS does not work with XDCR. There are really 2 separate issues here:

  1. XDCR imposes many limitations on the way you can use your Couchbase cluster. CBFS performs operations on the cluster that cannot be safely used when XDCR is also in use.
  2. Even if the previous point wasn't true, it still wouldn't do what you want. CBFS stores data on disk OUTSIDE of Couchbase, so just replicating two clusters using XDCR would not move actual file data from one CBFS cluster to another, it would just move the metadata.
chengchengmu commented 9 years ago

Thank you for your answer. This is unfortunate. Just moving metadata is not enough for my use.

Anyways I still have a issue with the 'cbfsclient download' command. For example I am on cluster B, I want to download a file foo.txt located at /tmp/localdata on cbfs daemon A. I receive this error : Error getting blobs: HTTP Error 404 Not Found - Error opening blob: open /tmp/localdata/e6/e6e8ea7465f12e4d3b5a067a4c4dc698436b3478: no such file or directory But I can download a file on localhost. In summary it seems 'cbfsclient download' command works only on local.

Is this the wanted behaviour ?

Thank you !

mschoch commented 9 years ago

The configuration you are running is not supported. CBFS uses Couchbase as its discovery mechanism, so in general a node in cluster A won't even know about a node in cluster B. On the surface it looks like XDCR would fix this, but obviously because of your error, its not.

cbfsclient download does work remotely, but your configuration is not one that is supported.

Also, you should be careful using /tmp/... as your data directory. This is the PERMANENT data storage for that node. Many operating systems automatically cleanup files in /tmp, and that would cause DATA LOSS for that cbfs node.

chengchengmu commented 9 years ago

Thank you for your fast answer!

Can you please give me a case where cbfsclient download works remotely ?

I don't really get it. On the same local area network, even in the same cluster, 'cbfsclient ls' works, but 'cbfs download' does not.

Best regards, CM

mschoch commented 9 years ago

Nodes 1 and 2 in the SAME cbfs cluster, talking to the SAME couchbase cluster.

cbfsdownload file.txt

This will work from Node 1, Node 2, or anywhere else with IP connectivity to those nodes, regardless of whether or not file.txt is one Node 1 or Node 2.

For example, for me the following command works from my laptop:

cbfsclient http://cbfs.hq.couchbase.com:8484 download mschoch/upgrade.txt

This command will not work for you because the network is private. But I can assure you that it is remote from my laptop.

chengchengmu commented 9 years ago

Thank you.

You are right. By putting cbfs nodes on the same couchbase cluster, download works. Moreover cbfs nodes are automatically synchronized. Wonderful. But (there is always a but), cbfs node 1 can upload files, cbfs node 2 cannot. Maybe it's an undeterminist issue. I will work on it tomorrow.

Anyways thank you very much for your help !