Wrong volume assignment in distributed master setup (when raft master election) (v0.61 amd64)

[GoogleCodeExporter]

I have five hosts to form weed-fs cluster.

At first I've tried using "server" command to start both master-server and 
volume-server. After encountering problems I've switched to separate master, 
volume processes.

Problem manifests itself as following.

When starting three or more volume servers, I've noticed that "dir/status" call 
lists only two of them. When I restart any volume server it doesn't join 
master. Number of volume servers in the list remains the same. What happens is 
that publicIp field and a name of one of servers changes, but ip field remains 
the same.

It seems to me that weed-fs doesn't correctly handle proxying. Master server 
considers two volume servers with the same ip-address as duplicates. When 
follower-master forwards request from volume server to leader-master, 
leader-master sees follower-master as a request originator and uses it's 
ip-address instead of volume server's ip-address. As a result of wrong 
ip-address assignment some volume servers are unable to join master, they are 
considered as duplicated.

My guess is that http.RemoteAddr field in go standard library doesn't support 
X-Forwarded-For header by default. Support for X-Forwarded-For should be rolled 
manually to fix this issue (see 
http://osdir.com/ml/go-language-discuss/2012-10/msg02165.html and 
https://groups.google.com/forum/#!msg/golang-nuts/lomWKs0kOfE/gwmoD3cTxesJ)

Original issue reported on code.google.com by asvirasp...@gmail.com on 28 Jul 2014 at 12:28

[GoogleCodeExporter]

Thanks for the details! I would need to see your network topology diagram to 
understand this case. The dir/status would also be helpful.

In topology/configuration.go file, line 49, the Locate(ip, datacenter, rack) 
function assumes IP address is unique. But it seems your case has multiple 
volume servers on the same IP address?

Original comment by chris...@gmail.com on 28 Jul 2014 at 2:58

[GoogleCodeExporter]

No each volume server has distinct IP-address.

The problem is that IP-address is erroneously detected.

It goes like this:

When master-servers are started on all three servers everything is ok: all 
master-servers join the same cluster and leader is elected:

When I start volume server

 * it connects not to leader master-server but to random master-server from cluster.
 * chosen master-server forwards join-request from volume server up to leader master-server.
 * leader master server confuses ip-address of volume server. It get ip-address of join-request originator as volume server ip-address, but it's not. It's an ip-address of some randomly chosen master server from cluster.
 * when two volume servers chooses same master server they pushes each other from cluster because leader master server considers that they have same ip-address.

Original comment by asvirasp...@gmail.com on 29 Jul 2014 at 8:25

[GoogleCodeExporter]

I see. How did you start the volume server? Did you also specify -ip="..." ?

Original comment by chris...@gmail.com on 29 Jul 2014 at 8:57

[GoogleCodeExporter]

 * indy

    - weed -v 10 -log_dir=/opt/weedfs/log/volume volume -dir=./storage -port=8083 -mserver=indy:9333 -publicIp=indy

    - weed -v 10 -log_dir=/opt/weedfs/log/master master -mdir=./storage -port=9333 -publicIp=indy -defaultReplication=001 -peers=austin:9333,indy:9333,marina:9333,nasca:9333,valencia:9333

 * valencia

    - weed -v 10 -log_dir=/opt/weedfs/log/volume volume -dir=./storage -port=8083 -mserver=valencia:9333 -publicIp=valencia

    - weed -v 10 -log_dir=/opt/weedfs/log/master master -mdir=./storage -port=9333 -publicIp=valencia -defaultReplication=001 -peers=austin:9333,indy:9333,marina:9333,nasca:9333,valencia:9333

 * nasca

    - weed -v 10 -log_dir=/opt/weedfs/log/volume volume -dir=./storage -port=8083 -mserver=nasca:9333 -publicIp=nasca

    - weed -v 10 -log_dir=/opt/weedfs/log/master master -mdir=./storage -port=9333 -publicIp=nasca -defaultReplication=001 -peers=austin:9333,indy:9333,marina:9333,nasca:9333,valencia:9333

Original comment by asvirasp...@gmail.com on 29 Jul 2014 at 9:43

[GoogleCodeExporter]

I've just tried --ip option and it works fine with explicit ip.

Original comment by asvirasp...@gmail.com on 29 Jul 2014 at 10:14

[GoogleCodeExporter]

When --ip option is specified for volume server it listens to connections on 
this specific ip. It would be more convinient if it listens on 0.0.0.0 ip for 
my use case. 

My use case is a case of network misadministration. Hosts has several 
ip-addresses, hosts are referenced by names from /etc/hosts file and there is a 
possibility that single host is referenced with two different ip addresses 
(/etc/hosts files are out of sync).

Original comment by asvirasp...@gmail.com on 29 Jul 2014 at 10:38

[GoogleCodeExporter]

It may be better to have three options:

--listen-on-ip (ip address for socket to listen on)
--internal-ip (ip address to pass out in weed-fs messages to master)
--external-ip (public ip address to use when downloading files through web, 
current --publicIp)

Original comment by asvirasp...@gmail.com on 29 Jul 2014 at 10:49

[GoogleCodeExporter]

Original comment by chris...@gmail.com on 25 Aug 2014 at 7:06

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I'd like to notice that there is still a bug. Weed-FS doesn't work without --ip 
flag in distributed master setup. Remote address detection should be fixed by 
adding support for X-Forwarded-For or --ip flag should be made mandatory.

Original comment by asvirasp...@gmail.com on 26 Aug 2014 at 5:58

[GoogleCodeExporter]

Original comment by chris...@gmail.com on 26 Aug 2014 at 5:29

• Added labels: Type-Defect
• Removed labels: Type-Enhancement

[GoogleCodeExporter]

Added "-ip.bind" option to volume servers.

Original comment by chris...@gmail.com on 21 Sep 2014 at 6:50

• Changed state: Fixed