Failed to write to replicas for volume 3

[GoogleCodeExporter]

What steps will reproduce the problem?
1. ./filestore-upload-test -weed=:9333 -parallel.write=5 -parallel.read=2 
-request.size.init=500000 2>&1 | grep ER

What is the expected output? What do you see instead?
Instead of empty (no error) I see:
2012/11/11 21:47:05 ERROR: error uploading: error POSTing to 
http://localhost:8080/3,01b47ff6a24911: errorcode=500 message={"error":"Failed 
to write to replicas for volume 3","size":0}

What version of the product are you using? On what operating system?
0.23 (01f2fcf72daac3e13cd9a664aaae125f4202e603)

Please provide any additional information below.
master:
gthomas@waterhouse:~/projects/go/src/github.com/tgulacsi/filestore-upload-test$ 
mkdir -p /tmp/weed ; ./weed volume -dir=/tmp/weed
2012/11/11 21:43:48 Volume Folder permission: -rwxr-xr-x
2012/11/11 21:43:48 Loading index file 1.idx size 64432
2012/11/11 21:43:48 In dir /tmp/weed reads volume =  1 , replicationType = 000
2012/11/11 21:43:48 Loading index file 2.idx size 62592
2012/11/11 21:43:48 In dir /tmp/weed reads volume =  2 , replicationType = 000
2012/11/11 21:43:48 Loading index file 3.idx size 64384
2012/11/11 21:43:48 In dir /tmp/weed reads volume =  3 , replicationType = 000
2012/11/11 21:43:48 Loading index file 4.idx size 63376
2012/11/11 21:43:48 In dir /tmp/weed reads volume =  4 , replicationType = 000
2012/11/11 21:43:48 Loading index file 5.idx size 63824
2012/11/11 21:43:48 In dir /tmp/weed reads volume =  5 , replicationType = 000
2012/11/11 21:43:48 Store started on dir: /tmp/weed with 5 volumes
2012/11/11 21:43:48 store joined at localhost:9333
2012/11/11 21:43:48 Start Weed volume server 0.23 at http://localhost:8080
2012/11/11 21:47:05 post to http://localhost:9333/dir/lookup EOF
2012/11/11 21:47:05 Failed to lookup for 3 EOF

Original issue reported on code.google.com by tgulacs...@gmail.com on 11 Nov 2012 at 8:53

[GoogleCodeExporter]

Seems the unnecessary master lookup for replication for 1-copy data caused this 
issue.

Fixed in the git head. Going to release a new version soon.

Original comment by chris...@gmail.com on 12 Nov 2012 at 9:31

• Changed state: Accepted

[GoogleCodeExporter]

with ecd0399f8d3e60b6f065258b92901e5b06faffe9:

tgulacsi@nell:~/projects/weed-fs$ mkdir -p /tmp/weed ; ./weed master 
-mdir=/tmp/weed & ./weed volume -dir=/tmp/weed
[1] 15212
2012/11/12 10:51:05 Volume Folder permission: -rwxr-xr-x
2012/11/12 10:51:05 In dir /tmp/weed reads volume =  1 , replicationType = 000
2012/11/12 10:51:05 In dir /tmp/weed reads volume =  2 , replicationType = 000
2012/11/12 10:51:05 In dir /tmp/weed reads volume =  3 , replicationType = 000
2012/11/12 10:51:05 In dir /tmp/weed reads volume =  4 , replicationType = 000
2012/11/12 10:51:05 In dir /tmp/weed reads volume =  5 , replicationType = 000
2012/11/12 10:51:05 Store started on dir: /tmp/weed with 5 volumes
2012/11/12 10:51:05 store joined at localhost:9333
2012/11/12 10:51:05 Start Weed volume server 0.24 at http://localhost:8080
2012/11/12 10:51:05 Loading file id sequence 20000 => 30000
2012/11/12 10:51:05 Volume Size Limit is 32768 MB
2012/11/12 10:51:05 Start Weed Master 0.24 at port 9333
topo adds child DefaultDataCenter
topo:DefaultDataCenter adds child DefaultRack
topo:DefaultDataCenter:DefaultRack adds child localhost:8080
2012/11/12 10:51:28 Saving file id sequence 40000 to /tmp/weed/weed.seq

Another window:

./filestore-upload-test -weed=:9333 -parallel.write=5 -parallel.read=2 
-request.size.init=500000 2>&1
2012/11/12 10:52:49 starting round 1...
2012/11/12 10:52:49 calling uploadRound
2012/11/12 10:52:49 POST http://localhost:8080/5,7533b405235e response: 
{"error":"Failed to write to local disk","size":500000}
2012/11/12 10:52:49 CU err=error POSTing to 
http://localhost:8080/5,7533b405235e: errorcode=500 message={"error":"Failed to 
write to local disk","size":500000}
2012/11/12 10:52:49 uploadRound: error uploading: error POSTing to 
http://localhost:8080/5,7533b405235e: errorcode=500 message={"error":"Failed to 
write to local disk","size":500000}
2012/11/12 10:52:49 error with round 1: error uploading: error POSTing to 
http://localhost:8080/5,7533b405235e: errorcode=500 message={"error":"Failed to 
write to local disk","size":500000}

:(

Original comment by tgulacs...@gmail.com on 12 Nov 2012 at 9:54

[GoogleCodeExporter]

Thanks. It is displaying the wrong error message. Now it is corrected in git 
head.

Original comment by chris...@gmail.com on 12 Nov 2012 at 6:52

• Changed state: Fixed

[GoogleCodeExporter]

gthomas@waterhouse:~/projects/go/src/github.com/tgulacsi/filestore-upload-test$ 
./filestore-upload-test -request.size.init=512000 -request.num=1000 
-parallel.read=2 -parallel.write=5 -weed=:9333 2>&1|grep ER
2012/11/12 21:45:27 ERROR: error uploading: error POSTing to 
http://localhost:8080/2,2c9dee3ecb3f: error POSTing &{Method:POST 
URL:http://localhost:8080/2,2c9dee3ecb3f Proto:HTTP/1.1 ProtoMajor:1 
ProtoMinor:1 Header:map[Accept-Encoding:[ident] 
Content-Type:[multipart/form-data; 
boundary=b362c614208f8bae73efe7bae72472089dcdce160ccb2a15ca85dbbc7cf4] 
Mime-Version:[1.0]] Body:{Reader:0xf8414f0f00} ContentLength:512243 
TransferEncoding:[] Close:false Host:localhost:8080 Form:map[] PostForm:map[] 
MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil>}: lookup 
localhost: no such host

what is this lookup localhost ?

Original comment by tgulacs...@gmail.com on 12 Nov 2012 at 8:46

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I can not pin-point this error. I got different errors when trying to 
reproduce, but they all seem before hitting the weedfs volume server.

For your particular error, the error is to lookup "localhost" before visiting 
"http://localhost:8080/2,2c9dee3ecb3f". Inside WeedFS, there are no http 
calls(for this setup). So this "localhost" lookup can only come from the client 
itself.

Original comment by chris...@gmail.com on 13 Nov 2012 at 5:29

[GoogleCodeExporter]

I just cannot make this new version work:

tgulacsi@nell:~/projects/weed-fs$ rm -rf /tmp/weed ; mkdir -p
/tmp/weed/master ; ./we master -mdir=/tmp/weed/master -debug=true &
./weed volume -dir=/tmp/weed -max=1 -debug=true
[1] 27149
2012/11/13 15:41:31 Volume Folder permission: -rwxr-xr-x
2012/11/13 15:41:31 Store started on dir: /tmp/weed with 0 volumes
2012/11/13 15:41:31 store joined at localhost:9333
2012/11/13 15:41:31 Start Weed volume server 0.24 at http://localhost:8080
2012/11/13 15:41:31 post to http://localhost:9333/dir/join dial tcp
127.0.0.1:9333: connection refused
2012/11/13 15:41:31 Setting file id sequence 10000
2012/11/13 15:41:31 Volume Size Limit is 32768 MB
2012/11/13 15:41:31 Start Weed Master 0.24 at port 9333
topo adds child DefaultDataCenter
topo:DefaultDataCenter adds child DefaultRack
topo:DefaultDataCenter:DefaultRack adds child localhost:8080

2012/11/13 15:41:45 In dir /tmp/weed adds volume = 1 , replicationType = 000
[assign volume = 1 , replicationType = 000 , error = <nil>]
Created Volume 1 on topo:DefaultDataCenter:DefaultRack:localhost:8080
2012/11/13 15:41:45 Saving file id sequence 20000 to /tmp/weed/master/weed.seq
[assign volume = 1 , replicationType = 000 , error = Volume Id 1
already exists!]
Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080]
[assign volume = 1 , replicationType = 000 , error = Volume Id 1
already exists!]
Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080]
[assign volume = 1 , replicationType = 000 , error = Volume Id 1
already exists!]
Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080]
[assign volume = 1 , replicationType = 000 , error = Volume Id 1
already exists!]
Failed to assign 1 to [topo:DefaultDataCenter:DefaultRack:localhost:8080]

By the way: how do you compile weed-fs? I needed to modify every
import of "pkg/ to "../../pkg/

GThomas

Original comment by tgulacs...@gmail.com on 13 Nov 2012 at 2:45

[GoogleCodeExporter]

Hi, GThomas, I can not reproduce your issue. But I added some more locking in 
order to avoid similar issue.

To compile:
1) set the GOPATH to the root of weed-fs
2) go to $GOPATH/src/cmd/weed/, type in "go build -o ../../../bin/weed"

Original comment by chris...@gmail.com on 13 Nov 2012 at 8:21

[GoogleCodeExporter]

Thanks, that worked.

Still see some upload error when concurrency is greater than 2 - I'll
have to investigate a little more - how do I enable copious logging?

Another question, which I couldn't answer by reading the Beaver paper:
do Haystack/Weed-FS store the uploaded file's mime type?
I think its essential to be able to serve the file (usually not photo
:)) back to browsers unaltered. Would this need to be added to the
needle structure? Would you accept such modification?

Original comment by tgulacs...@gmail.com on 14 Nov 2012 at 6:03

[GoogleCodeExporter]

When starting volume server, there is an option readTimeout. The default value 
is 1 second, which may be too low for the test.

I tried setting it to 3 seconds, and got no error now.

Original comment by chris...@gmail.com on 14 Nov 2012 at 6:45

• Changed title: Failed to write to replicas for volume 3

[GoogleCodeExporter]

With readTimeout=3, I have no errors.
Thank you!

Original comment by tgulacs...@gmail.com on 14 Nov 2012 at 8:17