Concurrent writes on a newly created volume leads to file not being written

fengshao0907 / weed-fs

Automatically exported from code.google.com/p/weed-fs

0 stars 0 forks source link

Concurrent writes on a newly created volume leads to file not being written #52

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago

What steps will reproduce the problem?
1. launch an empty volume server alone on a master
2. concurrently write many files, then try to read the created files

What is the expected output? What do you see instead?

Sometimes - the frequency of the event depends on the amount of write 
concurrency -  the first file written to a volume is missing: the volume server 
is issuing a 404 NOT FOUND error.

In debug mode the volume server logs:
I1029 16:18:28 30121 weed.go:225] [volume 35 reading 0xc21000fe60]
I1029 16:18:28 30121 weed.go:225] [read bytes -1 error Not Found]
I1029 16:18:28 30121 weed.go:225] [read error: Not Found /35,289ebbe3630a]

What version of the product are you using? On what operating system?

.43 linux

Original issue reported on code.google.com by philo...@gmail.com on 29 Oct 2013 at 3:27

GoogleCodeExporter commented 9 years ago

This seems a tough bug to reproduce.

If you try to get the same file again later, will it be successful?

Original comment by chris...@gmail.com on 29 Oct 2013 at 8:25

GoogleCodeExporter commented 9 years ago

No it always returns a 404 even after restarting the volume server.

Original comment by philo...@gmail.com on 29 Oct 2013 at 8:28

GoogleCodeExporter commented 9 years ago

Is this related to #51? Maybe volume 35 is assigned to other server previously 
and this server does not really have it?

Original comment by chris...@gmail.com on 29 Oct 2013 at 8:45

GoogleCodeExporter commented 9 years ago

No I don't think it is related to #51. 

The volume 35 exists, the writing of 35,289ebbe3630a does not return any error 
but the file does not exist. Subsequent writes to the volume 35 it are working 
as expected, every files are readable. I will provide you some sample code 
tomorrow to reproduce the issue.

Original comment by philo...@gmail.com on 29 Oct 2013 at 8:53

GoogleCodeExporter commented 9 years ago

Here is some sample code to reproduce the issue: 
https://github.com/zenria/weed-fs/commit/9ea74e4a5670530dd921d8bd1eadfce761f18ee
2

Original comment by philo...@gmail.com on 31 Oct 2013 at 3:14

GoogleCodeExporter commented 9 years ago

Issue 46 has been merged into this issue.

Original comment by chris...@gmail.com on 31 Oct 2013 at 8:22

GoogleCodeExporter commented 9 years ago

Thanks a lot for the test case! It helped tremendously!

The problem was in the compact map implementation, where the CompactSection 
needs to be kept sorted. This is fixed in version 0.44.

Original comment by chris...@gmail.com on 31 Oct 2013 at 8:25

Changed state: Fixed

GoogleCodeExporter commented 9 years ago

This issue still exists in 0.45.

I1206 17:06:33 03375 weed.go:226] [volume 3 reading 0x1a5540c0]
I1206 17:06:33 03375 weed.go:226] [read bytes -1 error Not Found]
I1206 17:06:33 03375 weed.go:226] [read error: Not Found 
/3,0572a4789ab9f1_6.jpg]

Original comment by claudiu....@gmail.com on 6 Dec 2013 at 4:39

GoogleCodeExporter commented 9 years ago

Is it possible to send me the data files? Specifically 3.dat and 3.idx.

Original comment by chris...@gmail.com on 6 Dec 2013 at 5:58

Changed state: Started

GoogleCodeExporter commented 9 years ago

This seems a different issue. The file should be written around 400MB.

The expected file is not in the 3.dat file you sent me. Do you have some error 
when saving the file?

Original comment by chris...@gmail.com on 8 Dec 2013 at 11:22

GoogleCodeExporter commented 9 years ago

Chris, I've sent you an email with more details.

Original comment by claudiu....@gmail.com on 9 Dec 2013 at 5:20

GoogleCodeExporter commented 9 years ago

If the files are not correctly written, there will be errors returned. From the 
data files you sent, I did not find the files. Can you make sure that you check 
the file writing response?

Original comment by chris...@gmail.com on 18 Dec 2013 at 8:35

GoogleCodeExporter commented 9 years ago

Original comment by chris...@gmail.com on 28 Dec 2013 at 1:16

Changed state: Invalid