search

opencurve / curve

Curve is a sandbox project hosted by the CNCF Foundation. It's cloud-native, high-performance, and easy to operate. Curve is an open-source distributed storage system for block and shared file storage.
https://opencurve.io
Apache License 2.0
2.32k stars 521 forks source link

potential inconsistent data when map a clone volume by curve-nbd with 512 block-size #985

Closed wu-hanqing closed 1 year ago

wu-hanqing commented 2 years ago

Describe the bug (描述bug)

for a clone volume, each chunk in ChunkServer has a bitmap, and each bit represents whether a 4k range has been written after clone, which means clone chunk can only receive 4k aligned write requests.

in this case, when map a clone volume by curve-nbd with 512 block-size, and user issues a 512 bytes write request, curve-client firstly reads the full 4k range data, and then paste 512 bytes into full 4k data, and send a 4k write request to chunkserver.

so, when user issues two non-overlapping 512 bytes within the same 4k range concurrently, one of the two requests' data may lost.

To Reproduce (复现方法)

I haven't reproduce this problem, but in theory it's reproducible, later I will construct scenarios to verify this problem.

aspirer commented 2 years ago

may be add some sleep() in the IO flow can reproduce this issue. such as: if (offset=0) { ## the first 512 bytes request sleep 5s; } the send another 512 bytes request during the sleeping 5s but offset=1kb, and the second request will be overwritten the first one.

wu-hanqing commented 2 years ago

this code can reproduce current problem https://github.com/wu-hanqing/curve/tree/issue-985

here is the output image image