crash trying to dedupe 1T worth data

kiatoa commented 9 years ago

I'm transferring some data from MooseFS to btrfs on Ubuntu 15.04 x86_64. Some of the data were snapshots with some changes. So to get back to the original efficiency of approximately 50% I was trying to run duperemove on about 1T of the data and got this error:

I don't need a fix - I'm just submitting this issue in the hopes it is helpful.

[0x3ad0990] Dedupe 29 extents with target: (0, 131072), "safe/snapshots/stuff/somefile.tga" FAILURE: Dedupe ioctl returns 9: Bad file descriptor ERROR: dedupe.c:306 [stack trace follows] duperemove(print_stack_trace+0x43) [0x409d27] duperemove(pop_one_dedupe_result+0x57) [0x408120] duperemove() [0x40c3c8] duperemove() [0x40cb93] duperemove() [0x40cd79] /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x712e8) [0x7f2f9d0c02e8] /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x70955) [0x7f2f9d0bf955] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76aa) [0x7f2f9c2376aa] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2f9c7c1eed] Aborted (core dumped)

markfasheh commented 9 years ago

quick question - what version of duperemove were you using?

kiatoa commented 9 years ago

$ duperemove duperemove v0.10.beta1

Thanks.

On Tue, Sep 1, 2015 at 12:59 PM, markfasheh notifications@github.com wrote:

quick question - what version of duperemove were you using?

— Reply to this email directly or view it on GitHub https://github.com/markfasheh/duperemove/issues/93#issuecomment-136842175 .

Matt -=-

The moment we want to believe something, we suddenly see all the arguments for it, and become blind to the arguments against it.

George Bernard Shaw

jimav commented 8 years ago

I get exaclty the same crash, but in my case no snapshots were ever taken. Also, FWIW, the fs was on an SSD device and the files were quite large, 10-50 GB (VM images). The command used was:

duperemove \
  --hashfile /file/on/another/filesystem \
  --lookup-extents=yes \
  --io-threads=3 \
  -r -A -h \
  -d \
  /mountpoint

markfasheh commented 8 years ago

Ok this looks like it could be a corruption of the dedupe lists duperemove makes during that phase. I thought I had found all instances but if we're still hitting it I'll have to take a closer look.

ivanbaldo commented 7 years ago

Has anyone encountered this problem in 2017?

markfasheh / duperemove

crash trying to dedupe 1T worth data #93