search

markfasheh / duperemove

Tools for deduping file systems
GNU General Public License v2.0
816 stars 81 forks source link

Crush on 4k blocks #64

Closed nefelim4ag closed 9 years ago

nefelim4ag commented 9 years ago

FAILURE: Dedupe ioctl returns 9: Bad file descriptor ERROR: dedupe.c:306 /home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc2/maps/c8m3_sewers.bsp start block: 5114 (20946944) /home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc3/maps/c13m4_cutthroatcreek.bsp start block: 2827 (11579392) [0x16aa800] Dedupe 1 extents with target: (20.0M, 4.0K), "/home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc2/maps/c8m3_sewers.bsp" add ioctl request /home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc3/maps/c13m4_cutthroatcreek.bsp, off: 11579392, dest: 1 /home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc3/maps/c10m3_ranchhouse.bsp start block: 6742 (27615232) /home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc3/maps/c13m4_cutthroatcreek.bsp start block: 3950 (16179200) [0x16aa800] Dedupe 1 extents with target: (26.3M, 4.0K), "/home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc3/maps/c10m3_ranchhouse.bsp" add ioctl request /home/nefelim4ag/.local/share/Steam/SteamApps/common/Left 4 Dead 2/left4dead2_dlc3/maps/c13m4_cutthroatcreek.bsp, off: 16179200, dest: 1 [stack trace follows] duperemove(print_stack_trace+0x2b) [0x407f3b] duperemove(pop_one_dedupe_result+0xf0) [0x406aa0] duperemove() [0x409ea3] /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x712e8) [0x7f56c1b7c2e8] /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x70955) [0x7f56c1b7b955] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76aa) [0x7f56c0cee6aa] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f56c1278eed]

I've try to dedup data, with 128k block, all works fine, but duperemove crushed for 32k-4k blocks. Say me, if i can help debug this. i've use master branch.

PS: Ubuntu 15.04 x64

markfasheh commented 9 years ago

That looks like this line in pop_one_dedupe_result():

https://github.com/markfasheh/duperemove/blob/master/dedupe.c#L306

So there's two things here, one is that we're aborting when we shouldn't and the 2nd is that you are getting EBADF from the dedupe ioctl. Presumably you aren't getting EBADF on the same files for 128k blocks.

Can you grab the 'issue#64' branch of duperemove.git and try that out?

https://github.com/markfasheh/duperemove/tree/issue%2364

It has two commits, one should prevent the program from aborting when it gets a dedupe failure. The other is printing some info on the dedupe ioctl every time it is run - I'd like to see what your failure looks like with the debugging on.

So basically we want to figure out why we're getting a bad file descriptor failure there, if you maybe put some debug prints inside dedupe_extent_list() (run_dedupe.c) to figure this out that would help.

nefelim4ag commented 9 years ago

Thanks @markfasheh, now i get add ioctl request /home/nefelim4ag/.local/share/Steam/SteamApps/common/Robocraft/Robocraft_Data/sharedassets8.assets, off: 46759936, dest: 110 FAILURE: Dedupe ioctl returns 9: Bad file descriptor Kernel processed data (excludes target files): 493.6M Comparison of extent info shows a net change in shared extents of: 10.7M

But duperemove not crush and this is cool.

markfasheh commented 9 years ago

Hey can you paste some of that debug info I put in the branch for you? I'd like to see why we're getting EBADF still.

markfasheh commented 9 years ago

Btw, this is all fixed in master as of now. If you hit any more problems let me know.