search

markfasheh / duperemove

Tools for deduping file systems
GNU General Public License v2.0
689 stars 75 forks source link

[RFC] Slow IO devices, hanging and crashes #319

Open auroraanon38 opened 8 months ago

auroraanon38 commented 8 months ago

This is most likely not the right place to mention this, but I didn't manage to find the mailing list.

I've found that mounting the filesystem to deduplicate with the sync (synchronous IO) option can help prevent hanging and kernel timeouts which I've seen mentioned in none of the other issues about hanging.

Has the thrashing aspect of async IO been looked at as a factor so far?

the8472 commented 8 months ago

Something that has helped me was schedtool -D ionice -c 3 duperemove ... which puts duperemove under CPU and IO idle priority policies.

If you're deduping HDDs then limiting it to 1 IO thread might also help because parallelism tends to make access more random and HDDs prefer more sequential IO patterns.

Another option is to use cgroups to set io.max or cpu.max policies

JackSlateur commented 7 months ago

Hello @auroraanon38 Your issue is quite strange to me

Could you reproduce the issue with the latest code, without using the sync mount option ? If you do reproduce, could you give me your kernel version and the btrfs-tools version used to create the filesystem ?

Thank you