search

laktak / chkbit

Check your files for data corruption
MIT License
112 stars 7 forks source link

ECC algorithm feature request #17

Closed BarsMonster closed 2 weeks ago

BarsMonster commented 4 months ago

Looking at examples of files damaged in storage, it looks like typically there is less than 512 bytes of wrong data consecutively. Detecting that error has happened is good, but it is very sad if one cannot do anything about it. One way to be able to deal with it is to add ECC correction algorithm in addition to checksums.

It looks like it would be needed to add around ~1.2-1.5kB of ECC data to reliably correct any 512bytes of consecutive errors.

laktak commented 3 months ago

Thanks for the idea. This sounds very similar to #4 though I guess ecc would be a lot smaller than par2.

https://github.com/laktak/chkbit-py/issues/4#issuecomment-1272607752

dia3olik commented 2 months ago

@laktak

Yeah, I still think it would be a must have since it's not always practical/possible to recover from a backup.

As soon as a problem is detected, the user could have the option to immediately repair the bit rotted data, this would prevent to corrupt the backup(s) (or at least less versions as possible of it). In fact it can happen that the broken data can propagate to the backups and you could discover the problem just after the last good backup has been deleted by the rotation schedule...

I think this feature would be ESSENTIAL at least in the most basic form of bit-rot protection (using ECC, par2turbo, reed-solomon or others).

chkbit could become a simpler and lightweight version of SnapRaid, and file based instead of disk based.

I'd love to help but I don't have dev skills sadly!

laktak commented 2 weeks ago

I'll close this as a duplicate.