Got error: Source and backup don't match

wamdam / backy2

backy2: Deduplicating block based backup software for ceph/rbd, image files and devices

http://backy2.com/

Other

196 stars 39 forks source link

Got error: Source and backup don't match #70

Closed gschoenberger closed 4 years ago

gschoenberger commented 4 years ago

Today I got this error while backing up one of our Ceph images:

   ERROR: [backy2.logging] Source and backup don't match in regions outside of the hints.
   ERROR: [backy2.logging] Looks like the hints don't match or the source is different.
   ERROR: [backy2.logging] Found wrong source data at block 1592: offset 6677331968 with max. length 4194304

Any more hints or descriptions why such errors can occur? THX, Georg

wamdam commented 4 years ago

Nice that this hit you. Actually that code never reached me. This is a sanity check. Let me explain: Theres this old backup and you want to create a new backup. The new one is based on the old one (-f ) AND you provide a hintfile (-r ). Then backy will chose 10 random blocks OUTSIDE of your hintfile and compare if they match to what it expects (i.e. that they are the same as in the old backup because according to the they shouldn't have changed).

So the error means, you're either basing on the wrong old backup or you have the wrong hint-file. Backy2 will deny a backup then, as this will not result in usable restorable data.

gschoenberger commented 4 years ago

Since adding "--whole-object" to the "rbd diff" command I did not get this error any more. So is it possible that without "--whole-object" the hints file (from "rbd diff") is invalid?

wamdam commented 4 years ago

Oh, interesting. Actually backy2 calculates whole objects internally even from a more detailed hintfile. So it should not make any difference except that it takes longer for ceph to create such a file. Would you be able to send me

the detailed hintfile
the whole-object hintfile for the given backup where it didn't work with the detailed one? I'd love to send them through the parser and see if there are differences. Because if, that would be a (very unexpected) bug because this code is in production for a long time now.

Hint: We're only using --whole-object here.

gschoenberger commented 4 years ago

I will try to reproduce the problem, I have to adapt the backup script also because the hints file is removed immediately after the backup. But first I will check if my referenced backy versions are correct...

gschoenberger commented 4 years ago

I now added some sanity checks to the backup script, to test if hints file and last backy UID are used correctly. Since Thursday, after I added the "whole-objects" flag, backup is running fine. Last time without "whole-objects" I've got the errors in 6 of 20 backups on the first day. So I guess there is definitively some problem with the hints file parser...

wamdam commented 4 years ago

As I can't reproduce this and there were no more hints, I'll close this. Feel free to open a new ticket if this happens again.

gschoenberger commented 4 years ago

I am sorry I did not have time to build up a test environment and check if I can reproduce the problem. The new backy server is already on in a production setup and I don't want to change the setup right now. All I can say is that with "whole-objects" I I did not have a single error since July! Without "whole-objects" I have got the errors from the first comment within the first day...