Closed johndoe31415 closed 1 year ago
johndoe31415
commented
2 years ago Also, because the memory issues have been there for a while (I originally filed #1723 before the prune logic was rewritten) I've now performed some memory profiling. Not sure if this is useful, but I've captured the memory of the restic process every 10 secs and turned it into a plot. In my case, prune takes up about 9 GB of RAM before it dies (X axis is time in seconds, Y axis is memory in GB):
MichaelEischer
commented
2 years ago The damaged pack files appear to be three years old by now, so there's probably not much point in investigating how they were damaged (there have been lots of changes all over the restic codebase). Note the difference between pack files and data blobs. Each pack file consists of several blobs. That is the reason why the pack filenames are totally different from the missing blob ids.
To repair the repository you'll have to follow the steps from https://github.com/restic/restic/issues/828#issuecomment-706186047 . For Route 1 you'll probably have to restore the old index and move the pack files back. If you need help with these steps, just ask.
Regarding the memory usage: the plot does not look totally unexpected, once #3899 is merged, it should top out at about 7GB. You could set the environment variable GOGC=50 to avoid swapping. This tell go to garbage collect more often.
johndoe31415
commented
2 years ago Interesting observation, all corrupt packfiles have the same timestamp. I'm guessing restic touches the files to have them have the same mtime as the snapshot, yes? I fully agree, how they got damaged -- water under the bridge.
But I gotta admit, the repair procedure -- of which I tried following route 2 (missing the difference between pack files and data files) is excruciating painful. I do consider myself a fairly advanced user but it's certainly out of my comfort zone, if not out of my depth.
If these are really old snapshots then I don't care and would just want to delete them, but I need to know that the recent backups are in no way affected (e.g., that there was not some deduplication and the recent backups think that it's fine because some files were backed up in the broken packs).
Right now it's late and I'm tired and I don't trust myself enough to do this correctly. The restic find command already runs for a few minutes (10?) with no results so far. And I have to do this 21 times for each pack? Oh and I just noticed I need to perform a debug build of restic. Which means I need to install go first and figure out how to build it again (I did it once before but Go is fairly alien to me to be honest).
This is all making me very unhappy. Not being sure if my backup is okay is definitely making me uneasy but seeing that the repair PR is unmerged 2 years later means it's probably not getting done anytime soon.
The pragmatic solution would be to remove my history and start fresh with a new repository. Which kindof defies the whole purpose of snapshotted incremental backups. No, this is really making me very unhappy :-( I'll see what I do tomorrow.
EDIT: Don't know if that is a workable solution, it'd take a week:
repository a6b293b6 opened (repository version 2) successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files
[42:08] 0.49% 242724 files 16.058 GiB, total 3603516 files 3.188 TiB, 0 errors ETA 142:02:17The pragmatic solution would be to remove my history and start fresh with a new repository. Which kindof defies the whole purpose of snapshotted incremental backups. No, this is really making me very unhappy :-( I'll see what I do tomorrow.
Running rebuild-index, then removing all snapshots and running backup afterwards would avoid reuploading everything, but should get the repository back into a useable shape. Although, it won't be possible to be completely sure unless restic check --read-data completes successfully.
But I gotta admit, the repair procedure -- of which I tried following route 2 (missing the difference between pack files and data files) is excruciating painful. I do consider myself a fairly advanced user but it's certainly out of my comfort zone, if not out of my depth.
My plan is to eventually let check report which snapshots are damaged which would make "route 2" much simpler. The repair PR will eventually be merged in some form (the command name will probably change), once I get around to review it.
Starting from restic 0.13.0 you can also pass multiple pack ids to find.
If these are really old snapshots then I don't care and would just want to delete them, but I need to know that the recent backups are in no way affected (e.g., that there was not some deduplication and the recent backups think that it's fine because some files were backed up in the broken packs).
After running rebuild-index the new backups will be ok, assuming that all damaged pack files have been removed before.
Which means I need to install go first and figure out how to build it again (I did it once before but Go is fairly alien to me to be honest).
Just get a recent go compiler version (at least >=1.15), checkout the master branch and run go build -tags debug ./cmd/restic. Then you'll have a new restic binary in the current folder.
MichaelEischer
commented
1 year ago Closing as there's nothing left to do here.
Output of
restic versionrestic 0.14.0 compiled with go1.19 on linux/amd64
How did you run restic exactly?
What backend/server/service did you use to store the repository?
Local file storage.
Expected behavior
I expect restic to prune my data.
Actual behavior
It doesn't prune, but crashes and tells me to file a bug -- which is this one.
Steps to reproduce the behavior
It reproducibly crashes when run as shown as above; the system in question only had 8 GB of RAM and so it previously crashed because of an OOM error. I've added 16 GB of swap and now the above error is the result.
Do you have any idea what may have caused this?
This is the backup of my main machine (~4 TB of storage). I've never pruned it before because space is plenty on my target. The oldest roughly 4 years old and occupies in total 6.7 TB in about 600 snapshots and 1.4 million files:
Recently, I wanted to start pruning the repository and maybe get rid of some stuff. Prune failed with a fatal error because some of the pack IDs were wrong. So I followed the advice I read in a ticket here and first did
Which found some corrupt packs among those:
I followed the advice and moved those packs out of their directories into a "corrupt" directory outside of the restic storage. Then, I have made a full copy of my index directory, and finally ran "restic rebuild-index".
Afterwards I again tried the pruning, but as you can see, it did not lead anywhere.
Note that I also checked the duplicate #3365 but I believe this to be a different root cause (they had some remote storage backend, I have just plain old local files).
Do you have an idea how to solve the issue?
No, but I can restore the index pre "rebuild-index" and I can also move the corrupt packs back in their old position if that would be helpful for debugging.
restic prunethen also deterministically crashes.Also note that the blobs that restic complains are missing are not the same that I removed (because they were corrupted). The corrupt ones are:
Did restic help you today? Did it make you happy in any way?
Restic continues to be awesome, no doubt about it.