search

rustic-rs / rustic

rustic - fast, encrypted, and deduplicated backups powered by Rust
https://rustic.cli.rs
Apache License 2.0
1.89k stars 71 forks source link

repair subcommand fails to repair repository #368

Closed liblit closed 1 year ago

liblit commented 1 year ago

A Repository in Need of Repair

I have an existing Restic repository with some damage to repair. Log output from rustic check consists of:

20:57:06 [INFO] repository .: password is correct.
20:57:06 [INFO] using cache at /home/liblit/.cache/rustic/6f6b61809f47f5987114f22dfe8db5dfb31df2bde574d545cfdbefca6360492b
20:57:07 [WARN] pack 49205189 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
20:57:07 [WARN] pack 5534aea9 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
20:57:07 [WARN] pack a2e0e110 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
20:57:07 [WARN] pack aebe7dfe not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
20:57:07 [WARN] pack de494bfa not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
20:57:10 [ERROR] file "..." blob fe38d0c4 is missing in index
20:57:10 [ERROR] file "..." blob 23067cb9 is missing in index
20:57:10 [ERROR] file "..." blob 87b43ef2 is missing in index
20:57:10 [ERROR] file "..." blob f530b70f is missing in index
20:57:10 [ERROR] file "..." blob 7f101ffb is missing in index
20:57:10 [ERROR] file "..." blob 44c8e19e is missing in index
20:57:11 [ERROR] file "..." blob fe38d0c4 is missing in index
20:57:11 [ERROR] file "..." blob 23067cb9 is missing in index
20:57:11 [ERROR] file "..." blob 87b43ef2 is missing in index
20:57:11 [ERROR] file "..." blob f530b70f is missing in index
20:57:11 [ERROR] file "..." blob 7f101ffb is missing in index
20:57:11 [ERROR] file "..." blob 44c8e19e is missing in index
20:57:14 [ERROR] file "..." blob 215fbce8 is missing in index
20:57:17 [ERROR] file "..." blob cdcbf8f1 is missing in index
20:57:17 [ERROR] file "..." blob bd9ea683 is missing in index
20:57:17 [ERROR] file "..." blob 2ea0adf6 is missing in index
20:57:17 [ERROR] file "..." blob 72ce55b5 is missing in index
20:57:17 [ERROR] file "..." blob ad2e8677 is missing in index
20:57:17 [ERROR] file "..." blob 0a662057 is missing in index
20:57:17 [ERROR] file "..." blob a331a9e1 is missing in index
20:57:17 [ERROR] file "..." blob 3e0af590 is missing in index
20:57:17 [ERROR] file "..." blob 8f99be56 is missing in index
20:57:17 [ERROR] file "..." blob a2bcf3e6 is missing in index
20:57:17 [ERROR] file "..." blob 4b4ffbdf is missing in index
20:57:17 [ERROR] file "..." blob f25d5a5d is missing in index
20:57:17 [ERROR] file "..." blob d60284d0 is missing in index
20:57:17 [ERROR] file "..." blob 3ba5bb26 is missing in index
20:57:17 [ERROR] file "..." blob a441984b is missing in index
20:57:17 [ERROR] file "..." blob 631013b1 is missing in index
20:57:17 [ERROR] file "..." blob 4bba84f4 is missing in index
20:57:17 [ERROR] file "..." blob 70a5dde7 is missing in index
20:57:17 [ERROR] file "..." blob e329e6eb is missing in index
20:57:17 [ERROR] file "..." blob 9303464d is missing in index
20:57:17 [ERROR] file "..." blob e208467b is missing in index
20:57:17 [ERROR] file "..." blob c3c58315 is missing in index
20:57:17 [ERROR] file "..." blob b6a40fbc is missing in index
20:57:17 [ERROR] file "..." blob 767e4244 is missing in index
20:57:17 [ERROR] file "..." blob 7634d9e4 is missing in index
20:57:17 [ERROR] file "..." blob 33000afa is missing in index
20:57:17 [ERROR] file "..." blob 5909ec60 is missing in index
20:57:17 [ERROR] file "..." blob cb32e57d is missing in index
20:57:17 [ERROR] file "..." blob 1e57dfe0 is missing in index
20:57:17 [ERROR] file "..." blob c454e30a is missing in index
20:57:18 [ERROR] file "..." blob 444238a8 is missing in index
20:57:18 [ERROR] file "..." blob 7d81de40 is missing in index
20:57:18 [ERROR] file "..." blob 7ca1cbca is missing in index

Index Repair Fails Immediately

Unfortunately, rustic repair index quickly fails, reporting Error: failed to fill whole buffer. The log output from rustic repair index is uninteresting:

21:03:50 [INFO] repository .: password is correct.
21:03:50 [INFO] using cache at /home/liblit/.cache/rustic/6f6b61809f47f5987114f22dfe8db5dfb31df2bde574d545cfdbefca6360492b

Snapshot Repair Does Not Persist

Perhaps I should be using rustic repair snapshots instead? That command appears to do much more work, according to the log output:

21:03:50 [INFO] repository .: password is correct.
21:03:50 [INFO] using cache at /home/liblit/.cache/rustic/6f6b61809f47f5987114f22dfe8db5dfb31df2bde574d545cfdbefca6360492b
21:07:22 [INFO] repository .: password is correct.
21:07:22 [INFO] using cache at /home/liblit/.cache/rustic/6f6b61809f47f5987114f22dfe8db5dfb31df2bde574d545cfdbefca6360492b
21:07:23 [INFO] processing snapshot b7cfb628
21:07:26 [INFO] snapshot b7cfb628 is ok.
21:07:26 [INFO] processing snapshot b4d608b1
21:07:29 [INFO] snapshot b4d608b1 is ok.
21:07:29 [INFO] processing snapshot deac1f8a
21:07:32 [INFO] snapshot deac1f8a is ok.
21:07:32 [INFO] processing snapshot 58aa49f7
21:07:33 [WARN] file ...: contents are missing
21:07:33 [INFO] saved modified snapshot as 957b1ec1.
21:07:33 [INFO] processing snapshot 023a7d22
21:07:33 [INFO] saved modified snapshot as dce424e7.
21:07:33 [INFO] processing snapshot ca19ca65
21:07:34 [INFO] snapshot ca19ca65 is ok.
21:07:34 [INFO] processing snapshot 6ff819b1
21:07:34 [INFO] snapshot 6ff819b1 is ok.
21:07:34 [INFO] processing snapshot 62ae57ee
21:07:34 [INFO] saved modified snapshot as 481a1896.
21:07:34 [INFO] processing snapshot 42dab911
21:07:37 [INFO] snapshot 42dab911 is ok.
21:07:37 [INFO] processing snapshot 01c37f06
21:07:37 [INFO] snapshot 01c37f06 is ok.
21:07:37 [INFO] processing snapshot d107ee36
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [WARN] file ...: contents are missing
21:07:37 [INFO] saved modified snapshot as 96f02dc0.
21:07:37 [INFO] processing snapshot b37cee85
21:07:39 [INFO] snapshot b37cee85 is ok.
21:07:39 [INFO] processing snapshot 51643fbb
21:07:42 [INFO] snapshot 51643fbb is ok.
21:07:42 [INFO] processing snapshot c044f7af
21:07:42 [INFO] snapshot c044f7af is ok.
21:07:42 [INFO] processing snapshot 669a439b
21:07:42 [INFO] saved modified snapshot as 0619948a.
21:07:42 [INFO] processing snapshot 36806e11
21:07:43 [INFO] snapshot 36806e11 is ok.
21:07:43 [INFO] processing snapshot caa2d63d
21:07:43 [INFO] saved modified snapshot as 1a5a0e52.
21:07:43 [INFO] processing snapshot 047c83d3
21:07:43 [INFO] saved modified snapshot as c9f3df7b.
21:07:43 [INFO] processing snapshot e84e1b32
21:07:43 [INFO] snapshot e84e1b32 is ok.
21:07:43 [INFO] processing snapshot 225d696f
21:07:43 [INFO] snapshot 225d696f is ok.
21:07:43 [INFO] processing snapshot cdbab780
21:07:43 [INFO] snapshot cdbab780 is ok.
21:07:43 [INFO] processing snapshot 6732309a
21:07:46 [INFO] saved modified snapshot as fc519646.
21:07:46 [INFO] processing snapshot 04c7ceed
21:07:46 [INFO] saved modified snapshot as 2867ca9e.
21:07:46 [INFO] processing snapshot e879616d
21:07:46 [INFO] snapshot e879616d is ok.
21:07:46 [INFO] processing snapshot 535b6922
21:07:46 [INFO] snapshot 535b6922 is ok.
21:07:46 [INFO] processing snapshot d37be4e9
21:07:49 [INFO] saved modified snapshot as 11a8cf85.
21:07:49 [INFO] processing snapshot 04be72e5
21:07:49 [INFO] snapshot 04be72e5 is ok.
21:07:49 [INFO] processing snapshot 2d262cac
21:07:49 [INFO] snapshot 2d262cac is ok.
21:07:49 [INFO] processing snapshot bed9bb00
21:07:50 [INFO] snapshot bed9bb00 is ok.
21:07:50 [INFO] processing snapshot 6fecc6b1
21:07:50 [INFO] saved modified snapshot as 6ed0c271.
21:07:50 [INFO] processing snapshot ad02320c
21:07:50 [INFO] snapshot ad02320c is ok.
21:07:50 [INFO] processing snapshot 05e07d2a
21:07:51 [INFO] saved modified snapshot as 66d7021b.
21:07:51 [INFO] processing snapshot 3d83d62e
21:07:51 [INFO] snapshot 3d83d62e is ok.
21:07:51 [INFO] processing snapshot a90da217
21:07:51 [INFO] snapshot a90da217 is ok.
21:07:51 [INFO] processing snapshot e0e5b9d7
21:07:51 [INFO] saved modified snapshot as 9477d65f.
21:07:51 [INFO] processing snapshot 6fc87724
21:07:52 [INFO] saved modified snapshot as bde12d5d.
21:07:52 [INFO] processing snapshot 0c2d015f
21:07:54 [INFO] saved modified snapshot as 4b76516d.
21:07:54 [INFO] processing snapshot 2596bc69
21:07:54 [INFO] snapshot 2596bc69 is ok.
21:07:54 [INFO] processing snapshot e77a9f26
21:07:54 [INFO] snapshot e77a9f26 is ok.
21:07:54 [INFO] processing snapshot 695c5c84
21:07:54 [INFO] snapshot 695c5c84 is ok.
21:07:54 [INFO] processing snapshot 59a4b316
21:07:54 [INFO] snapshot 59a4b316 is ok.
21:07:54 [INFO] processing snapshot 810effb8
21:07:54 [INFO] snapshot 810effb8 is ok.
21:07:54 [INFO] processing snapshot e13719ff
21:07:54 [INFO] saved modified snapshot as f22abfac.
21:07:54 [INFO] processing snapshot 0c35cc43
21:07:55 [INFO] saved modified snapshot as 3eca4567.
21:07:55 [INFO] processing snapshot 702ed171
21:07:57 [INFO] saved modified snapshot as 7a1b59fa.
21:07:57 [INFO] processing snapshot 80b1ab95
21:07:57 [INFO] snapshot 80b1ab95 is ok.
21:07:57 [INFO] processing snapshot c451a005
21:07:57 [INFO] snapshot c451a005 is ok.
21:07:57 [INFO] processing snapshot d1b5cd41
21:07:57 [INFO] snapshot d1b5cd41 is ok.
21:07:57 [INFO] processing snapshot e14c3852
21:07:57 [INFO] saved modified snapshot as 395f9587.
21:07:57 [INFO] processing snapshot 5d74933c
21:07:57 [INFO] snapshot 5d74933c is ok.
21:07:57 [INFO] processing snapshot 70f7d551
21:07:57 [INFO] saved modified snapshot as e7ef5dc4.
21:07:57 [INFO] processing snapshot a377be06
21:07:58 [INFO] snapshot a377be06 is ok.
21:07:58 [INFO] processing snapshot 0d8d962c
21:07:58 [INFO] saved modified snapshot as 9a6d650e.
21:07:58 [INFO] processing snapshot 6211ca20
21:07:58 [INFO] snapshot 6211ca20 is ok.
21:07:58 [INFO] processing snapshot d174b6c8
21:07:58 [INFO] snapshot d174b6c8 is ok.
21:07:58 [INFO] processing snapshot feb5c3bf
21:07:58 [INFO] snapshot feb5c3bf is ok.
21:07:58 [INFO] processing snapshot 37fabb26
21:07:58 [INFO] snapshot 37fabb26 is ok.
21:07:58 [INFO] processing snapshot e54602bc
21:07:58 [INFO] saved modified snapshot as 01f80bc5.
21:07:58 [INFO] processing snapshot e74d794a
21:07:58 [INFO] snapshot e74d794a is ok.
21:07:58 [INFO] processing snapshot 7135d1d9
21:08:00 [INFO] saved modified snapshot as d38b16cc.
21:08:00 [INFO] processing snapshot 29c4e9b3
21:08:00 [INFO] snapshot 29c4e9b3 is ok.
21:08:00 [INFO] processing snapshot 1d0aaa64
21:08:03 [INFO] snapshot 1d0aaa64 is ok.
21:08:03 [INFO] processing snapshot 29b59beb
21:08:03 [INFO] snapshot 29b59beb is ok.
21:08:03 [INFO] processing snapshot 88c20ab7
21:08:06 [INFO] snapshot 88c20ab7 is ok.
21:08:06 [INFO] processing snapshot 856dd1cb
21:08:06 [INFO] snapshot 856dd1cb is ok.
21:08:06 [INFO] processing snapshot 5a5671bd
21:08:06 [INFO] snapshot 5a5671bd is ok.
21:08:06 [INFO] processing snapshot 75b64b80
21:08:06 [WARN] file ...: contents are missing
21:08:06 [INFO] saved modified snapshot as 9300251f.
21:08:06 [INFO] processing snapshot 142b35e8
21:08:06 [INFO] saved modified snapshot as fa1dc2b3.
21:08:06 [INFO] processing snapshot e9bbb259
21:08:06 [INFO] saved modified snapshot as 40c949f1.
21:08:06 [INFO] processing snapshot c1dd39c8
21:08:06 [INFO] snapshot c1dd39c8 is ok.
21:08:06 [INFO] processing snapshot ade394fd
21:08:06 [INFO] snapshot ade394fd is ok.
21:08:06 [INFO] processing snapshot 3e4cca83
21:08:06 [INFO] snapshot 3e4cca83 is ok.
21:08:06 [INFO] processing snapshot 76e3e908
21:08:06 [INFO] saved modified snapshot as 86d906ab.
21:08:06 [INFO] processing snapshot 5be33574
21:08:06 [INFO] snapshot 5be33574 is ok.
21:08:06 [INFO] processing snapshot 35afa250
21:08:06 [INFO] snapshot 35afa250 is ok.
21:08:06 [INFO] processing snapshot e2a12f4a
21:08:06 [INFO] saved modified snapshot as 00eb3d5f.
21:08:06 [INFO] processing snapshot e77b915e
21:08:06 [INFO] snapshot e77b915e is ok.
21:08:06 [INFO] processing snapshot 75c0f519
21:08:06 [INFO] saved modified snapshot as 695730fe.
21:08:06 [INFO] processing snapshot 2dbd2a1a
21:08:06 [INFO] snapshot 2dbd2a1a is ok.
21:08:06 [INFO] processing snapshot 7a1f8bc8
21:08:06 [INFO] snapshot 7a1f8bc8 is ok.
21:08:06 [INFO] processing snapshot 14def7ee
21:08:06 [INFO] saved modified snapshot as 9c425440.
21:08:06 [INFO] processing snapshot 2777e37d
21:08:06 [INFO] snapshot 2777e37d is ok.
21:08:06 [INFO] processing snapshot 6085f1b5
21:08:06 [INFO] snapshot 6085f1b5 is ok.
21:08:06 [INFO] processing snapshot ec0f4628
21:08:06 [INFO] saved modified snapshot as 00af0c7b.
21:08:06 [INFO] processing snapshot a5c33d41
21:08:06 [INFO] snapshot a5c33d41 is ok.
21:08:06 [INFO] processing snapshot 45fffc36
21:08:06 [INFO] snapshot 45fffc36 is ok.
21:08:06 [INFO] processing snapshot 77fb13c0
21:08:06 [INFO] saved modified snapshot as 1287ec53.
21:08:06 [INFO] processing snapshot 1ac3388b
21:08:06 [INFO] saved modified snapshot as 3361e1a7.
21:08:06 [INFO] processing snapshot 11303e6c
21:08:06 [INFO] snapshot 11303e6c is ok.
21:08:06 [INFO] processing snapshot 7c00dd70
21:08:06 [INFO] snapshot 7c00dd70 is ok.
21:08:06 [INFO] processing snapshot 30981754
21:08:06 [INFO] snapshot 30981754 is ok.
21:08:06 [INFO] processing snapshot f13826d5
21:08:06 [INFO] saved modified snapshot as 92c70f56.
21:08:06 [INFO] processing snapshot 34f99e31
21:08:06 [INFO] snapshot 34f99e31 is ok.
21:08:06 [INFO] processing snapshot ccd52ea8
21:08:06 [INFO] snapshot ccd52ea8 is ok.
21:08:06 [INFO] processing snapshot 23e4fd80
21:08:06 [INFO] saved modified snapshot as 3fb9cf73.
21:08:06 [INFO] processing snapshot afc87cf1
21:08:06 [INFO] snapshot afc87cf1 is ok.
21:08:06 [INFO] processing snapshot 78fdbd10
21:08:06 [INFO] saved modified snapshot as a39e10ae.
21:08:06 [INFO] processing snapshot fb324fe9
21:08:06 [INFO] snapshot fb324fe9 is ok.
21:08:06 [INFO] processing snapshot f2983710
21:08:06 [INFO] saved modified snapshot as 2c25e98e.
21:08:06 [INFO] processing snapshot 1d7be876
21:08:06 [INFO] saved modified snapshot as 1df19441.
21:08:06 [INFO] processing snapshot 06ab23bb
21:08:06 [INFO] snapshot 06ab23bb is ok.
21:08:06 [INFO] processing snapshot 7821fa6c
21:08:06 [INFO] snapshot 7821fa6c is ok.
21:08:06 [INFO] processing snapshot 5787fa7a
21:08:06 [INFO] snapshot 5787fa7a is ok.
21:08:06 [INFO] processing snapshot fa1ff84e
21:08:06 [INFO] snapshot fa1ff84e is ok.
21:08:06 [INFO] processing snapshot f710bc74
21:08:06 [INFO] saved modified snapshot as 650feec3.
21:08:06 [INFO] processing snapshot 26901e7b
21:08:06 [INFO] saved modified snapshot as 30ef8a94.
21:08:06 [INFO] processing snapshot 69a7145a
21:08:06 [INFO] snapshot 69a7145a is ok.
21:08:06 [INFO] processing snapshot 91d93c3a
21:08:06 [INFO] snapshot 91d93c3a is ok.
21:08:06 [INFO] processing snapshot 8d2776b1
21:08:06 [INFO] saved modified snapshot as 5dda4c37.
21:08:06 [INFO] processing snapshot 2d370209
21:08:06 [INFO] snapshot 2d370209 is ok.
21:08:06 [INFO] processing snapshot 28339af0
21:08:07 [INFO] saved modified snapshot as 40870226.
21:08:07 [INFO] processing snapshot f8ffb876
21:08:07 [INFO] saved modified snapshot as c7ba642b.
21:08:07 [INFO] processing snapshot a030972f
21:08:07 [INFO] snapshot a030972f is ok.
21:08:07 [INFO] processing snapshot 923ec47e
21:08:07 [INFO] snapshot 923ec47e is ok.
21:08:07 [INFO] processing snapshot 5120d426
21:08:07 [INFO] snapshot 5120d426 is ok.
21:08:07 [INFO] processing snapshot 74086bd4
21:08:07 [INFO] snapshot 74086bd4 is ok.
21:08:07 [INFO] processing snapshot 2e1c32bb
21:08:07 [INFO] saved modified snapshot as da7e8789.
21:08:07 [INFO] processing snapshot 991871e9
21:08:07 [INFO] snapshot 991871e9 is ok.
21:08:07 [INFO] processing snapshot 91cce741
21:08:07 [INFO] snapshot 91cce741 is ok.
21:08:07 [INFO] processing snapshot 8a116f2d
21:08:07 [INFO] saved modified snapshot as 5812d6db.
21:08:07 [INFO] processing snapshot a2c43d36
21:08:07 [INFO] snapshot a2c43d36 is ok.
21:08:07 [INFO] processing snapshot acb19db3
21:08:07 [INFO] snapshot acb19db3 is ok.
21:08:07 [INFO] processing snapshot f94498e9
21:08:07 [INFO] saved modified snapshot as e5003d66.
21:08:07 [INFO] processing snapshot 33cdc9f1
21:08:07 [INFO] snapshot 33cdc9f1 is ok.
21:08:07 [INFO] processing snapshot 2b3f82e8
21:08:07 [INFO] saved modified snapshot as 8bae8b82.
21:08:07 [INFO] processing snapshot 91515295
21:08:07 [INFO] snapshot 91515295 is ok.
21:08:07 [INFO] processing snapshot dabe8a95
21:08:07 [INFO] snapshot dabe8a95 is ok.
21:08:07 [INFO] processing snapshot 028e0c16
21:08:07 [INFO] snapshot 028e0c16 is ok.
21:08:07 [INFO] processing snapshot f9751fde
21:08:07 [INFO] saved modified snapshot as b741252d.
21:08:07 [INFO] processing snapshot 5aa7dba2
21:08:07 [INFO] snapshot 5aa7dba2 is ok.
21:08:07 [INFO] processing snapshot 254a3f08
21:08:07 [INFO] snapshot 254a3f08 is ok.
21:08:07 [INFO] processing snapshot 95a213db
21:08:07 [INFO] saved modified snapshot as c230994d.
21:08:07 [INFO] processing snapshot 32ebc150
21:08:07 [INFO] snapshot 32ebc150 is ok.
21:08:07 [INFO] processing snapshot 2ee81a05
21:08:07 [INFO] saved modified snapshot as 798ae7ed.
21:08:07 [INFO] processing snapshot bc64046b
21:08:07 [INFO] snapshot bc64046b is ok.
21:08:07 [INFO] processing snapshot 0751fbbd
21:08:07 [INFO] snapshot 0751fbbd is ok.
21:08:07 [INFO] processing snapshot 97e00a8d
21:08:07 [INFO] saved modified snapshot as bf8da200.
21:08:07 [INFO] processing snapshot ea6959ea
21:08:07 [INFO] snapshot ea6959ea is ok.
21:08:07 [INFO] processing snapshot feb75994
21:08:07 [INFO] saved modified snapshot as 59962757.
21:08:07 [INFO] processing snapshot f3215483
21:08:07 [INFO] snapshot f3215483 is ok.
21:08:07 [INFO] processing snapshot dd42be61
21:08:07 [INFO] snapshot dd42be61 is ok.
21:08:07 [INFO] processing snapshot 3aa7610c
21:08:07 [INFO] saved modified snapshot as f991e1c7.
21:08:07 [INFO] processing snapshot ef8d4004
21:08:07 [INFO] snapshot ef8d4004 is ok.
21:08:07 [INFO] processing snapshot 9e4adc9f
21:08:11 [INFO] snapshot 9e4adc9f is ok.
21:08:11 [INFO] processing snapshot fedb194b
21:08:11 [INFO] saved modified snapshot as e445a4b8.
21:08:11 [INFO] processing snapshot 94528fd1
21:08:11 [INFO] snapshot 94528fd1 is ok.
21:08:11 [INFO] processing snapshot b59cdc5b
21:08:11 [INFO] snapshot b59cdc5b is ok.
21:08:11 [INFO] processing snapshot a4af7ef9
21:08:11 [INFO] snapshot a4af7ef9 is ok.
21:08:11 [INFO] processing snapshot 33986da3
21:08:11 [INFO] saved modified snapshot as 747943fb.
21:08:11 [INFO] processing snapshot 83aa9f59
21:08:11 [INFO] snapshot 83aa9f59 is ok.
21:08:11 [INFO] processing snapshot ef259993
21:08:11 [INFO] snapshot ef259993 is ok.
21:08:11 [INFO] processing snapshot 9c2fb666
21:08:11 [INFO] saved modified snapshot as bfb4d33b.
21:08:11 [INFO] processing snapshot ff204d7d
21:08:11 [INFO] saved modified snapshot as 821e6fb1.
21:08:11 [INFO] processing snapshot 4b950eac
21:08:11 [INFO] snapshot 4b950eac is ok.
21:08:11 [INFO] processing snapshot 147ec3b3
21:08:11 [INFO] snapshot 147ec3b3 is ok.
21:08:11 [INFO] processing snapshot a46dbf0b
21:08:11 [INFO] snapshot a46dbf0b is ok.
21:08:11 [INFO] processing snapshot af90d42e
21:08:11 [INFO] snapshot af90d42e is ok.
21:08:11 [INFO] processing snapshot 43393d82
21:08:11 [INFO] snapshot 43393d82 is ok.
21:08:11 [INFO] processing snapshot f734cd5d
21:08:11 [INFO] saved modified snapshot as e8d5ab35.
21:08:11 [INFO] processing snapshot 3b43b1b0
21:08:11 [INFO] saved modified snapshot as d7c67060.
21:08:11 [INFO] processing snapshot a93a2bab
21:08:11 [INFO] saved modified snapshot as 5adbb813.
21:08:11 [INFO] processing snapshot 9ba1a991
21:08:12 [INFO] snapshot 9ba1a991 is ok.
21:08:12 [INFO] processing snapshot 32d6ddec
21:08:12 [INFO] snapshot 32d6ddec is ok.
21:08:12 [INFO] processing snapshot a5420f03
21:08:12 [INFO] saved modified snapshot as 92046e0d.
21:08:12 [INFO] processing snapshot 3dba2207
21:08:12 [INFO] saved modified snapshot as e0172295.
21:08:12 [INFO] processing snapshot 9ec043b0
21:08:12 [INFO] snapshot 9ec043b0 is ok.
21:08:12 [INFO] processing snapshot 0d998299
21:08:12 [INFO] snapshot 0d998299 is ok.
21:08:12 [INFO] processing snapshot 7f61c1b7
21:08:12 [INFO] snapshot 7f61c1b7 is ok.
21:08:12 [INFO] processing snapshot 40876cc0
21:08:12 [INFO] saved modified snapshot as 5ede8b98.
21:08:12 [INFO] processing snapshot d10d3bbe
21:08:12 [INFO] snapshot d10d3bbe is ok.
21:08:12 [INFO] processing snapshot e21d122e
21:08:12 [INFO] snapshot e21d122e is ok.
21:08:12 [INFO] processing snapshot 3eedd5d2
21:08:12 [INFO] saved modified snapshot as 154b01a9.
21:08:12 [INFO] processing snapshot b36bef28
21:08:12 [INFO] saved modified snapshot as fc7cc88b.
21:08:12 [INFO] processing snapshot 08291c91
21:08:12 [INFO] snapshot 08291c91 is ok.
21:08:12 [INFO] processing snapshot 7ca8e732
21:08:12 [INFO] snapshot 7ca8e732 is ok.
21:08:12 [INFO] processing snapshot 53a2511c
21:08:12 [INFO] snapshot 53a2511c is ok.
21:08:12 [INFO] processing snapshot 37ef1cd4
21:08:12 [INFO] snapshot 37ef1cd4 is ok.
21:08:12 [INFO] processing snapshot c604b084
21:08:12 [INFO] saved modified snapshot as f91f8d09.
21:08:12 [INFO] processing snapshot b442b025
21:08:12 [INFO] saved modified snapshot as bb08774f.
21:08:12 [INFO] processing snapshot 64a8f7a7
21:08:12 [INFO] snapshot 64a8f7a7 is ok.
21:08:12 [INFO] processing snapshot 42243d8d
21:08:12 [INFO] saved modified snapshot as 4f11e293.
21:08:12 [INFO] processing snapshot c18bd4a2
21:08:12 [INFO] snapshot c18bd4a2 is ok.
21:08:12 [INFO] processing snapshot a8c70430
21:08:12 [INFO] snapshot a8c70430 is ok.
21:08:12 [INFO] processing snapshot f4cd716f
21:08:12 [INFO] snapshot f4cd716f is ok.
21:08:12 [INFO] processing snapshot c85ae99c
21:08:12 [INFO] snapshot c85ae99c is ok.
21:08:12 [INFO] processing snapshot 3e16e4b3
21:08:12 [INFO] saved modified snapshot as 3a427bf4.
21:08:12 [INFO] processing snapshot 3db0a2ac
21:08:12 [INFO] saved modified snapshot as 5ec10572.
21:08:12 [INFO] processing snapshot bc79aed6
21:08:12 [INFO] saved modified snapshot as 7f4be8e8.
21:08:12 [INFO] processing snapshot 0a0b8252
21:08:12 [INFO] snapshot 0a0b8252 is ok.
21:08:12 [INFO] processing snapshot d968fb95
21:08:12 [INFO] snapshot d968fb95 is ok.
21:08:12 [INFO] processing snapshot 73c9955b
21:08:12 [INFO] snapshot 73c9955b is ok.
21:08:12 [INFO] processing snapshot cfd9680d
21:08:12 [INFO] snapshot cfd9680d is ok.
21:08:12 [INFO] processing snapshot 43bf6928
21:08:12 [INFO] saved modified snapshot as 2e89bb3d.
21:08:12 [INFO] processing snapshot 3555de95
21:08:12 [INFO] saved modified snapshot as 4e948eb9.
21:08:12 [INFO] processing snapshot bf090309
21:08:12 [INFO] snapshot bf090309 is ok.
21:08:12 [INFO] processing snapshot 684407c5
21:08:12 [INFO] snapshot 684407c5 is ok.
21:08:12 [INFO] processing snapshot c4d7a078
21:08:12 [INFO] snapshot c4d7a078 is ok.
21:08:12 [INFO] processing snapshot bd86c1d7
21:08:12 [INFO] saved modified snapshot as 591f345b.
21:08:12 [INFO] processing snapshot 581668b1
21:08:12 [INFO] snapshot 581668b1 is ok.
21:08:12 [INFO] processing snapshot 7b3931e5
21:08:12 [INFO] snapshot 7b3931e5 is ok.
21:08:12 [INFO] processing snapshot 85a853f8
21:08:12 [INFO] saved modified snapshot as c180d8b0.
21:08:12 [INFO] processing snapshot bf7d1261
21:08:12 [INFO] saved modified snapshot as 60725a8f.
21:08:12 [INFO] processing snapshot ad105180
21:08:12 [INFO] snapshot ad105180 is ok.
21:08:12 [INFO] processing snapshot d1fb0f4e
21:08:12 [INFO] saved modified snapshot as 9030b43f.
21:08:12 [INFO] processing snapshot 83772155
21:08:12 [INFO] snapshot 83772155 is ok.
21:08:12 [INFO] processing snapshot 278926fe
21:08:12 [INFO] snapshot 278926fe is ok.
21:08:12 [INFO] processing snapshot 466a52e3
21:08:12 [INFO] saved modified snapshot as 5981b9de.
21:08:12 [INFO] processing snapshot c4a21719
21:08:12 [INFO] saved modified snapshot as 95f49fe5.
21:08:12 [INFO] processing snapshot d8d3d3f0
21:08:12 [INFO] snapshot d8d3d3f0 is ok.
21:08:12 [INFO] processing snapshot a21a8318
21:08:12 [INFO] snapshot a21a8318 is ok.
21:08:12 [INFO] processing snapshot de0514ad
21:08:12 [INFO] saved modified snapshot as b6a3cfe1.
21:08:12 [INFO] processing snapshot 4bc168da
21:08:12 [INFO] saved modified snapshot as 9836a400.
21:08:12 [INFO] processing snapshot ac74139c
21:08:12 [INFO] snapshot ac74139c is ok.
21:08:12 [INFO] processing snapshot f88103e0
21:08:12 [INFO] snapshot f88103e0 is ok.
21:08:12 [INFO] processing snapshot 4ef568e2
21:08:12 [INFO] snapshot 4ef568e2 is ok.
21:08:12 [INFO] processing snapshot bfef4848
21:08:12 [INFO] saved modified snapshot as 36b279a0.
21:08:12 [INFO] processing snapshot 0d3eb8cf
21:08:12 [INFO] snapshot 0d3eb8cf is ok.
21:08:12 [INFO] processing snapshot a5a438da
21:08:12 [INFO] snapshot a5a438da is ok.
21:08:12 [INFO] processing snapshot ca6c2818
21:08:12 [INFO] saved modified snapshot as 3524bfde.
21:08:12 [INFO] processing snapshot 51f4b81a
21:08:12 [INFO] saved modified snapshot as f1ecc86e.
21:08:12 [INFO] processing snapshot 73538b5e
21:08:12 [INFO] snapshot 73538b5e is ok.
21:08:12 [INFO] processing snapshot 627fafd0
21:08:12 [INFO] snapshot 627fafd0 is ok.
21:08:12 [INFO] processing snapshot 780beeb0
21:08:12 [INFO] snapshot 780beeb0 is ok.
21:08:12 [INFO] processing snapshot 0b68fd4a
21:08:12 [INFO] snapshot 0b68fd4a is ok.
21:08:12 [INFO] processing snapshot c9a53048
21:08:12 [INFO] saved modified snapshot as 92136cab.
21:08:12 [INFO] processing snapshot de0424ff
21:08:12 [INFO] saved modified snapshot as f371727b.
21:08:12 [INFO] processing snapshot 4d23089f
21:08:12 [INFO] saved modified snapshot as 8344dc0e.
21:08:12 [INFO] processing snapshot d5d4e90e
21:08:12 [INFO] saved modified snapshot as 069ef6ce.
21:08:12 [INFO] processing snapshot 73fd1875
21:08:12 [INFO] saved modified snapshot as 1bff3219.
21:08:12 [INFO] processing snapshot b13aeb94
21:08:12 [INFO] snapshot b13aeb94 is ok.
21:08:12 [INFO] processing snapshot 0581c577
21:08:12 [INFO] snapshot 0581c577 is ok.
21:08:12 [INFO] processing snapshot 1ef21cd0
21:08:12 [INFO] snapshot 1ef21cd0 is ok.
21:08:12 [INFO] processing snapshot 8a00c31c
21:08:12 [INFO] saved modified snapshot as cd06233e.
21:08:12 [INFO] processing snapshot d5df19e2
21:08:12 [INFO] snapshot d5df19e2 is ok.
21:08:12 [INFO] processing snapshot 2ce10524
21:08:12 [INFO] snapshot 2ce10524 is ok.
21:08:12 [INFO] processing snapshot 81069fc7
21:08:12 [INFO] snapshot 81069fc7 is ok.
21:08:12 [INFO] processing snapshot d74e02a4
21:08:12 [INFO] snapshot d74e02a4 is ok.
21:08:12 [INFO] processing snapshot a2802f6c
21:08:12 [INFO] saved modified snapshot as f5d70c59.
21:08:12 [INFO] processing snapshot 55cea95e
21:08:12 [INFO] saved modified snapshot as 26084c29.
21:08:12 [INFO] processing snapshot becf57fc
21:08:12 [INFO] snapshot becf57fc is ok.
21:08:12 [INFO] processing snapshot 3b6826bf
21:08:12 [INFO] snapshot 3b6826bf is ok.
21:08:12 [INFO] processing snapshot 870313d4
21:08:12 [INFO] snapshot 870313d4 is ok.
21:08:12 [INFO] processing snapshot 4ee57834
21:08:12 [INFO] snapshot 4ee57834 is ok.
21:08:12 [INFO] processing snapshot d4417017
21:08:12 [INFO] snapshot d4417017 is ok.
21:08:12 [INFO] processing snapshot 3fa7e9f4
21:08:12 [INFO] snapshot 3fa7e9f4 is ok.
21:08:12 [INFO] processing snapshot b50a744d
21:08:12 [INFO] snapshot b50a744d is ok.
21:08:12 [INFO] processing snapshot ec44bc77
21:08:12 [INFO] snapshot ec44bc77 is ok.
21:08:12 [INFO] processing snapshot 18db3d03
21:08:12 [INFO] saved modified snapshot as 8208f221.
21:08:12 [INFO] processing snapshot fd31633f
21:08:12 [INFO] snapshot fd31633f is ok.
21:08:12 [INFO] processing snapshot 9945f1a7
21:08:12 [INFO] snapshot 9945f1a7 is ok.
21:08:12 [INFO] processing snapshot 3af52d17
21:08:12 [INFO] snapshot 3af52d17 is ok.
21:08:12 [INFO] processing snapshot 84701c5d
21:08:12 [INFO] snapshot 84701c5d is ok.
21:08:12 [INFO] processing snapshot db4f00eb
21:08:12 [INFO] snapshot db4f00eb is ok.
21:08:12 [INFO] processing snapshot 842b3e76
21:08:12 [INFO] snapshot 842b3e76 is ok.
21:08:12 [INFO] processing snapshot 6f82d06d
21:08:12 [INFO] saved modified snapshot as 4f4632b0.
21:08:12 [INFO] processing snapshot 791204cf
21:08:12 [INFO] snapshot 791204cf is ok.
21:08:12 [INFO] processing snapshot 976a8de1
21:08:12 [INFO] snapshot 976a8de1 is ok.
21:08:12 [INFO] processing snapshot 739a1bec
21:08:12 [INFO] snapshot 739a1bec is ok.
21:08:12 [INFO] processing snapshot 41f23d8c
21:08:12 [INFO] snapshot 41f23d8c is ok.
21:08:12 [INFO] processing snapshot f2bac0c5
21:08:12 [INFO] snapshot f2bac0c5 is ok.
21:08:12 [INFO] processing snapshot 387e0975
21:08:12 [INFO] snapshot 387e0975 is ok.
21:08:12 [INFO] processing snapshot 5451d5f9
21:08:12 [INFO] snapshot 5451d5f9 is ok.
21:08:12 [INFO] processing snapshot 8c88f524
21:08:12 [INFO] saved modified snapshot as 92645425.
21:08:12 [INFO] processing snapshot d65300cd
21:08:12 [INFO] snapshot d65300cd is ok.
21:08:12 [INFO] processing snapshot fb4c8922
21:08:12 [INFO] snapshot fb4c8922 is ok.
21:08:12 [INFO] processing snapshot 22d101aa
21:08:12 [INFO] snapshot 22d101aa is ok.
21:08:12 [INFO] processing snapshot b64d0df3
21:08:12 [INFO] snapshot b64d0df3 is ok.
21:08:12 [INFO] processing snapshot 4ad8af84
21:08:12 [INFO] snapshot 4ad8af84 is ok.
21:08:12 [INFO] processing snapshot b047541f
21:08:12 [INFO] snapshot b047541f is ok.
21:08:12 [INFO] processing snapshot 07c77176
21:08:12 [INFO] saved modified snapshot as 6726cf9a.
21:08:12 [INFO] processing snapshot a6309334
21:08:12 [INFO] snapshot a6309334 is ok.
21:08:12 [INFO] processing snapshot b6f8dea1
21:08:12 [INFO] snapshot b6f8dea1 is ok.
21:08:12 [INFO] processing snapshot 5ddda615
21:08:12 [INFO] snapshot 5ddda615 is ok.
21:08:12 [INFO] processing snapshot 076e6c82
21:08:12 [INFO] snapshot 076e6c82 is ok.
21:08:12 [INFO] processing snapshot 5785db23
21:08:12 [INFO] snapshot 5785db23 is ok.
21:08:12 [INFO] processing snapshot 99e0e34a
21:08:12 [INFO] snapshot 99e0e34a is ok.
21:08:12 [INFO] processing snapshot f56d2c64
21:08:12 [INFO] snapshot f56d2c64 is ok.
21:08:12 [INFO] processing snapshot a6862419
21:08:12 [INFO] snapshot a6862419 is ok.
21:08:12 [INFO] processing snapshot 4773cf5e
21:08:12 [INFO] snapshot 4773cf5e is ok.
21:08:12 [INFO] processing snapshot 55b3d3f0
21:08:12 [INFO] snapshot 55b3d3f0 is ok.
21:08:12 [INFO] processing snapshot acde0e6d
21:08:12 [INFO] snapshot acde0e6d is ok.
21:08:12 [INFO] processing snapshot 3809e7dd
21:08:12 [INFO] snapshot 3809e7dd is ok.
21:08:12 [INFO] processing snapshot 37bbeecb
21:08:12 [INFO] snapshot 37bbeecb is ok.
21:08:12 [INFO] processing snapshot cc5ab9b6
21:08:12 [INFO] snapshot cc5ab9b6 is ok.
21:08:12 [INFO] processing snapshot c917be1f
21:08:12 [INFO] snapshot c917be1f is ok.
21:08:12 [INFO] processing snapshot f7989fe8
21:08:12 [INFO] snapshot f7989fe8 is ok.
21:08:12 [INFO] processing snapshot 1ce683a5
21:08:12 [INFO] snapshot 1ce683a5 is ok.
21:08:12 [INFO] processing snapshot 7f2f1165
21:08:12 [INFO] snapshot 7f2f1165 is ok.
21:08:12 [INFO] processing snapshot f87cb793
21:08:12 [INFO] snapshot f87cb793 is ok.
21:08:12 [INFO] processing snapshot fd3e66a2
21:08:12 [INFO] snapshot fd3e66a2 is ok.
21:08:12 [INFO] processing snapshot 9b8f0914
21:08:12 [INFO] snapshot 9b8f0914 is ok.
21:08:12 [INFO] processing snapshot ed49ec2f
21:08:12 [INFO] snapshot ed49ec2f is ok.
21:08:12 [INFO] processing snapshot 3c618a91
21:08:12 [INFO] snapshot 3c618a91 is ok.
21:08:12 [INFO] processing snapshot 650bd60a
21:08:12 [INFO] snapshot 650bd60a is ok.
21:08:12 [INFO] processing snapshot c1b5a526
21:08:12 [INFO] snapshot c1b5a526 is ok.
21:08:12 [INFO] processing snapshot a9082229
21:08:12 [INFO] snapshot a9082229 is ok.
21:08:12 [INFO] processing snapshot 3bcd7a0e
21:08:12 [INFO] snapshot 3bcd7a0e is ok.
21:08:12 [INFO] processing snapshot 5101d814
21:08:12 [INFO] snapshot 5101d814 is ok.
21:08:12 [INFO] processing snapshot 10e69771
21:08:12 [INFO] snapshot 10e69771 is ok.
21:08:12 [INFO] processing snapshot 02933c97
21:08:12 [INFO] snapshot 02933c97 is ok.
21:08:12 [INFO] processing snapshot 3d706139
21:08:12 [INFO] snapshot 3d706139 is ok.
21:08:12 [INFO] processing snapshot 1328d296
21:08:12 [INFO] snapshot 1328d296 is ok.
21:08:12 [INFO] processing snapshot 58b5de31
21:08:12 [INFO] snapshot 58b5de31 is ok.
21:08:12 [INFO] processing snapshot b4db6be5
21:08:12 [INFO] snapshot b4db6be5 is ok.
21:08:12 [INFO] processing snapshot 2a6cb40f
21:08:12 [INFO] snapshot 2a6cb40f is ok.
21:08:12 [INFO] processing snapshot 747821c2
21:08:12 [INFO] snapshot 747821c2 is ok.
21:08:12 [INFO] processing snapshot 597b44ef
21:08:12 [INFO] snapshot 597b44ef is ok.
21:08:12 [INFO] processing snapshot 958b79eb
21:08:12 [INFO] snapshot 958b79eb is ok.
21:08:12 [INFO] processing snapshot cfe43910
21:08:12 [INFO] snapshot cfe43910 is ok.
21:08:12 [INFO] processing snapshot 3caec451
21:08:12 [INFO] snapshot 3caec451 is ok.
21:08:12 [INFO] processing snapshot b263781d
21:08:12 [INFO] snapshot b263781d is ok.
21:08:12 [INFO] processing snapshot c3c68770
21:08:12 [INFO] snapshot c3c68770 is ok.
21:08:12 [INFO] processing snapshot 3d86441f
21:08:12 [INFO] snapshot 3d86441f is ok.
21:08:12 [INFO] processing snapshot ba8ec864
21:08:12 [INFO] snapshot ba8ec864 is ok.
21:08:12 [INFO] processing snapshot 2459bfd2
21:08:12 [INFO] snapshot 2459bfd2 is ok.
21:08:12 [INFO] processing snapshot 57733273
21:08:12 [INFO] snapshot 57733273 is ok.
21:08:12 [INFO] processing snapshot c4c402b7
21:08:12 [INFO] snapshot c4c402b7 is ok.
21:08:12 [INFO] processing snapshot 633b6f3f
21:08:12 [INFO] snapshot 633b6f3f is ok.
21:08:12 [INFO] processing snapshot 7843ec2f
21:08:12 [INFO] snapshot 7843ec2f is ok.
21:08:12 [INFO] processing snapshot 960e7721
21:08:12 [INFO] snapshot 960e7721 is ok.
21:08:12 [INFO] processing snapshot c220e2f6
21:08:12 [INFO] snapshot c220e2f6 is ok.
21:08:12 [INFO] processing snapshot e05db06c
21:08:12 [INFO] snapshot e05db06c is ok.
21:08:12 [INFO] processing snapshot 93512171
21:08:12 [INFO] snapshot 93512171 is ok.
21:08:12 [INFO] processing snapshot 568ac65a
21:08:12 [INFO] snapshot 568ac65a is ok.
21:08:12 [INFO] processing snapshot 93b2b1e6
21:08:12 [INFO] snapshot 93b2b1e6 is ok.
21:08:12 [INFO] processing snapshot c6680f99
21:08:12 [INFO] snapshot c6680f99 is ok.
21:08:12 [INFO] processing snapshot f501b98a
21:08:12 [INFO] snapshot f501b98a is ok.
21:08:12 [INFO] processing snapshot 6943359c
21:08:12 [INFO] snapshot 6943359c is ok.
21:08:12 [INFO] processing snapshot 7c5be57b
21:08:12 [INFO] snapshot 7c5be57b is ok.
21:08:12 [INFO] processing snapshot 879b931c
21:08:12 [INFO] snapshot 879b931c is ok.
21:08:12 [INFO] processing snapshot edfc1229
21:08:12 [INFO] snapshot edfc1229 is ok.
21:08:12 [INFO] processing snapshot a78ba168
21:08:12 [INFO] snapshot a78ba168 is ok.
21:08:12 [INFO] processing snapshot 4dc774c1
21:08:12 [INFO] snapshot 4dc774c1 is ok.
21:08:12 [INFO] processing snapshot 080364b3
21:08:12 [INFO] snapshot 080364b3 is ok.
21:08:12 [INFO] processing snapshot 0666e67e
21:08:12 [INFO] snapshot 0666e67e is ok.
21:08:12 [INFO] processing snapshot 1031f1e0
21:08:12 [INFO] snapshot 1031f1e0 is ok.
21:08:12 [INFO] processing snapshot 195565c0
21:08:12 [INFO] snapshot 195565c0 is ok.
21:08:12 [INFO] processing snapshot 21f124f1
21:08:12 [INFO] snapshot 21f124f1 is ok.
21:08:12 [INFO] processing snapshot 8ea6b8ed
21:08:12 [INFO] snapshot 8ea6b8ed is ok.
21:08:12 [INFO] processing snapshot f245d32e
21:08:12 [INFO] snapshot f245d32e is ok.
21:08:12 [INFO] processing snapshot d8ed767e
21:08:12 [INFO] snapshot d8ed767e is ok.
21:08:12 [INFO] processing snapshot a3ad963e
21:08:12 [INFO] snapshot a3ad963e is ok.
21:08:12 [INFO] processing snapshot 5b2dec52
21:08:12 [INFO] snapshot 5b2dec52 is ok.
21:08:12 [INFO] processing snapshot a8306a1f
21:08:12 [INFO] snapshot a8306a1f is ok.
21:08:12 [INFO] processing snapshot d967a42f
21:08:12 [INFO] snapshot d967a42f is ok.
21:08:12 [INFO] processing snapshot 6b65cd5a
21:08:12 [INFO] snapshot 6b65cd5a is ok.
21:08:12 [INFO] processing snapshot 15a89db8
21:08:12 [INFO] snapshot 15a89db8 is ok.
21:08:12 [INFO] processing snapshot aec9f83c
21:08:12 [INFO] snapshot aec9f83c is ok.
21:08:12 [INFO] processing snapshot fbfe06ec
21:08:12 [INFO] snapshot fbfe06ec is ok.
21:08:12 [INFO] processing snapshot d97239d7
21:08:12 [INFO] snapshot d97239d7 is ok.
21:08:12 [INFO] processing snapshot c70c2e03
21:08:12 [INFO] snapshot c70c2e03 is ok.
21:08:12 [INFO] processing snapshot 0bfb0699
21:08:12 [INFO] snapshot 0bfb0699 is ok.
21:08:12 [INFO] processing snapshot fc9d2e22
21:08:12 [INFO] snapshot fc9d2e22 is ok.
21:08:12 [INFO] processing snapshot 94219b9f
21:08:12 [INFO] snapshot 94219b9f is ok.
21:08:12 [INFO] processing snapshot cdd0fba6
21:08:12 [INFO] snapshot cdd0fba6 is ok.
21:08:12 [INFO] processing snapshot 8a9c61e0
21:08:12 [INFO] snapshot 8a9c61e0 is ok.
21:08:12 [INFO] processing snapshot c138a37a
21:08:12 [INFO] snapshot c138a37a is ok.
21:08:12 [INFO] processing snapshot 091eeb50
21:08:12 [INFO] snapshot 091eeb50 is ok.
21:08:12 [INFO] processing snapshot 8f2d5348
21:08:12 [INFO] snapshot 8f2d5348 is ok.
21:08:12 [INFO] processing snapshot 6fe21bf0
21:08:12 [INFO] snapshot 6fe21bf0 is ok.
21:08:12 [INFO] processing snapshot 5fd4da19
21:08:12 [INFO] snapshot 5fd4da19 is ok.
21:08:12 [INFO] processing snapshot ff305eae
21:08:12 [INFO] snapshot ff305eae is ok.
21:08:12 [INFO] processing snapshot 00efb377
21:08:12 [INFO] snapshot 00efb377 is ok.
21:08:12 [INFO] processing snapshot 10c09663
21:08:12 [INFO] snapshot 10c09663 is ok.
21:08:12 [INFO] processing snapshot 8bdae0db
21:08:12 [INFO] snapshot 8bdae0db is ok.
21:08:12 [INFO] processing snapshot 4bb79337
21:08:12 [INFO] snapshot 4bb79337 is ok.
21:08:12 [INFO] processing snapshot 1bd592d6
21:08:12 [INFO] snapshot 1bd592d6 is ok.
21:08:12 [INFO] processing snapshot 00fe610f
21:08:12 [INFO] snapshot 00fe610f is ok.
21:08:12 [INFO] processing snapshot c2fe4cdf
21:08:12 [INFO] snapshot c2fe4cdf is ok.
21:08:12 [INFO] processing snapshot 323c6075
21:08:12 [INFO] snapshot 323c6075 is ok.
21:08:12 [INFO] processing snapshot 7786e8d3
21:08:12 [INFO] snapshot 7786e8d3 is ok.
21:08:12 [INFO] processing snapshot c8432eab
21:08:12 [INFO] snapshot c8432eab is ok.
21:08:12 [INFO] processing snapshot 0b19801f
21:08:12 [INFO] snapshot 0b19801f is ok.
21:08:12 [INFO] processing snapshot 711175cb
21:08:12 [INFO] snapshot 711175cb is ok.
21:08:12 [INFO] processing snapshot f4486334
21:08:12 [INFO] snapshot f4486334 is ok.
21:08:12 [INFO] processing snapshot 58725adb
21:08:12 [INFO] snapshot 58725adb is ok.
21:08:12 [INFO] processing snapshot 88e915d0
21:08:12 [INFO] snapshot 88e915d0 is ok.
21:08:12 [INFO] processing snapshot 86c7f3e7
21:08:12 [INFO] snapshot 86c7f3e7 is ok.
21:08:12 [INFO] processing snapshot 7464be07
21:08:12 [INFO] snapshot 7464be07 is ok.
21:08:12 [INFO] processing snapshot cf89dd8e
21:08:12 [INFO] snapshot cf89dd8e is ok.
21:08:12 [INFO] processing snapshot 6aa31c95
21:08:12 [INFO] snapshot 6aa31c95 is ok.
21:08:12 [INFO] processing snapshot 6be3bab3
21:08:12 [INFO] snapshot 6be3bab3 is ok.
21:08:12 [INFO] processing snapshot af7dd3ce
21:08:12 [INFO] snapshot af7dd3ce is ok.
21:08:12 [INFO] processing snapshot 44586241
21:08:12 [INFO] snapshot 44586241 is ok.
21:08:12 [INFO] processing snapshot 7c57dc52
21:08:12 [INFO] snapshot 7c57dc52 is ok.
21:08:12 [INFO] processing snapshot 6c6afdc9
21:08:12 [INFO] snapshot 6c6afdc9 is ok.
21:08:12 [INFO] processing snapshot 74ec952d
21:08:12 [INFO] snapshot 74ec952d is ok.
21:08:12 [INFO] processing snapshot 3db0b53c
21:08:12 [INFO] snapshot 3db0b53c is ok.
21:08:12 [INFO] processing snapshot b5198a5a
21:08:12 [INFO] snapshot b5198a5a is ok.
21:08:12 [INFO] processing snapshot a903bde5
21:08:12 [INFO] snapshot a903bde5 is ok.
21:08:12 [INFO] processing snapshot 9d493b01
21:08:12 [INFO] snapshot 9d493b01 is ok.
21:08:12 [INFO] processing snapshot d35cfa61
21:08:12 [INFO] snapshot d35cfa61 is ok.
21:08:12 [INFO] processing snapshot 95b82937
21:08:12 [INFO] snapshot 95b82937 is ok.
21:08:12 [INFO] processing snapshot a346a822
21:08:12 [INFO] snapshot a346a822 is ok.
21:08:12 [INFO] processing snapshot 45aa28ca
21:08:12 [INFO] snapshot 45aa28ca is ok.
21:08:12 [INFO] processing snapshot 222023fc
21:08:12 [INFO] snapshot 222023fc is ok.
21:08:12 [INFO] processing snapshot 6d5e9f51
21:08:12 [INFO] snapshot 6d5e9f51 is ok.
21:08:12 [INFO] processing snapshot 662fbf7f
21:08:12 [INFO] snapshot 662fbf7f is ok.
21:08:12 [INFO] processing snapshot b7f6aba1
21:08:12 [INFO] snapshot b7f6aba1 is ok.
21:08:12 [INFO] processing snapshot 29b3b5e4
21:08:12 [INFO] snapshot 29b3b5e4 is ok.
21:08:12 [INFO] processing snapshot 125a1079
21:08:12 [INFO] snapshot 125a1079 is ok.
21:08:12 [INFO] processing snapshot 4ced2e91
21:08:12 [INFO] snapshot 4ced2e91 is ok.
21:08:12 [INFO] processing snapshot ee01a1db
21:08:12 [INFO] snapshot ee01a1db is ok.
21:08:12 [INFO] processing snapshot 3842a341
21:08:12 [INFO] snapshot 3842a341 is ok.
21:08:12 [INFO] processing snapshot 4a8a4306
21:08:12 [INFO] snapshot 4a8a4306 is ok.
21:08:12 [INFO] processing snapshot 8a5d814b
21:08:12 [INFO] snapshot 8a5d814b is ok.
21:08:12 [INFO] processing snapshot 41aa18bb
21:08:12 [INFO] snapshot 41aa18bb is ok.
21:08:12 [INFO] processing snapshot 00a8599d
21:08:12 [INFO] snapshot 00a8599d is ok.
21:08:12 [INFO] processing snapshot cbb6f5eb
21:08:12 [INFO] snapshot cbb6f5eb is ok.
21:08:12 [INFO] processing snapshot 3b88e087
21:08:12 [INFO] snapshot 3b88e087 is ok.
21:08:12 [INFO] processing snapshot e6aafeb6
21:08:12 [INFO] snapshot e6aafeb6 is ok.
21:08:12 [INFO] processing snapshot 8d105e63
21:08:12 [INFO] snapshot 8d105e63 is ok.
21:08:12 [INFO] processing snapshot f84a8b15
21:08:12 [INFO] snapshot f84a8b15 is ok.
21:08:12 [INFO] processing snapshot 8e1bb5b2
21:08:12 [INFO] snapshot 8e1bb5b2 is ok.
21:08:12 [INFO] processing snapshot 550c26ad
21:08:12 [INFO] snapshot 550c26ad is ok.
21:08:12 [INFO] processing snapshot 75ebcf35
21:08:12 [INFO] snapshot 75ebcf35 is ok.
21:08:12 [INFO] processing snapshot 2a0011e9
21:08:12 [INFO] snapshot 2a0011e9 is ok.
21:08:12 [INFO] processing snapshot 4a94148e
21:08:12 [INFO] snapshot 4a94148e is ok.
21:08:12 [INFO] processing snapshot f814c5e8
21:08:12 [INFO] snapshot f814c5e8 is ok.
21:08:12 [INFO] processing snapshot 22af4ed7
21:08:12 [INFO] snapshot 22af4ed7 is ok.
21:08:12 [INFO] processing snapshot 7429ee89
21:08:12 [INFO] snapshot 7429ee89 is ok.
21:08:12 [INFO] processing snapshot f3bd9dd4
21:08:12 [INFO] snapshot f3bd9dd4 is ok.
21:08:12 [INFO] processing snapshot 06f2ba6f
21:08:12 [INFO] snapshot 06f2ba6f is ok.
21:08:12 [INFO] processing snapshot 2a9077bd
21:08:12 [INFO] snapshot 2a9077bd is ok.
21:08:12 [INFO] processing snapshot 4cd71ede
21:08:12 [INFO] snapshot 4cd71ede is ok.
21:08:12 [INFO] processing snapshot 2e653f82
21:08:12 [INFO] snapshot 2e653f82 is ok.
21:08:12 [INFO] processing snapshot 1c4183b1
21:08:12 [INFO] snapshot 1c4183b1 is ok.
21:08:12 [INFO] processing snapshot 30c68344
21:08:12 [INFO] snapshot 30c68344 is ok.
21:08:12 [INFO] processing snapshot a9a534ea
21:08:12 [INFO] snapshot a9a534ea is ok.
21:08:12 [INFO] processing snapshot 4863665a
21:08:12 [INFO] snapshot 4863665a is ok.
21:08:12 [INFO] processing snapshot 01ad8824
21:08:12 [INFO] snapshot 01ad8824 is ok.
21:08:12 [INFO] processing snapshot 630654be
21:08:12 [INFO] snapshot 630654be is ok.
21:08:12 [INFO] processing snapshot 64c92204
21:08:12 [INFO] snapshot 64c92204 is ok.
21:08:12 [INFO] processing snapshot 9b681b7a
21:08:12 [INFO] snapshot 9b681b7a is ok.
21:08:12 [INFO] processing snapshot 14ad05ae
21:08:12 [INFO] snapshot 14ad05ae is ok.
21:08:12 [INFO] processing snapshot d928f733
21:08:12 [INFO] snapshot d928f733 is ok.
21:08:12 [INFO] processing snapshot bd4e0601
21:08:12 [INFO] snapshot bd4e0601 is ok.
21:08:12 [INFO] processing snapshot 0ce4b18c
21:08:12 [INFO] snapshot 0ce4b18c is ok.
21:08:12 [INFO] processing snapshot 85d87454
21:08:12 [INFO] snapshot 85d87454 is ok.
21:08:12 [INFO] processing snapshot 0d0a4ea8
21:08:12 [INFO] snapshot 0d0a4ea8 is ok.
21:08:12 [INFO] processing snapshot 88067e78
21:08:12 [INFO] snapshot 88067e78 is ok.
21:08:12 [INFO] processing snapshot 0ded5af3
21:08:12 [INFO] snapshot 0ded5af3 is ok.
21:08:12 [INFO] processing snapshot 52dd597a
21:08:12 [INFO] snapshot 52dd597a is ok.
21:08:12 [INFO] processing snapshot fd2d49cb
21:08:12 [INFO] snapshot fd2d49cb is ok.
21:08:12 [INFO] processing snapshot fe71eb70
21:08:12 [INFO] snapshot fe71eb70 is ok.
21:08:12 [INFO] processing snapshot b0136f42
21:08:12 [INFO] snapshot b0136f42 is ok.
21:08:12 [INFO] processing snapshot caf22158
21:08:12 [INFO] snapshot caf22158 is ok.
21:08:12 [INFO] processing snapshot 5f1d59dd
21:08:12 [INFO] snapshot 5f1d59dd is ok.
21:08:12 [INFO] processing snapshot 5a730578
21:08:12 [INFO] snapshot 5a730578 is ok.
21:08:12 [INFO] processing snapshot fc986511
21:08:12 [INFO] snapshot fc986511 is ok.
21:08:12 [INFO] processing snapshot 6d5b0de7
21:08:12 [INFO] snapshot 6d5b0de7 is ok.
21:08:12 [INFO] processing snapshot 680e0442
21:08:12 [INFO] snapshot 680e0442 is ok.
21:08:12 [INFO] processing snapshot 95a5c470
21:08:12 [INFO] snapshot 95a5c470 is ok.
21:08:12 [INFO] processing snapshot a21df0fb
21:08:12 [INFO] snapshot a21df0fb is ok.
21:08:12 [INFO] processing snapshot 26bb70f5
21:08:12 [INFO] snapshot 26bb70f5 is ok.
21:08:12 [INFO] processing snapshot 00b4803d
21:08:12 [INFO] snapshot 00b4803d is ok.
21:08:12 [INFO] processing snapshot aac77af6
21:08:12 [INFO] snapshot aac77af6 is ok.
21:08:12 [INFO] processing snapshot 981ede21
21:08:12 [INFO] snapshot 981ede21 is ok.
21:08:12 [INFO] processing snapshot 66c7a4c4
21:08:12 [INFO] snapshot 66c7a4c4 is ok.
21:08:12 [INFO] processing snapshot e760d126
21:08:12 [INFO] snapshot e760d126 is ok.
21:08:12 [INFO] processing snapshot 8e17112c
21:08:12 [INFO] snapshot 8e17112c is ok.
21:08:12 [INFO] processing snapshot b39649bb
21:08:12 [INFO] snapshot b39649bb is ok.
21:08:12 [INFO] processing snapshot fb675400
21:08:12 [INFO] snapshot fb675400 is ok.
21:08:12 [INFO] processing snapshot bfeb704f
21:08:12 [INFO] snapshot bfeb704f is ok.
21:08:12 [INFO] processing snapshot 8e3eaf20
21:08:12 [INFO] snapshot 8e3eaf20 is ok.
21:08:12 [INFO] processing snapshot 3a9becd3
21:08:12 [INFO] snapshot 3a9becd3 is ok.
21:08:12 [INFO] processing snapshot 3d928c1d
21:08:12 [INFO] snapshot 3d928c1d is ok.
21:08:12 [INFO] processing snapshot 5fa8f070
21:08:12 [INFO] snapshot 5fa8f070 is ok.
21:08:12 [INFO] processing snapshot 3d21491f
21:08:12 [INFO] snapshot 3d21491f is ok.
21:08:12 [INFO] processing snapshot 54b76ab4
21:08:12 [INFO] snapshot 54b76ab4 is ok.
21:08:12 [INFO] processing snapshot fa5a6260
21:08:12 [INFO] snapshot fa5a6260 is ok.
21:08:12 [INFO] processing snapshot 6f80ea6a
21:08:12 [INFO] snapshot 6f80ea6a is ok.
21:08:12 [INFO] processing snapshot d328e3fd
21:08:12 [INFO] snapshot d328e3fd is ok.
21:08:12 [INFO] processing snapshot 76b6d222
21:08:12 [INFO] snapshot 76b6d222 is ok.
21:08:12 [INFO] processing snapshot c905ec4a
21:08:12 [INFO] snapshot c905ec4a is ok.
21:08:12 [INFO] processing snapshot 3fe3c4ce
21:08:12 [INFO] snapshot 3fe3c4ce is ok.
21:08:12 [INFO] processing snapshot c3f6532f
21:08:12 [INFO] snapshot c3f6532f is ok.
21:08:12 [INFO] processing snapshot bf09dd01
21:08:12 [INFO] snapshot bf09dd01 is ok.
21:08:12 [INFO] processing snapshot 0475c099
21:08:12 [INFO] snapshot 0475c099 is ok.
21:08:12 [INFO] processing snapshot 3f21a5d4
21:08:12 [INFO] snapshot 3f21a5d4 is ok.
21:08:12 [INFO] processing snapshot 2434969b
21:08:12 [INFO] snapshot 2434969b is ok.

However, any changes this command made do not seem to persist. Running the same rustic repair snapshots command again reports and repairs the same problems, no matter how many times I run it. Running rustic check again after repair snapshots reports the same damage it found when I started.

What Next?

What can I try next to recover and repair as much of this repository as possible?

aawsome commented 1 year ago

Thanks for reporting, @liblit

What bothers me most is how your repo could get into this corrupt state, as this should never happen during normal rustic operation. Can you give me more information about what happened? Which kind of repo are you using? Which rustic versions did you run on the repo? Did you run some specific command which turned the repo into this state?

About the repair index command: Seem I need to improve the error handling here. My first guess is that you have some empty packfile (which, of course, shouldn't happen). Can you rerun repair index using --log-level debug?

About the repair snapshots command: This is intended behavior. Unless you specify --delete, only new (repaired) snapshots are saved while the original (broken) ones are still kept. Therefore your subsequent check run still complains about broken files in the original snapshots. You should see some extra snapshots (having the tag repaired) which allow you to see the impact of the repair. But note: If the index doesn't reflect the actual pack files, repair snapshots cannot work properly, so make sure that repair index succeeds before!

liblit commented 1 year ago

Thank you for the speedy and detailed reply, @aawsome!

What bothers me most is how your repo could get into this corrupt state, as this should never happen during normal rustic operation.

You will be relieved to know that Rustic did not cause this corruption. This repository has been built up over years of using Restic. Furthermore, it was originally stored in a cloud storage service (pCloud) with which I have recently had data-integrity problems. This repository has never been modified using Rustic. I only started experimenting with Rustic as a Restic alternative quite recently, to see if it could do a better job of repairing whatever damage Restic + pCloud have caused over the years.

About the repair snapshots command: This is intended behavior.

Ah! Shame on me for not reading the documentation more carefully. Thank you for clarifying. I look forward to using repair snapshots with --delete once we've figure out how to get repair index working.

About the repair index command: Seem I need to improve the error handling here. My first guess is that you have some empty packfile (which, of course, shouldn't happen). Can you rerun repair index using --log-level debug?

Done. There's not much to it:

17:27:27 [INFO] repository .: password is correct.
17:27:27 [INFO] using cache at /home/liblit/.cache/rustic/6f6b61809f47f5987114f22dfe8db5dfb31df2bde574d545cfdbefca6360492b
17:27:28 [DEBUG] (1) rustic::commands::repair: reading pack 49205189...

The pack in question is present and non-empty:

% ls -l data/49/49205189
-rw-r-----. 1 liblit liblit 10698752 Dec 14 11:29 data/49/49205189d56ed447e3dd42ccc574f3721c14bb5c171ab83ba91eeb5c9afbd417
aawsome commented 1 year ago

I just merged #369 which improves the error handling. So it should give better error output by default and ignores pack files in repair index which cannot be processed. It is also available in the latest beta build.

About your pack 49205189...: I fear that this will still not work, most likely this pack file is corrupt. The reason could be a partial file being saved or some kind of bit flip. About both I was already thinking of methods how (try to) to repair (better said: try to salvage as much data as possible) them, but they are not yet implemented. Anyway, please give it a try and report the outcome! If packs are not processed by repair index please try to run a sha256sum on the pack files and check if the result matches the filename!

liblit commented 1 year ago

Using Rustic v0.4.2-5-g53c5fd2, the rustic repair index command consistently reports problems with five packs, no matter how many times I use repair index:

[WARN] error reading pack a2e0e110 (not processed): Header is longer than file? Header size: 1445089623, file size: 8912896
[WARN] error reading pack aebe7dfe (not processed): Header is longer than file? Header size: 1713880441, file size: 9764864
[WARN] error reading pack de494bfa (not processed): Header is longer than file? Header size: 2143406542, file size: 7798784
[WARN] error reading pack 5534aea9 (not processed): Header is longer than file? Header size: 3488278857, file size: 7536640
[WARN] error reading pack 49205189 (not processed): Header is longer than file? Header size: 2266810390, file size: 10698752

For each of these five files, the contents' SHA-256 checksum does not match the name:

% sha256sum data/*/{aebe7dfe,a2e0e110,de494bfa,49205189,5534aea9}*
1264a6e32e305b91c9563c4c08e9ccd3a584f3d2d764387beb18078661d42b9b  data/ae/aebe7dfe7fa6fe6312c229c95627831369676efc52d6e47d3952769fa3301ae2
119122727fe035cc8080661082dfa11ebcdb64aeaa8031162d6109915aaf9101  data/a2/a2e0e11095d3d26271a5d911015334e061b9f03dc506b7e7e3bb02b83ad17254
c5d4b4e6203cfe50ca909f8696a7cd14a5f559a466909fa39523f25058ecee06  data/de/de494bfa7e6f7a8c474e3e493ab19adbb6e71d62564991019d4204d77ee072b2
5f5f9cabef2d3fdd33231366e1c0925f5b2acba12f37388acea1d5272abd2bc1  data/49/49205189d56ed447e3dd42ccc574f3721c14bb5c171ab83ba91eeb5c9afbd417
e791491672a0037a469cae5a35a2230fca88e41c2306573fb641aaf03d28d73b  data/55/5534aea91023728424198b60729aee48d924abbc8924ba2a1ecf332ff85ab954

Presumably these five files are irrecoverably corrupted? If that's the case, should I remove them entirely, then see what else can be partially repaired and recovered? Let's try that and see what happens:

  1. rm data/*/{aebe7dfe,a2e0e110,de494bfa,49205189,5534aea9}*
  2. rustic -r. repair index completed in under one second, producing no warnings or errors.
  3. rustic -r. repair snapshots --delete completed in 46 seconds. It warned about many files with missing contents and created many modified snapshots.
  4. rustic -r. repair snapshots --delete run a second time completed in 43 seconds, producing no warnings or errors.
  5. rustic -r. check completed in 22 seconds, producing no warnings or errors.
  6. rustic -r. check --read-data completed in 20 minutes, producing no warnings or errors.

So it seems that I now have a best-effort recovered repository. Yay! As for the missing data, eh, I'm OK with that. This repository holds automated backups that are unlikely to actually be needed. Of course I want that data if I need it, but if some pieces of it have gaps, that's unlikely to ever matter in practice. I am satisfied with what I got back. In a case like mine, where some file data has been corrupted, I greatly appreciated Rustic's ability to create modified snapshots from whatever subset of files were still available and valid. Nicely done, Rustic! :+1:

I'll close this issue now, since the repository is now recovered to my satisfaction. Thanks you for the hints and help, @aawsome!

aawsome commented 1 year ago

@liblit Thanks for your report! So, yes your packfiles are all corrupt - most likely truncated. To salvage at least some data from truncated pack files, we basically would have to go byte-per-byte through the data and check if the so-far read data is an correctly encrypted and MACed blob (data chunk) - if yes, it can be salvaged and saved to another pack file. But with a very high probability, some blobs are already lost forever.

Good if you can live with the modified snapshots - this is exactly, what the repair snapshots command is for!