duplicatibot
commented
3 years ago This issue has been mentioned on Duplicati. There might be relevant details there:
https://forum.duplicati.com/t/ftp-delete-not-atomic-verified-transactional/12936/5
Open ts678 opened 3 years ago
duplicatibot
commented
3 years ago This issue has been mentioned on Duplicati. There might be relevant details there:
https://forum.duplicati.com/t/ftp-delete-not-atomic-verified-transactional/12936/5
duplicatibot
commented
3 years ago This issue has been mentioned on Duplicati. There might be relevant details there:
duplicatibot
commented
2 years ago This issue has been mentioned on Duplicati. There might be relevant details there:
https://forum.duplicati.com/t/restore-from-b2-with-no-local-database/15107/16
Environment info
Description
During a test of
Restore from configuration, I got two errors which I can't find in server log, but profiling log files has them as:I think these are "extra" dindex files left over from a June 20 network glitch, which unfortunately means RemoteOperation table had purged the original time based on log-retention, but sometimes those are unreliable anyway due to transaction rollback. A profiling log is good though, and doesn't roll back. I also have information from the database, so here's an attempt to reconstruct the failure.
Remotevolume table has 386 Verified Blocks volumes and 388 Verified Index volumes now. IndexBlockLink table has 386 rows. They should all be the same number, I think, so this hints there are 2 extra dindex, but which ones? Rather than write SQL, I just searched Remotevolume table for the two dindex named in the errors, and found both ID absent from IndexVolumeID in IndexBlockLink table.
I could probably just delete these seemingly extra dindex manually, or maybe Repair would, but might as well keep looking for cause. Because the files still exist, the
Datafrom anylistOperation in RemoteOperation will show them. Here's their view seen in JSON:and here is selected log history that might explain how two code paths led to two dindex files, or some similar sort of retry confusion. Both files will suffer a seeming network outage on upload (how does upload time out in 0 seconds?!), but actually make it up full size.
When this is eventually seen, the two files are promoted to Uploaded, however I don't think this records them in IndexBlockLink too. Instead, RecreateMissingIndexFile puts up some new ones. This is a known "feature" during manual Repair, and I'm not sure whether or not I had auto-cleanup on. Regardless, the two failed-then-found-uploaded files might be extra files whose dblocks are now gone.
Steps to reproduce
There are lots of reported errors with the same general results, but none of them ever dug deep enough to say if it was this problem.
Some subtleties of how retries need to be handled are in the below issue and pull request. In current issue, its retries were exhausted.
dblock put retry corrupts dindex, using old dblock name for index -- canary regression #3932 Ensure that dindex references correct dblock file after renames #3938
Screenshots
Debug log
debug_log.zip