search

AmesCornish / buttersink

Buttersink is like rsync for btrfs snapshots
GNU General Public License v3.0
193 stars 18 forks source link

Partial snapshot updates? #64

Closed Gooberpatrol66 closed 5 years ago

Gooberpatrol66 commented 6 years ago

I'm trying to synchronize snapshots, but my network is too unreliable to finish it.

At subvol /mnt/store/.snapshots/10/snapshot
ERROR: failed to read stream from kernel: Broken pipe

S|btrfs receive errors
At subvol snapshot
ERROR: writing to Libraries/Videos/youtube/Filly_Story___The_Neigh_Back_Home__No_BGM__hd720.mp4 failed: No space left on device
  ERROR: {u'traceback': u'  File "/usr/lib64/python2.7/site-packages/buttersink/Butter.py", line 193, in write', u'errorType': u'IOError', u'command': u'write', u'server': True, u'error': u'[Errno 32] Broken pipe'}.

When I run the command again, it starts over completely. Is it possible to resume?

Gooberpatrol66 commented 6 years ago

Screwed up the post.

eugene-bright commented 6 years ago

No space left on device - is a badly btrfs bugs symptom. Kernel update and btrfs filesystem balancing are recommended https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space

Gooberpatrol66 commented 6 years ago

I don't think it's bugged, it's just full of partial snapshots from retrying this. The --delete option doesn't work either btw

eugene-bright commented 6 years ago

Talking in respect of btrfs snapshot transfer resuming I'm almost sure that it's impossible. As I know btrfs send stream consists of encoded commands such as "create new file", "append this", "replace that". btrfs receive just does all this things from userspace as a normal application (doing only minor actions with ioctl calls: such as setting received UUID of volume and marking it read only). So, there is no idempotence. Double appending would break the things. Also I'm not aware if btrfs keep track of stream applying. If the number of last successfully applied command had been known it would have been possible to skip the part of stream next time. But it's a theory the implementation of which is not known to me. For now you can manually store btrfs send stream as a local file, rsync it to another host and then apply it on the remote machine.

eugene-bright commented 6 years ago

For the full history please attach uname -a output to this issue.

eugene-bright commented 6 years ago

And one more thing. Buttersink in a promising tool. But it's currently practically unmaintained. I somehow tuned my own fork for my tasks. Btrfs quota performance make current algorithm of btrfs <-> btrfs transferring impractical. See #62. We need egg-headed hero to refactor buttersink massively to work it around.

AmesCornish commented 5 years ago

As Eugene commented, we would need btrfs send and receive to support resuming, which they currently don't.