search

mpalmer / lvmsync

Synchronise LVM LVs across a network by sending only snapshotted changes
http://theshed.hezmatt.org/lvmsync
GNU General Public License v3.0
382 stars 61 forks source link

Allow lvmsync to operate locally #1

Closed borgstrom closed 11 years ago

borgstrom commented 12 years ago

This adds a new --local switch that makes lvmsync operate without ssh on a snapshot and LV that are local to the host.

Thanks for writing this! Made a recent migration painless. :)

ProTip commented 12 years ago

I prefer to write stuff such that localhost:/dev/blahblah/ gets run locally. Or not specifying a host, same result. My 2 cents.

borgstrom commented 12 years ago

Sure, localhost:/dev/blahblah works, but it adds the overhead of an unnecessary hop through SSH for the data to pass through.

As for not specifying a host, IMHO explicit is better than implicit so I'd much rather be very aware that I'm operating in a mode that is intended for local use when all of the interactive help talks about SSH for the default mode.

ProTip commented 12 years ago

I believe I may not have communicated that properly. I have been writing my own classes so that specifying 'localhost' for the host will cause system commands to be run locally(i.e. no ssh). So, 'localhost:/dev/whatever' would, in this case, not use ssh for the transfer. However, I've also been making host an optional argument so that if it's not specified it is assumed your meaning local( think scp and rsync).

borgstrom commented 12 years ago

I still feel that explicit is better than implicit here and magically removing ssh from the equation when you specify "localhost:/dev/blahblah" seems to me like a idea that's bound to cause confusion down the road.

lvmsync was written for sync'ing via SSH and I don't think that the default mode should be changed from that mind set. I still believe that "--local" is the most succinct way to denote that we want to operate in a way that the destination device is local.

I guess it's really up to @mpalmer though :)

ProTip commented 12 years ago

I almost always side with explicit over implicit. I like my languages strongly typed! Though, I also like Ruby. Go figure. Anyway, device mapper is a linux thing and in the linux world '[user@host:]path' is the usual idiom for specifying a resource location where the resource can be local or remote.

The default mode is SSH because it's a simple tool and hasn't been expanded to choose the best method for handling the data transfer. SSH is used for network security; no reason to use it locally.

That's my 2 cent. Open source + github = it's up to YOU. Just a fork away from your own implementation;)

mpalmer commented 12 years ago

Clearly we've got a couple of different ideas circulating here. Here's what I'd like to see implemented (and which I'll eventually implement if nobody else gets around to it):

  1. Make SSH optional:
    • If you use the <hostname>:<path> destination syntax (even if <hostname> is localhost), then SSH+lvmsync will be invoked to transfer the data.
    • If, on the other hand, your path doesn't have a colon in it, then lvmsync will assume the path you specify is a block device (or image) that you want to apply changes to, and will do so.
  2. Allow the changes to be stored in a file, with a new command-line option (I'm thinking --dump perhaps?), in which case the contents of the destination file will be the same data that would have been sent over the network to perform the same sync, and which can be applied using lvmsync --apply.

Does that cover everyone's use cases?

ProTip commented 12 years ago

Very shortly I'm going to need the ability to rate-limit the tool so I can control the impact on disk/network IO. But, maybe that should be submitted as another issue?

mpalmer commented 12 years ago

Yes, that should definitely be submitted as an entirely separate issue. This one's gotten big and hairy enough already.

shaneau commented 12 years ago

@mpalmer I would love to see those two ideas implemented. I'm looking to use the script to do a local sync of a snapshot to an image on a USB HD.

phase5 commented 11 years ago

Hi Guys,

Pretty old thread but still looks open so thought I would add my comment..

Consider the situation where iSCSI is being used to create a local device for the destination of lvmsync. (Going over SSH not an option)

It is very common to use the symbolic links in /dev/disk/by-path/ when working with iSCSI because its the only reasonable way to identify a particular Target/LUN when more than one is connected on the local system.

The problem here is that the names of these symlinks always contain a colon, which prompts lvmsync into trying to connect over SSH. (which promptly fails, obviously)

Here is an example local destination path that I need to pass to lvmsync:

/dev/disk/by-path/ip-10.0.0.52:3260-iscsi-iqn.localnet:target250-lun-1-part1

I think in this case (Which I am sure is reasonably common!) being able to explicitly specify that the supplied destination is local (so it treats the colon as literal) would be useful.

I may be wrong - just thought I would point it out.

mpalmer commented 11 years ago

As a side effect of some other changes I've been making, this use-case should be possible -- you'll be able to specify that you're dumping to a file (or stdout) and then reading from stdin, so no colon-interpretation required. Keep an eye out for pushes (the code's mostly complete, I'm just testing the bejesus out of it).

mpalmer commented 11 years ago

OK, I've just pushed up the changes I've been making to lvmsync. You can now sync locally to a device with colons in it by the not-so-trivial expedient of doing a "dump to stdout" and then applying from there. It ain't pretty, but it works. Closing this issue off.