WMP
commented
9 years ago Open WMP opened 9 years ago
WMP
commented
9 years ago 
mpalmer
commented
9 years ago I'm not sure what's going on here -- my initial impression is that this output appears to be missing the remote-side output. Assuming you're using the latest released version of lvmsync (3.2.0) on the sender side, all output from the remote should be displayed prefixed by remote:, and there shouldn't be a case where the apply fails without producing some output -- even if it's just saying "lvmsync not found", "SSH: Permission denied (publickey)", or some other system-level error message.
So, yeah, mystery. It may be that you've found a corner-case I hadn't considered, that's failing the receiver silently, or perhaps gnomes are eating your errors.
A couple of things to try, to rule out fundamental problems:
lvmsync --version locally, and make sure that's showing 3.2.0.ssh <remote> lvmsync --version and make sure that's showing 3.2.0.That makes sure that lvmsync is OK on both ends, and can talk to each other. If that's all OK, I'll probably have to make a few lvmsync release with more debugging output on apply, to see why it's failing.
WMP
commented
9 years ago root@monica ~ # lvmsync --version
lvmsync 3.2.0
root@monica ~ # ssh backup lvmsync --version
lvmsync 3.2.0
root@monica ~ #Maybe is problem with path? lvmsync execute remote commands from bash?
mpalmer
commented
9 years ago On the receiver side, lvmsync does very little -- it opens the destination file and writes chunks where its told. There's not a lot else that it does.
I'll work on releasing a new version of lvmsync with improved debugging capabilities, which should show up what's going on. Look for it by the end of the weekend.
mpalmer
commented
9 years ago OK, version 3.3.0 is out, and now supports --debug. If you could upgrade both sides, then do a run with that option enabled, you should see a whole lot of lines starting with remote: talking about where data blocks are being written to. Hopefully that should shed some light on where things are dying.
WMP
commented
9 years ago root@monica ~ # lvmsync --version
lvmsync 3.3.0
root@monica ~ # ssh backup lvmsync --version
lvmsync 3.3.0
root@monica ~ # bash -x lvmbackup.sh -s backup -v vg0 -o virtual mysql-disk
+ server=
+ virtualgroup_server=
+ virtualgroup_original=
+ getopts “s:v:o:h” option
+ case $option in
+ server=backup
+ getopts “s:v:o:h” option
+ case $option in
+ virtualgroup_server=vg0
+ getopts “s:v:o:h” option
+ case $option in
+ virtualgroup_original=virtual
+ getopts “s:v:o:h” option
+ shift 6
+ [[ -z backup ]]
+ [[ -z vg0 ]]
+ [[ -z virtual ]]
+ [[ 1 -lt 1 ]]
+ (( 1 ))
+ echo mysql-disk:
mysql-disk:
+ [[ ! -e /dev/virtual/mysql-disk ]]
++ ssh backup 'test -e /dev/vg0/mysql-disk && echo 1 || echo 0'
+ FileExists=1
++ lvdisplay --units m virtual/mysql-disk
++ awk '{print $3}'
++ sed -e 's/\..*//'
++ grep 'LV Size'
+ size=51200
+ snapshot_size=5120
+ lvcreate --snapshot -L 5120m -n mysql-disk-snapshot virtual/mysql-disk
Logical volume "mysql-disk-snapshot" created
+ [[ 1 -eq 0 ]]
+ echo lvmsync...
lvmsync...
++ date +%Y-%m-%d-%H:%M
+ lvmsync --debug /dev/virtual/mysql-disk-snapshot backup:/dev/vg0/mysql-disk --snapback /snapback/mysql-disk/2014-11-07-11:44
Data source: /dev/mapper/virtual-mysql--disk
Sending chunk 0..4095...
Sending chunk 590241792..590245887...
Sending chunk 25904021504..25904025599...
Sending chunk 25908326400..25908330495...
Sending chunk 25908330496..25908334591...
Sending chunk 25908334592..25908338687...
Sending chunk 563085312..563089407...
Sending chunk 563089408..563093503...
Sending chunk 563093504..563097599...
Sending chunk 563097600..563101695...
Sending chunk 563101696..563105791...
Sending chunk 563105792..563109887...
Sending chunk 563109888..563113983...
Sending chunk 563113984..563118079...
Sending chunk 563118080..563122175...
Sending chunk 563122176..563126271...
Sending chunk 563126272..563130367...
Sending chunk 563130368..563134463...
Sending chunk 563134464..563138559...
Sending chunk 563138560..563142655...
Sending chunk 563142656..563146751...
Sending chunk 563146752..563150847...
Sending chunk 563150848..563154943...
Sending chunk 563154944..563159039...
Sending chunk 563159040..563163135...
Sending chunk 563163136..563167231...
Sending chunk 563167232..563171327...
Sending chunk 563171328..563175423...
Sending chunk 563175424..563179519...
Sending chunk 563179520..563183615...
Sending chunk 563183616..563187711...
Sending chunk 563187712..563191807...
Sending chunk 563191808..563195903...
Sending chunk 563195904..563199999...
Sending chunk 563200000..563204095...
Sending chunk 563204096..563208191...
Sending chunk 563208192..563212287...
Sending chunk 563212288..563216383...
Sending chunk 563216384..563220479...
Sending chunk 563220480..563224575...
Sending chunk 563224576..563228671...
Sending chunk 563228672..563232767...
Sending chunk 563232768..563236863...
Sending chunk 563236864..563240959...
Sending chunk 563240960..563245055...
Sending chunk 563245056..563249151...
Sending chunk 563249152..563253247...
Sending chunk 563253248..563257343...
Sending chunk 563257344..563261439...
Sending chunk 563261440..563265535...
Sending chunk 50 of 115, 0.13MB/sSending chunk 563265536..563269631...
Sending chunk 563269632..563273727...
Sending chunk 563273728..563277823...
Sending chunk 563277824..563281919...
Sending chunk 563281920..563286015...
Sending chunk 563286016..563290111...
Sending chunk 563290112..563294207...
Sending chunk 563294208..563298303...
Sending chunk 563298304..563302399...
Sending chunk 563302400..563306495...
Sending chunk 563306496..563310591...
Sending chunk 563310592..563314687...
Sending chunk 563314688..563318783...
Sending chunk 563318784..563322879...
Sending chunk 563322880..563326975...
Sending chunk 563326976..563331071...
Sending chunk 563331072..563335167...
Sending chunk 563335168..563339263...
Sending chunk 563339264..563343359...
Sending chunk 563343360..563347455...
Sending chunk 563347456..563351551...
Sending chunk 563351552..563355647...
Sending chunk 563355648..563359743...
Sending chunk 563359744..563363839...
Sending chunk 563363840..563367935...
Sending chunk 563367936..563372031...
Sending chunk 563372032..563376127...
Sending chunk 563376128..563380223...
Sending chunk 563380224..563384319...
Sending chunk 563384320..563388415...
Sending chunk 563388416..563392511...
Sending chunk 563392512..563396607...
Sending chunk 563396608..563400703...
Sending chunk 563400704..563404799...
Sending chunk 563404800..563408895...
Sending chunk 563408896..563412991...
Sending chunk 563412992..563417087...
Sending chunk 563417088..563421183...
Sending chunk 563421184..563425279...
Sending chunk 563425280..563429375...
Sending chunk 563429376..563433471...
Sending chunk 563433472..563437567...
Sending chunk 563437568..563441663...
Sending chunk 563441664..563445759...
Sending chunk 563445760..563449855...
Sending chunk 563449856..563453951...
Sending chunk 563453952..563458047...
Sending chunk 563458048..563462143...
Sending chunk 563462144..563466239...
Sending chunk 563466240..563470335...
Sending chunk 563470336..563474431...
Sending chunk 563474432..563478527...
Sending chunk 25908338688..25908342783...
Sending chunk 25908342784..25908346879...
Sending chunk 25908346880..25908350975...
Sending chunk 25908350976..25908355071...
Sending chunk 25908355072..25908359167...
Sending chunk 25908359168..25908363263...
Sending chunk 25908363264..25908367359...
Sending chunk 25908367360..25908371455...
Sending chunk 25908371456..25908375551...
Sending chunk 568803328..568807423...
Sending chunk 568807424..568811519...
Sending chunk 568811520..568815615...
Sending chunk 568815616..568819711...
Transferred 471040 bytes in 1.54 seconds
You transferred your changes 113975.65x faster than a full dd!
FATAL ERROR: APPLY FAILED.
+ echo 'lvmsync done'
lvmsync done
+ echo cleanup
cleanup
+ lvremove -f /dev/virtual/mysql-disk-snapshot
Logical volume "mysql-disk-snapshot" successfully removed
+ shift
+ (( 0 ))
root@monica ~ #Maybe this is problem?
root@monica ~ # ssh backup which lvmsync
/usr/local/bin/lvmsync
root@monica ~ # ssh backup cat `which lvmsync`
#!/usr/bin/env bash
# load rvm ruby
source /usr/local/rvm/environments/ruby-2.0.0-p576
ruby /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync $@
root@monica ~ #Ok, i thinks i found error:
root@backup ~ # cat /usr/local/bin/lvmsync
#!/usr/bin/env bash
# load rvm ruby
source /usr/local/rvm/environments/ruby-2.0.0-p576
echo $@ >> /var/log/lvmsync.log
eval 'ruby /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync $@' 2>&1 | tee -a "/var/log/lvmsync.log"
root@backup ~ #
root@monica ~ # bash lvmbackup.sh -s backup -v vg0 -o virtual mysql-disk
mysql-disk:
Logical volume "mysql-disk-snapshot" created
lvmsync...
Data source: /dev/mapper/virtual-mysql--disk
Sending chunk 0..4095...
Transferred 4096 bytes in 0.04 seconds
You transferred your changes 13107200.00x faster than a full dd!
lvmsync done
cleanup
Logical volume "mysql-disk-snapshot" successfully removed
root@monica ~ #
root@backup ~ # cat /var/log/lvmsync.log
--apply - --snapback /snapback/mysql-disk/2014-11-07-12:06 -v -d /dev/vg0/mysql-disk
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:129:in `initialize': No such file or directory - /snapback/mysql-disk/2014-11-07-12:06 (Errno::ENOENT)
from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:129:in `open'
from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:129:in `run_apply'
from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:96:in `main'
from /usr/local/rvm/gems/ruby-2.0.0-p576/gems/lvmsync-3.3.0/bin/lvmsync:357:in `<top (required)>'
from /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync:23:in `load'
from /usr/local/rvm/gems/ruby-2.0.0-p576/bin/lvmsync:23:in `<main>'
--apply - --snapback /snapback/mysql-disk/2014-11-07-12:07 -v -d /dev/vg0/mysql-disk
Writing changed data to "/dev/vg0/mysql-disk"
Seeking to 0
Wrote 4096 bytes at 0
--apply - --snapback /snapback/mysql-disk/2014-11-07-12:08 -v -d /dev/vg0/mysql-disk
Writing changed data to "/dev/vg0/mysql-disk"
Seeking to 0
Wrote 4096 bytes at 0
root@backup ~ # You must fix --verbose and --debug ;)
Jasperswaagman
commented
6 years ago If anyone else stumbles into this. Make sure the directory you want to store the snapback in exists on the server/remote. If it does not exist it will give a:
FATAL ERROR: APPLY FAILED.