lsync dies after initial sync

[GoogleCodeExporter]

What steps will reproduce the problem?
1.  Start lysncd
2.  let it fulfill sync and then hang
3.  restart lsyncd, repeat

What is the expected behavior? What does Lsyncd do instead?
I expect it to continuously monitor folders and sync as necessary.  Lsyncd dies 
after initial sync upon invocation.  I am attempting to sync 51655 directories, 
but have bumped my inotify settings accordingly:

fs.inotify.max_user_watches =  750000
fs.inotify.max_queued_events = 750000 

What version of Lsyncd are you using? On what operating system (host and
target(s)?
2.0.5
I have a centos6.0 x86 "proxy" that is syncing an NFSv3 mount to an rsync 
server running Solaris Express 11.  rsync 3.0.6

If applicable please copy/paste your Lsyncd configuration file.

settings = {
        statusFile = "/var/log/lsyncd-vortex.status",
        logfile = "/var/log/lsyncd-vortex.log",
        --statusInterval = 1,
        maxDelays = 1,
}

sync{
        default.rsync,
        source="/vortex/",
        target="e-it-vbr01::vortex",
        rsyncOpts={"-aP", "--ignore-errors"},
}

If applicable please copy/paste a logfile generated with '-log all' of a
minimal testcase that raises this problem.

Please provide any additional information below.

Original issue reported on code.google.com by keir.as...@gmail.com on 17 Oct 2011 at 5:28

[GoogleCodeExporter]

What does "/var/log/lsyncd-vortex.log" say near the end?

BTW  --statusInterval = 1 is invalid syntax, wondering why the Lua compiler 
doesnt hickup there. this should be:
statusInterval = 1,

Original comment by axk...@gmail.com on 17 Oct 2011 at 5:52

[GoogleCodeExporter]

Thanks for the quick response.  Doesn't "--" denote comments?  Sorry for my 
ignorance.

dispatchers/send_to_Waseca/3759303/
rsync: chown "/dispatchers/send_to_Waseca/3759303" (in vortex) failed: Invalid a
rgument (22)
dispatchers/send_to_Waseca/3759959/
rsync: chown "/dispatchers/send_to_Waseca/3759959" (in vortex) failed: Invalid a
rgument (22)
dispatchers/send_to_Waseca/3763974/
rsync: chown "/dispatchers/send_to_Waseca/3763974" (in vortex) failed: Invalid a
rgument (22)
dispatchers/send_to_Waseca/444666/
rsync: chown "/dispatchers/send_to_Waseca/444666" (in vortex) failed: Invalid ar
gument (22)
dispatchers/send_to_Waseca/714004/
rsync: chown "/dispatchers/send_to_Waseca/714004" (in vortex) failed: Invalid ar
gument (22)
dispatchers/send_to_Waseca/991188/
rsync: chown "/dispatchers/send_to_Waseca/991188" (in vortex) failed: Invalid ar
gument (22)
rsync: chown "/platedecoration/Thumbs.db" (in vortex) failed: Invalid argument (
22)
rsync: chown "/platedecoration/Thumbs.db:encryptable" (in vortex) failed: Invali
d argument (22)
workflowsIO/MassTransit_Received/
workflowsIO/MassTransit_Received/3777807/
workflowsIO/MassTransit_Received/3817191/
workflowsIO/MassTransit_Received/3867254/
workflowsIO/MassTransit_Received/4069820/
workflowsIO/MassTransit_Received/4078926/
workflowsIO/Optimization_OUT/
workflowsIO/Optimization_OUT/048_LBDLRWFall11 (Back Cover).pdf
     8864442 100%   25.92MB/s    0:00:00 (xfer#247, to-check=1136/250022)

sent 634543112 bytes  received 273259 bytes  3280704.76 bytes/sec
total size is 801514699885  speedup is 1262.59
rsync error: some files/attrs were not transferred (see previous errors) (code 2
3) at main.c(1039) [sender=3.0.6]
Mon Oct 17 13:34:59 2011 Normal: Startup of '/vortex/' finished.  

Original comment by keir.as...@gmail.com on 17 Oct 2011 at 6:00

[GoogleCodeExporter]

Oh yes it does, sorry forgot that, was doing too much javascript lately. Its 
valid Lua syntax. My fault :-)

Lsyncd fails because the startup wasn't clean since you seem to miss some 
permissions for some files/dirs on the target. This is a new feature of Lsyncd 
2.0.5. Either fix these errors of rsyncs' failed chowns or set 'settings.insist 
= 1' so Lsyncd will ignore errors on startup.

Original comment by axk...@gmail.com on 17 Oct 2011 at 6:30

[GoogleCodeExporter]

And the 'insist' setting here?:
settings = {
        statusFile = "/var/log/lsyncd-vortex.status",
        logfile = "/var/log/lsyncd-vortex.log",
        --statusInterval = 1,
        maxDelays = 1,
        insist = 1,
}

Original comment by keir.as...@gmail.com on 17 Oct 2011 at 6:47

[GoogleCodeExporter]

yes

Original comment by axk...@gmail.com on 17 Oct 2011 at 7:13

[GoogleCodeExporter]

It still doesn't appear to be 're-syncing':

     3438815 100%    5.71MB/s    0:00:00 (xfer#296, to-check=1024/248463)
workflowsIO/MassTransit_Received/3867254/
workflowsIO/MassTransit_Received/4036068/
workflowsIO/MassTransit_Received/4079087/
workflowsIO/Optimization_OUT/

sent 3228099392 bytes  received 61640 bytes  7586747.43 bytes/sec
total size is 804737167697  speedup is 249.29
rsync error: some files/attrs were not transferred (see previous errors) (code 2
3) at main.c(1039) [sender=3.0.6]
Mon Oct 17 14:43:29 2011 Normal: Startup of '/vortex/' finished.

Thanks for your continued help!

Original comment by keir.as...@gmail.com on 17 Oct 2011 at 8:43

[GoogleCodeExporter]

As insist says, Lsyncd will with it turned on will just ignore these startup 
errors and keep running. I suppose you are missing the necessary rights to 
write these files on the target. Either the user you use is not allowed to, or 
these files have the o+rw permissions not set (that is, even the owner may not 
write them!) 

Original comment by axk...@gmail.com on 17 Oct 2011 at 9:46

[GoogleCodeExporter]

I've reduced my test to a folder with 46 file and it still won't sync without 
restart.  There doesn't appear to be permission issues.  Here is the config:

settings = {
        statusFile = "/var/log/lsyncd-BrownCTP.status",
        logfile = "/var/log/lsyncd-BrownCTP.log",
        --statusInterval = 0,
        maxDelays = 0,
        insist = 1,
}

sync{
        default.rsync,
        source="/vortex/BrownCTP",
        target="/backup-vortex/BrownCTP",
        rsyncOpts="-aP","--ignore-errors",
}

Last few lines of logs:

backup-vortex/BrownCTP/
sending incremental file list
./
keir-test/

sent 1077 bytes  received 34 bytes  2222.00 bytes/sec
total size is 145314949  speedup is 130796.53
Tue Oct 18 08:43:10 2011 Normal: Startup of '/vortex/BrownCTP/' finished.

Original comment by keir.as...@gmail.com on 18 Oct 2011 at 1:46

[GoogleCodeExporter]

The last log looks by itself good? Is the problems it doesn't react on changes? 

You shouldn't add -a to rsyncOpts since it implies recursive and interfers with 
Lsyncds rsync calling logic. I've never tested -P as well. You are missing -s 
though which is needed for files with spaces in them. 

Since rsyncOpts kinda suggests you can change it the way you like, but some 
settings break Lsyncd, I plan to remove it from Lsyncd 2.1 :-)

Save settings are for exampel:
-v  
-u
-l
-o
-g
-t (required when doing -u)
-T (required for master-master)
--numeric-ids
-p
-n
-z
-4
-6

Settings that can break Lsyncd are:
-r (since Lsyncd makes any recursive Logic itself)
-a (since it implies -r)
-R (since it ruins all the paths)
--from0 (since Lsyncd uses that to transfer to rsync)

Default is '-ltus'

If changing it to ltus or something similar doesn't help please call lsyncd 
with "-log all" and post or email me the log.

Thank you for hanging in, Axel

Original comment by axk...@gmail.com on 18 Oct 2011 at 2:04

[GoogleCodeExporter]

I have the defaults run and no luck:
--lsyncd conf file

settings = {
        statusFile = "/var/log/lsyncd-BrownCTP.status",
        logfile = "/var/log/lsyncd-BrownCTP.log",
        --statusInterval = 0,
        maxDelays = 0,
        insist = 1,
}

sync{
        default.rsync,
        source="/vortex/BrownCTP",
        target="/backup-vortex/BrownCTP",
        --rsyncOpts="-ltus",
}

I am attaching the log as requested.  Thank you!

Original comment by keir.as...@gmail.com on 18 Oct 2011 at 3:14

Attachments:

• lsyncd-log.txt

[GoogleCodeExporter]

Can you explify what you mean with "no luck"? What did you do, what did you 
expect to happen and what happens instead?

Original comment by axk...@gmail.com on 18 Oct 2011 at 3:46

[GoogleCodeExporter]

I observed the same results (failing to sync automatically) by commenting out 
the rsyncOpts.  That assumes -ltus, correct?  Nothing happens.  

Original comment by keir.as...@gmail.com on 18 Oct 2011 at 4:05

[GoogleCodeExporter]

You mean in your log before, you touched/changed a file in /vortex/BrownCTP 
after 10:10:21, but before you shutted down lsyncd?

Note that there is a 15 seconds delay...

Original comment by axk...@gmail.com on 18 Oct 2011 at 4:33

[GoogleCodeExporter]

After lsyncd initial completes its sync after restart, I add/change 
files/directories to the source.  They do not sync to the destination, ever, 
unless I kill and restart lsyncd or reboot. 

Original comment by keir.as...@gmail.com on 18 Oct 2011 at 5:12

[GoogleCodeExporter]

In your log there never appears an inotifyEvent(), the kernel never reports to 
lsyncd that something has changed, so there is something seriously amiss. I 
don't know. Maybe try to install inotify-tools and make sure that 
"inotify-wait" works as it should.

Original comment by axk...@gmail.com on 18 Oct 2011 at 5:22

[GoogleCodeExporter]

Oh just had a flash.Is /vortex/BrownCTP by chance a network mounted filesystem? 
If yes, then it cannot work. Lsyncd needs to run on the systems the filesystem 
is locally. Networked filesystems do not normally support notification 
mechanisms like inotify. Not Lsyncd's fault, but network-filesystem-designers.

Original comment by axk...@gmail.com on 18 Oct 2011 at 9:55

[GoogleCodeExporter]

BINGO.  Unfortunately, it is indeed a NFS mount.  Thanks for all your help 
trying to get this working for my situation.  

Original comment by keir.as...@gmail.com on 18 Oct 2011 at 10:21

[GoogleCodeExporter]

You are welcome :-)

Original comment by axk...@gmail.com on 18 Oct 2011 at 10:40

• Changed state: Done