php-sync

[mokraemer]

Hi, I'm still trying to use the tool. The fuse part works, performance is good. But I'm havin trouble with the php-sync. I have changed the rsync options to deal with ssh (-e ssh). I enabled Verbose output.

The files (test, test2) are added on "titus"

What I can see is, batches get transfered, first in /mnt/batches/pull/, then the file is synced and the batch is moved to /mnt/batches. That works. So current situation is: /mnt/batches: ./1460643785_titus_titus.fqn.de_3482_00000_norec.batch ./pull ./pull/othello ./pull/titus ./tmp

/mnt/data: ./.sfs.mounted ./test ./test2

But no files are transfered to the destination.

The output from sync is: $ sfs-sync.php -c /etc/sysconfig/sfs/config.php sfs-sync(main)[31472]: Started sfs-sync(batchq)[31473]: Executing command rsync -e ssh -acduhO --remove-source-files --include='./' --include='.batch' --exclude='' titus:/mnt/batches/ /mnt/batches//pull/titus sfs-sync(pull)[31474]: Process batches Array ( [0] => 1460643785_titus_titus.fqn.de_3482_00000_norec.batch ) for node titus sfs-sync(pull)[31474]: Executing command rsync -d --no-r -e ssh -ltpDcuhRO --exclude /.sfs.conf --exclude /.sfs.mounted --delete-missing-args --delete-delay --files-from=/mnt/batches//pull/titus/1460643785_titus_titus.fqn.de_3482_00000_norec.batch titus:/mnt/data/ /mnt/data/ sfs-sync(pull)[31474]: Batches pull Array ( [0] => 1460643785_titus_titus.fqn.de_3482_00000norec.batch ) succeeded sfs-sync(batchq)[31473]: Pull batches from titus succeeded sfs-sync(batchq)[31473]: Executing command rsync -e ssh -acduhO --remove-source-files --include='./' --include='.batch' --exclude='' othello:/mnt/batches/ /mnt/batches//pull/othello sfs-sync(batchq)[31473]: Pull batches from othello succeeded sfs-sync(batchq)[31473]: Executing command rsync -e ssh -acduhO --remove-source-files --include='./' --include='.batch' --exclude='_' titus:/mnt/batches/ /mnt/batches//pull/titus

Am I missing sth? The syn-daemon is a bit more confusing to debug, so if you can point me, where I can debug, this will help me.

[mokraemer]

I think the relevant info from the config: "NODES" => array( "titus" => array("DATA" => "titus:/mnt/data/", "BATCHES" => "titus:/mnt/batches/"), "othello" => array("DATA" => "othello:/mnt/data/", "BATCHES" => "othello:/mnt/batches/")),

[mokraemer]

sometimes batches are synced from the host & transfered to /mnt/batches/pull/titus/ - but they're only processed when I restart the script.

[lucabrunox]

The php-sync daemon should keep syncing the files. Did you solve the problem since you closed the issue?

[mokraemer]

no. must have touched the close button instead of comment.

[lucabrunox]

I see it's working correctly. It tries to pull batches regularly from the remote node "titus", to the local node. What files do you want to be transferred to the destination? What is the "destination"? Can you be a little more clear about the topology?

[mokraemer]

it is a 3 node topology, like in your example. It consits of titus, othello and macbeth macbeth is the superviser running sync titus, othello run sfs (currently only titus) but for the simple transfer this should not matter since I should see a command execution.

[lucabrunox]

You expect the transfer from macbeth to othello to happen but it doesn't? Can you ls -R /mnt/batches ?

[mokraemer]

obove is the output of find (which is similar) but here it is, with a few more transfers: /mnt/batches: 1460643785_titus_titus.fqn.de_3482_00000_norec.batch 1460649827_titus_titus.fqn.de_3482_00000_norec.batch 1460644799_titus_titus.fqn.de_3482_00000_norec.batch 1460650076_titus_titus.fqn.de_3482_00000_norec.batch 1460644807_titus_titus.fqn.de_3482_00000_norec.batch 1460650396_titus_titus.fqn.de_3482_00000_norec.batch 1460644814_titus_titus.fqn.de_3482_00000_norec.batch 1460650422_titus_titus.fqn.de_3482_00000_norec.batch 1460644859_titus_titus.fqn.de_3482_00000_norec.batch pull/ 1460647502_titus_titus.fqn.de_3482_00000_norec.batch tmp/ 1460649245_titus_titus.fqn.de_3482_00000_norec.batch

/mnt/batches/pull: othello/ titus/

/mnt/batches/pull/othello:

/mnt/batches/pull/titus:

/mnt/batches/tmp:

[mokraemer]

I added a few more debugs, I don't see any reception on msg_queue of pushLoop & in scheduleBatches no message of that type is sent - so this is consistent (but not desired).

[lucabrunox]

Please try creating the directory /mnt/batches/push/othello

[lucabrunox]

Sorry, forget that. It should have worked. Do you see a batchq process in your process list? Also

[mokraemer]

that was my thought to, since scheduleBatches will not do anything. But this didn't change anything.

What I'm missing is the place where the batch file is transferred to push-directory.

as of the debug output: sfs-sync(batchq)[2345]: Pull batches from othello succeeded sfs-sync(batchq)[2345]: Executing command rsync -e ssh -acduhO --remove-source-files --include='./' --include='.batch' --exclude='' titus:/mnt/batches/ /mnt/batches//pull/titus sfs-sync(batchq)[2345]: Pull batches from titus succeeded sfs-sync(batchq)[2345]: Executing command rsync -e ssh -acduhO --remove-source-files --include='./' --include='.batch' --exclude='' othello:/mnt/batches/ /mnt/batches//pull/othello

ps output: root 2344 0.0 0.1 259344 24228 pts/0 S+ 18:29 0:00 sfs-sync(sched) root 2345 0.0 0.0 259344 11072 pts/0 S+ 18:29 0:00 sfs-sync(batchq) root 2346 0.0 0.0 259344 11280 pts/0 S+ 18:29 0:00 sfs-sync(pull) root 2347 0.0 0.0 259344 9604 pts/0 S+ 18:29 0:00 sfs-sync(push 0) root 2349 0.0 0.0 259344 9604 pts/0 S+ 18:29 0:00 sfs-sync(push 1) root 2350 0.0 0.0 259344 9604 pts/0 S+ 18:29 0:00 sfs-sync(push 2) root 2351 0.0 0.0 259344 9604 pts/0 S+ 18:29 0:00 sfs-sync(push 3)

[lucabrunox]

The linking should happen in the linkLocalBatches function. Perhaps you can try debugging it? There must be a reason for it not doing anything. Sorry for the inconvenience.

[mokraemer]

ok, I'll check that - thanks for your help so far.

[mokraemer]

looks like the regex in this function is not like all others, and didn't recognize the files. That is at least one problem. I have now one file transfered, batches are now hardlinked in the push-directories.

I'll look for that too. I will create a new merge with my latest changes than :-)

[lucabrunox]

@mokraemer great thanks!