Initial target directory missing with SSH backend

[tmannerm]

My initial attempt to do a backup failed with lots of timed out errors because the target directory was missing. Is it really required to precreate the target directory when using rsync:// protocol? Not sure if this is a limitation in duplicty or duply or if this module should just make sure the target directory exists.

I worked around this by adding pre script hook to use sftp to create the target directory.

[tohuwabohu]

From a first look this sounds like a rsync issue.

Depending on what exactly the target directory represents, I would expect this directory being created either as part of the rsync daemon file module setup or by rsync itself when synchronizing the files.

Can you share more information which directory fails to be created? Assuming you have a rsync daemon config like ...

[files]
path = /path/to/rsync

and a rsync config like: rsync -manyflags rsync://hostname:port/foobar.

I presume that rsync failed to create the foobar directory, is that correct?

[linuxmail]

hi,

I have the same problem. Rsync does not create the required folder. So I have to create is manual, before the first run. We use a Synology as Rsync SSH backend.

cu denny

[tohuwabohu]

Ok, thanks for the feedback. Can you perhaps clarify which folder is not created? Given rsync is able to create intermediary directories I somewhat have trouble understand why it fails on the first one ...

[linuxmail]

hi,

we use Synology as backend for Rsync over SSH. So the path is: /volume1/Backup_Linux/<hostname>/

The folder <hostname> is replaced bei Puppet with the real hostname (template) for example "foobar". So the folder "foobar" needs to be created first, until duplicity can upload files. Maybe that's a special problem from Synology rsync, I don't know. I should test it :-)

[punycode]

Since this is a transport related problem, I don't know if this is in scope of this puppet module. We simply fixed this in our Puppet setup by adding a small pre script to the duply profiles:

#!/bin/bash

# Create the target directory on rsync server, when using rsync modules
if echo $TARGET_URL_PROT | grep -q '^rsync://'; then
    # Must use different parameter for rsync server vs. rsync via ssh
    if echo $TARGET_URL_HOSTPATH | grep -q '::'; then
      DESTINATION=${TARGET_URL_USER}@${TARGET_URL_HOSTPATH}
    else
      DESTINATION=${TARGET_URL_PROT}${TARGET_URL_USER}@${TARGET_URL_HOSTPATH}
    fi

    # Set password for rsync if present
    [ ! -z $TARGET_URL_PASS ] && export RSYNC_PASSWORD=$TARGET_URL_PASS

    # Create an empty directory as source
    EMPTY_DIR=`/bin/mktemp -d`

    # Ensure it is there
    /usr/bin/rsync -avz --no-p --no-g --no-o $EMPTY_DIR/ $DESTINATION >/dev/null 2>&1

    /bin/rmdir $EMPTY_DIR
fi

It simply rsyncs an empty directory to the target location. It runs every time backup is running, but it doesn't really hurt, since there is no real data to be transferred.