[Request] Auto Cleanup of Remote Files

[Smoz]

In my LFTP script I have it auto remove the directories and files once the transfer is complete. This would be a great option to have available. I use the following built in LFTP feature to do that:

--Remove-source-dirs

It might also be nice to implement the removal of source files in the event some people prefer that option.

--Remove-source-files

[jamauai]

How do you get it to automatically remove source files/dirs when destination files/dirs are removed (ie. via rename script like Filebot)?

[Smoz]

Well for my setup, file moving and renaming isn't done by filebot it done by sonarr/radarr. My LFTP script just removes the source directories/files once they are finished transferring from the remote server to my local server.

So once a download is finished it's hardlinked to a completed directory on the remote server, then my LFTP script fires off and begins transferring, once finished that's when the hardlinks are removed but the original files remain to continue seeding.

Here's my LFTP script for you to take a look at.

#!/bin/bash

# Login name used to access your server
# Password used to access your server
# Hostname of your server; leave off the http://,https://,www. |  ex. vamp.seedtor.com
# Port needed to access your server; this is probably good at 22
# Remote directory; the one on the server where your downloads are
# Local directory; the one on the machine you want the files downloaded to

login=''
pass=''
host=''
port='22'
remote_dir='~/downloads/completed'
local_dir='/mnt/storage/downloads/complete'

# Number of files to download simultaneously
# Number of segments/parts to split the downloads into
# Minimum size each chunk (part) should be

nfile='2'
nsegment='30'
minchunk='1'

# The rest of the script

base_name="$(basename "$0")"
lock_file="/tmp/${base_name}.lock"
echo "${0} Starting at $(date)"
trap "rm -f ${lock_file}" SIGINT SIGTERM
if [ -e "${lock_file}" ]
then
        echo "${base_name} is running already."
        exit
else
    touch "${lock_file}"
    lftp -p "${port}" -u "${login},${pass}" sftp://"${host}" << EOF
    mv "${remote_dir}" "${remote_dir}_lftp"
    mkdir -p "${remote_dir}"
    set ftp:list-options -a
    set sftp:auto-confirm yes
    set pget:min-chunk-size ${minchunk}
    set pget:default-n ${nsegment}
    set mirror:use-pget-n ${nsegment}
    set mirror:parallel-transfer-count ${nfile}
    set mirror:parallel-directories yes
    set xfer:use-temp-file yes
    set xfer:temp-file-name *.lftp    
    mirror -c -v --loop --Remove-source-dirs "${remote_dir}_lftp" "${local_dir}"
    quit
EOF
        rm -f "${lock_file}"
        trap - SIGINT SIGTERM
        echo "${0} Finished at $(date)"
        exit
fi

[ipsingh06]

@Smoz Thanks for the suggestion.

I took a look at the --Remove-source-dirs option in LFTP. There are two main issues:

1. This option was added to LFTP in a recent release (post 2016). The version of LFTP in Ubuntu repos is old (2015). I would rather not add a dependency on a LFTP version.

2. This option only applies to the mirror command. Seedsync also uses the pget command to download lone files. Pget doesn't seem to have a similar option.

For these reasons this feature is not as simple as specifying a flag to LFTP. I'll take a look at the best way to do this in Seedsync, but this feature won't be coming real soon unfortunately.

[Smoz]

@ipsingh06 wow...I cannot believe that the Ubuntu repos still haven't been updated...2 years later lol I understand the issues though. Definitely something to think about, maybe you can pass the original file locations on the remote server back into a bash script to remove the files, something like:

rm -rf {directory}/*

I'm just brainstorming here, I'm not sure what the solution is; one solution would be for the Ubuntu repos to get updated!

[irish8096]

Just tossing out my thoughts. There is already the ability to delete remote, couldn't you just call that when the status goes from 'Downloading' to 'Downloaded'?

[kallama]

+1 to have an option for --Remove-source-files as my workflow is nearly identical to Smoz's.

My Ubuntu 16.04 runs LFTP 4.6.3a which came out in 2015, and it does support --Remove-source-files. I'm not sure why you'd need the dirs option as files will remove directories as well.

[ordinarygulp]

@ipsingh06 either --Remove-source-files or --Move should work on older versions, right?

Also it should be possible to use mirror instead of pget for single files, no?

[gblavl]

+1 --Remove-source-files is essential for my workflow. As @d2dyno mentioned, that option should be available in much older Ubuntu builds than the --Remove-source-dirs switch (which I don't need).

The absence of the ability to remove source files on completed downloads is the only thing keeping me from being able to use Seedsync, which I think is otherwise beautifully implemented. Thank you @ipsingh06 for a wonderful program!

[crzykidd]

To have an option to delete source files would be great. instead of doing it with lftp you could just manually call your delete remote function when the download is complete.

To work around this I have a cron job that runs once a day on my remote server and removes any file older than 2 days. (This gives me enough time in case there is an issue)

[tmchow]

Agree with others. My current script is nearly the same as the above example posted but I use --Move which is equivalent to --Remove-source-dirs. In my script, I do have a comment to myself that folders need to have a trailing slash for my workflow to work correctly as there are cases where i do NOT want the source directory removed due to way i want my mirroring to happen (mirror contents of a set of subfolders, but leave the parent folders intact on source).

The trailing slash comment is in reference to this change: https://github.com/lavv17/lftp/issues/233

I would echo what @gblavl stated above:

The absence of the ability to remove source files on completed downloads is the only thing keeping me from being able to use Seedsync, which I think is otherwise beautifully implemented. Thank you @ipsingh06 for a wonderful program!