--exclude

[tkd4444]

Hey @martymac

Is the --exclude option supported in file mode ?

Thanks so much !

[martymac]

Hi David,

Yes, it should be :

$ mkdir /tmp/{src,dst}                                                                                    
$ touch /tmp/src/{foo,bar,baz}                                                                                 
$ fpsync -o '-lptgoD -v --numeric-ids --exclude=bar'  /tmp/src /tmp/dst                                        
$ ls /tmp/dst                                                                                                  
baz foo

[tkd4444]

Yeah, just tried that but it still syncs that folder. I think it is worth to mention that I'm trying to exclude a folder and not a file.

sudo "PATH=$PATH" /usr/local/bin/fpsync -o '-lptgoD -v --numeric-ids --exclude=cache' \
`pwd`/source/ `pwd`/destination/

It still copies over the cache folder which I'm trying to exclude.

[martymac]

Right, this is because fpsync calls fpart which, by default, generates a list of files to synchronize. Thus, excluding a directory at rsync level cannot work because it will always create a parent directory even if it is excluded :

$ mkdir /tmp/{src,dst}
$ mkdir /tmp/src/{foo,bar,baz}
$ touch /tmp/src/{foo,bar,baz}/file
$ fpsync -o '-lptgoD -v --numeric-ids --exclude=bar'  /tmp/src /tmp/dst
$ ls /tmp/dst/bar
file

Here, rsync created the 'bar' directory because it was asked to synchronize bar/file.

You have to exclude the directory earlier by using fpart options :

$ fpsync -O '-x bar' /tmp/src /tmp/dst
$ ls /tmp/dst
baz foo

That way, bar/file will not appear at all in the (internal) file list to synchronize.

[tkd4444]

@martymac thanks for your assistance on this. Will it accept path in the -x arg ? As it's now it's excluding everything from /tmp/src with a name bar e.g. fpsync -O '-x /bar' /tmp/src /tmp/dst

Thanks !

[martymac]

This is the intended behaviour. Unfortunately, there is currently no way to exclude a specific path from fpart as -x only acts on file or directory names (not paths, but that could be an improvement).

Coming back to my first answer, rsync accepts relative paths in its --exclude option, so you may (this is to be tested) be able to exclude all files from that directory with something like : --exclude=bar/*

Note that fpart will crawl that directory and add its files to the produced partitions anyway.

Best regards, Ganael.

[martymac]

Hello David,

I've implemented ability to specify paths with options '-y', '-Y', '-x' and '-X', that should solve your problem. An exclusion will be considered to be a path when it contains at leat a '/' character. You can mix file names and paths and inclusion/exclusion rules remain the same as before.

Hope this helps,

Best regards,

Ganael.

[tkd4444]

Hello Ganael,

Thanks for spending time on this ! Much appreciated.

All the best ! David

[tkd4444]

Hey @martymac Downloaded and installed the v1.1.0

I just tried: sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X /cache' /tmp/src/ /tmp/dest/

and nothing really changed. It's still copying over the entire cache folder.

Thanks !

[martymac]

Hi David,

The fix is not in 1.1.0, but in git only (master branch) ; it will be integrated in the next release. You can rebuild fpart from source to get it now if necessary.

Best regards,

Ganael.

[tkd4444]

I tried to use the master and compiled it but it still copies over the directory.

Used:

sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X /cache' /tmp/src/ /tmp/dest/

$ fpsync
fpsync v1.2.0 - Sync directories in parallel using fpart
Copyright (c) 2014-2019 Ganael LAPLANCHE <ganael.laplanche@martymac.org>

[martymac]

Hi David,

Excluded path must patch the root path of the crawled filesystem. In your case, you should exclude the following path : '/tmp/src/cache'. Can you try that ?

[tkd4444]

Hey Ganael,

Nope, tried it and it still copies it over.

sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X '`pwd`'/src/cache' `pwd`/src/ `pwd`/trg/

[martymac]

Hi,

are you sure fpsync uses the latest version of fpart you compiled and not a previous version of the binary you installed before ?

[tkd4444]

Hey,

Yes, looks like it.

$ sudo /usr/local/bin/fpsync -v
fpsync v1.2.0 - Sync directories in parallel using fpart
Copyright (c) 2014-2019 Ganael LAPLANCHE <ganael.laplanche@martymac.org>
WWW: http://contribs.martymac.org
Usage: /usr/local/bin/fpsync [OPTIONS...] src_dir/ dst_url/
Usage: /usr/local/bin/fpsync [-r jobname] [OPTIONS...]

[martymac]

Hello,

I mean, check that the fpsync script itself uses the latest version of fpart. That latest version should be in your $PATH; it is determined that way by the script :

FPART_BIN="$(which fpart)"

Best regards,

Ganael.

[tkd4444]

Yes it is.

fpart v1.2.0
Copyright (c) 2011-2019 Ganael LAPLANCHE <ganael.laplanche@martymac.org>
WWW: http://contribs.martymac.org
Build options: debug=no, fts=system

[martymac]

OK, I see.

fpart exclusions, when used through fpsync, are relative to the src path (because fpsync changed its current working directory to the src dir), so you should use the following :

sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X ./cache'pwd/src/pwd/trg/

Remember you can build fpart with the --enable-debug option to get feedback about included/excluded files and directories.

[tkd4444]

Hey Ganael,

This did the trick:

'-X ./cache'

Thanks so much for your assistance !