fpsync does not provide a way to specify number of file parts

[ispringer]

I'm trying to use fpsync in rsync mode to copy a MongoDB data dir that's several terabytes from one system to another. I want to use 4 parallel jobs, since we have found that is optimal. But I also want to use 4 file parts, so that all rsync processes get kicked off at the start of the sync. The reason for this is that the overall sync takes over 3 hours, but our 2FA ssh sessions between the 2 systems have a 1 hour TTL. We don't want new rsync calls to be happening throughout the sync, otherwise the operator will have to babysit it the entire time and re-enter their 2FA code when prompted.

This is what I tried:

fpsync -v -n 4 -o "-e \\\"ssh -q -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR\\\"" -O "-n|4" /data/mongodb/ t1m1r1:/data/mongodb/

but it fails with:

Option -n is incompatible with options -f, -s and -L.
Usage: fpart [OPTIONS] -n num | -f files | -s size [FILE or DIR...]

I messed around with modifying fpsync to not pass -f/-s/-L to fpart if -n was in the specified fpart options, but fpart/fpsync failed mysteriously.

Is there a way to make this work?

Thanks!

[martymac]

Hello Ian,

Fpart's live mode (option -L) is what is used by fpsync to be able to produce partitions while rsync'ing them. That mode is only compatible with options -f (max file number) and -s (max size) because those are the only options that allow producing partitions on the fly. Option -n produces balanced partitions (number and files) and thus requires to have crawled the entire filesystem to be able to sort and group files.

What you could do is use fpart only to produce those 4 partitions (using option -n) and manually start your rsync jobs afterwards, together. If file lists do not change, you can even restart synchronizations regularly without having to regenerate those file lists.

Hope this helps, Best regards,

Ganael.

[ispringer]

Hi Ganael,

Thanks. Calling fpart first to produce the 4 parts followed by calling rsync 4 times in parallel is what we have today. It works well, but I was hoping I could simplify our script by using a single fpsync call. Any chance you will support this use case in the future, or is it outside what you want to support? Thanks.

Best, Ian

[martymac]

Hello Ian,

Fpsync heavily relies on fpart's live mode (hooks + several options depending on files NOT being sorted). It would be hard to support creating exactly partitions and that would not follow the original spirit of fpsync (which main idea is to start transfers as fast as possible), so it will hardly be implemented. I first thought on hacking fpsync's prepare mode but that would need too many code change for (IMHO) few benefits...

I think using fpart as the basis of an external wrapper, as you do now, is probably the best choice...

Cheers,

Ganael.

[ispringer]

Makes sense. Thanks again!