search

martymac / fpart

Sort files and pack them into partitions
https://www.fpart.org/
BSD 2-Clause "Simplified" License
230 stars 39 forks source link

Single files support #33

Closed d3dx9 closed 3 years ago

d3dx9 commented 3 years ago

Hi!

I'm currently using rsync to transfer multiple TB per day in single files. They are each around 100GB and have the problem that the destination only gets its fullspeed when it uses multiple connections (like what the application axel does for the web) so I found this which seems to be pretty interesting.

Is it somehow possible to sync single files over the day?

martymac commented 3 years ago

Hello,

Yes, you can use fpsync to generate single-file partitions :

$ fpsync -f 1 /src/ /dst/

If used in conjunction with several sync jobs (-n x), you will be ably to sync x files in parallel.

Best regards,

Ganael.

d3dx9 commented 3 years ago

Hi Ganel,

it actually says the Source directory doesn't exist.

(venv) root@34253A1:~# fpsync -f 1 /root/1GB.bin rsync://rsyncdaemon/plots/ Source directory does not exist (or is not a directory): /root/1GB.bin

Best, Julian

martymac commented 3 years ago

Hello Julian,

Right, fpsync is designed to work with directories only. You would have to play with fpart options (through fpsync's '-O' switch) to include wanted files only (maybe something like -O '-y .bin') from src dir. But* there seem to be a unwanted expansion when passing the wildcard character to fpart. I'll check that and try to come with a fix ASAP.

Best regards,

Ganael.

martymac commented 3 years ago

Hello Julian,

I've just pushed a fix for that long-standing issue. Can you try something like :

$ fpsync -f 1 -O '-y|*.bin' /root/ rsync://rsyncdaemon/plots/

It will scan /root/ for files named *.bin and generate single-filed partitions.

I hope this will answer what you were looking for.

Best regards, Ganael.

martymac commented 3 years ago

Closing for now, thanks again for your report!