fpart do not fork

[mcha-connectics]

Hello there, I am planning to migrate a bunch a gigabytes with parsyncfp2, but so far my first attempts fail. The linux OS I am working on is an old 'CentOS Linux release 7.9.2009 (Core)'. The data should move from a NAS device to a new one. For now both are mounted via NFS. This is the command I am using:

parsyncfp2 --verbose 3 --nowait \
--maxload=5 --NP=4 --chunksize=$((100 * 1024 * 1024)) \
--startdir='/mnt/nfs/source/' myfolder \
/mnt/nfs/dest/

Error: any valid prefix is expected rather than "/mnt/nfs/source/myfolder/".
Command line is not complete. Try option "help"
vs-inf-prd-exp-fr-502 INFO: Using [] to send data and to monitor

WARN: About to remove all the old cached chunk, log, 
  and PID files from [/root/.pfp2/fpcache]
  and the previous rsync log files from [/root/.pfp2].       
  Enter ^C to stop this.
  If you specified '--nowait', cache will be cleared in 1s regardless.
  Otherwise, hit [Enter] and I'll clear them. 
INFO: [8] files from the previous run have been cleared .. continuing.
INFO: Waiting [756]s for chunk files to be written.

I tried with a SSH destination url like 'user@remoteip:/full/path' but the problem still the same and don't have much clue about the issue. Any help would be much appreciated. Regards

[hjmangalam]

HI there (and thanks for trying pfp2), I realize that the latest internals of pfp2 are not documented correctly. Specifically, the help still encourages filesystem to filesystem use: [[ It can also be used for non-host-based transfers (between mounted filesystems) but the network bandwidth continues to be (usually pointlessly) shown.

[NB: Between mounted filesystems, parsyncfp2 sometimes works very poorly for reasons still mysterious. In such cases, I recommend the fpsync tool contained in the fpart package above]. ]] but internally, I've given up on supporting it since rsync doesn't internally support the block-checksummed exchanges that the host-based rsync does. As I understand it, FS to FS rsyncs essentially use 'cp' to transfer data over and so lacks the fantastic efficiency of the host-based transfers. (the reference above to "parsyncfp2 sometimes works very poorly for reasons still mysterious" - all parallel rsync wrappers will work similarly poorly on FS to FS transfers since they're all subject to rsync's poor FS to FS perf. So internally, pfp2 now is pretty rude when asked to transfer to a non-host destination.

I'll correct the misleading info in the help file and make the refusal to support it less abrupt. Thanks for the note and apologies for your wasted time.

Best wishes Harry

On Tue, Aug 16, 2022 at 11:24 AM mcha-connectics @.***> wrote:

Hello there, I am planning to migrate a bunch a gigabytes with parsyncfp2, but so far my first attempts fail. The linux OS I am working on is an old 'CentOS Linux release 7.9.2009 (Core)'. The data should move from a NAS device to a new one. For now both are mounted via NFS. This is the command I am using:

parsyncfp2 --verbose 3 --nowait \ --maxload=5 --NP=4 --chunksize=$((100 1024 1024)) \ --startdir='/mnt/nfs/source/' myfolder \ /mnt/nfs/dest/

Error: any valid prefix is expected rather than "/mnt/nfs/source/myfolder/". Command line is not complete. Try option "help" vs-inf-prd-exp-fr-502 INFO: Using [] to send data and to monitor

WARN: About to remove all the old cached chunk, log, and PID files from [/root/.pfp2/fpcache] and the previous rsync log files from [/root/.pfp2]. Enter ^C to stop this. If you specified '--nowait', cache will be cleared in 1s regardless. Otherwise, hit [Enter] and I'll clear them. INFO: [8] files from the previous run have been cleared .. continuing. INFO: Waiting [756]s for chunk files to be written.

I tried with a SSH destination url like @.***:/full/path' but the problem still the same and don't have much clue about the issue. Any help would be much appreciated. Regards

— Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp2/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YZNY4HAXGSZGVRN7A3VZOQDDANCNFSM56WBOYTQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Harry Mangalam

[hjmangalam]

Oh wait.. You say that you tried it with a host-based destination and it still fails..?

parsyncfp2 --verbose 3 --nowait \ --maxload=5 --NP=4 --chunksize=$((100 1024 1024)) \ --startdir='/mnt/nfs/source/' myfolder \ @.***:/home/hjm/testparsync

That should definitely work. A similar command on my laptop shows this: [image: image.png] hopefully this img will post. If not, let me know and I'll post the text.

Which version are you using incidentally?

Harry

On Tue, Aug 16, 2022 at 11:24 AM mcha-connectics @.***> wrote:

Hello there, I am planning to migrate a bunch a gigabytes with parsyncfp2, but so far my first attempts fail. The linux OS I am working on is an old 'CentOS Linux release 7.9.2009 (Core)'. The data should move from a NAS device to a new one. For now both are mounted via NFS. This is the command I am using:

parsyncfp2 --verbose 3 --nowait \ --maxload=5 --NP=4 --chunksize=$((100 1024 1024)) \ --startdir='/mnt/nfs/source/' myfolder \ /mnt/nfs/dest/

Error: any valid prefix is expected rather than "/mnt/nfs/source/myfolder/". Command line is not complete. Try option "help" vs-inf-prd-exp-fr-502 INFO: Using [] to send data and to monitor

WARN: About to remove all the old cached chunk, log, and PID files from [/root/.pfp2/fpcache] and the previous rsync log files from [/root/.pfp2]. Enter ^C to stop this. If you specified '--nowait', cache will be cleared in 1s regardless. Otherwise, hit [Enter] and I'll clear them. INFO: [8] files from the previous run have been cleared .. continuing. INFO: Waiting [756]s for chunk files to be written.

I tried with a SSH destination url like @.***:/full/path' but the problem still the same and don't have much clue about the issue. Any help would be much appreciated. Regards

— Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp2/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YZNY4HAXGSZGVRN7A3VZOQDDANCNFSM56WBOYTQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Harry Mangalam

[mcha-connectics]

Hi,

The version used in august was:

# parsyncfp2 --version
parsyncfp version 2.44 (Sarty Rd)
June 24, 2022

Regards

[hjmangalam]

Apologies that I'm only now getting to closing the issue. I believe that the current version addresses both the FS to FS use as well as the network issue. If there's still a problem, please let me know. hjm