lsrami
commented
2 months ago 或许他与这个issues相关
Open lsrami opened 2 months ago
lsrami
commented
2 months ago I added the --debug option and got more detailed information. It is clear that the command in line 1770 of the code does not have a given target path.
[cd && rsync --bwlimit=0 -as --log-file=/root/.pfp2/rsync-log-11.18.04_2024-05-08_1 --files-from=/root/.pfp2/fpcache/f.1 '/root' & echo "${!}" >> /root/.pfp2/rsync-PIDs-11.18.04_2024-05-08]
(pytorch) root@notebook-lisirui-cpu1-c7ewc7-notebook-0:~# parsyncfp2 --NP=1 --interface=eth0 --verbose=3 --debug --startdir=/root one /root/two
204 : **DEBUG PROGRESS**
306 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [356]: FPARTSIZE set to: [10G]
FPARTSIZE_N set to [10737418240]
Press [ENTER] to continue.
403 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 WARN: The --NP value is only [1]; to take advantage of pfp2, this should be
set to at least 4. Continuing inefficiently..
587 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Using [eth0] to send data and to monitor
690 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [706]: Using network interface [eth0] with connection quality [ 10000Mb/s]
Press [ENTER] to continue.
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [730]: 1m load is [2.54] and the 1m Load:#CPU ratio is [0.0264583333333333] ( [96] CPU cores).
OK to continue.
Press [ENTER] to continue.
notebook-lisirui-cpu1-c7ewc7-notebook-0 WARN: About to remove all the old cached chunk, log,
and PID files from [/root/.pfp2/fpcache]
and the previous rsync log files from [/root/.pfp2].
Enter ^C to stop this.
Press [ENTER] to continue.
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: [17] files from the previous run have been cleared.
Re-initializing dirs and configuring fpart.. This may take a few secs.
786 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [795]: Composing the new fpart target dirtmp in a loop.
Press [ENTER] to continue.
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [827]: Looping to add the fpart target: ['/root/one' ]
Press [ENTER] to continue.
901 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [977]: fpart cmd:
[(fpart -vvv -L -W 'mv $FPART_PARTFILENAME /root/.pfp2/fpcache' -z -s 10737418240 -o /root/.pfp2/fpcache/hold/f '/root/one' 2> /root/.pfp2/fpcache/FPART.log.11.18.04_2024-05-08 & fpid=$!; fpr='r'; while [ ! -z "$fpr" ]; do sleep 1; fpr=`ps -h -p $fpid`; done; touch /root/.pfp2/fpcache/FPART_DONE ) & ]
Press [ENTER] to continue.
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: fpart launched with command:
[(fpart -vvv -L -W 'mv $FPART_PARTFILENAME /root/.pfp2/fpcache' -z -s 10737418240 -o /root/.pfp2/fpcache/hold/f '/root/one' 2> /root/.pfp2/fpcache/FPART.log.11.18.04_2024-05-08 & fpid=$!; fpr='r'; while [ ! -z "$fpr" ]; do sleep 1; fpr=`ps -h -p $fpid`; done; touch /root/.pfp2/fpcache/FPART_DONE ) & ]
998 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Waiting [2]s for first chunk files to be written.
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: [/root/.pfp2/fpcache/f.1] visible.
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Waiting [4]s for first chunk files to be written.
1108 : **DEBUG PROGRESS**
1184 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Starting the 1st [1] rsyncs ..
1279 : **DEBUG PROGRESS**
1468 : **DEBUG PROGRESS**
1468 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [1581]: sPIDs string = []
Press [ENTER] to continue.
1591 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [1770]: Starting [cd && rsync --bwlimit=0 -as --log-file=/root/.pfp2/rsync-log-11.18.04_2024-05-08_1 --files-from=/root/.pfp2/fpcache/f.1 '/root' & echo "${!}" >> /root/.pfp2/rsync-PIDs-11.18.04_2024-05-08]
Press [ENTER] to continue.
1776 : **DEBUG PROGRESS**
notebook-lisirui-cpu1-c7ewc7-notebook-0 DEBUG: [1779]: Starting [cd && rsync --bwlimit=0 -as --log-file=/root/.pfp2/rsync-log-11.18.04_2024-05-08_1 --files-from=/root/.pfp2/fpcache/f.1 '/root' & echo "${!}" >> /root/.pfp2/rsync-PIDs-11.18.04_2024-05-08]
Press [ENTER] to continue.
hjmangalam
commented
2 months ago HI, Thanks for reporting this. I'm travelling and away from my main rig, but from looking at the versions involved and what you're trying to do (rsync in a local filesystem), the problem is that you're using a recent version of fpart (1.51) with an older version of pfp2. The newer version of fpart starts numbering the chunk files at 1 instead of 0, so pfp2 is waiting for f.0 to appear, which will never happen. I believe that you have to go back to a version of fpart before 1.5 for it to start numbering at 0. More recent versions of pfp2 check compatibility of fpart. You can pull an old version of fpart here: https://github.com/martymac/fpart/releases/tag/fpart-1.4.0
The second possible problem is that pfp2 was not meant for rsyncing across local file systems and I thought that it actually refused to do this (but I do not see the refusal text in your listing). The current version will do this, tho not particularly well for many reasons. Using pfp2 with an NP=1 and syncing across a local file system will definitely be slower than just using native rsync - is there another reason for trying that? hjm
On Tue, May 7, 2024 at 8:00 PM whoami @.***> wrote:
Issue:
Parsyncfp2 encounters an error when parsing the rsync command. From the logs, it seems that there is a problem with the target address parsed on line 1770 of the code, but I'm unable to fix it. System Version:
- Ubuntu VERSION="22.04.4 LTS (Jammy Jellyfish)"
- fpart v1.5.1
- parsyncfp2 version 2.59
- rsync version 3.2.7, protocol version 31
Command Run:
parsyncfp2 --NP=1 --interface=eth0 --verbose=3 --startdir=/root one /root/two Output:
notebook-lisirui-cpu1-c7ewc7-notebook-0 WARN: The --NP value is only [1]; to take advantage of pfp2, this should be set to at least 4. Continuing inefficiently.. notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Using [eth0] to send data and to monitor
notebook-lisirui-cpu1-c7ewc7-notebook-0 WARN: About to remove all the old cached chunk, log, and PID files from [/root/.pfp2/fpcache] and the previous rsync log files from [/root/.pfp2]. Enter ^C to stop this. Press [ENTER] to continue.
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: [9] files from the previous run have been cleared. Re-initializing dirs and configuring fpart.. This may take a few secs.
notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: fpart launched with command: [(fpart -vvv -L -W 'mv $FPART_PARTFILENAME /root/.pfp2/fpcache' -z -s 10737418240 -o /root/.pfp2/fpcache/hold/f '/root/one' 2> /root/.pfp2/fpcache/FPART.log.10.51.44_2024-05-08 & fpid=$!; fpr='r'; while [ ! -z "$fpr" ]; do sleep 1; fpr=
ps -h -p $fpid; done; touch /root/.pfp2/fpcache/FPART_DONE ) & ] notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Waiting [2]s for first chunk files to be written. notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: [/root/.pfp2/fpcache/f.1] visible. notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Waiting [4]s for first chunk files to be written. notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: Starting the 1st [1] rsyncs .. [notebook-lisirui-cpu1-c7ewc7-notebook-0] : Starting [cd && rsync --bwlimit=0 -as --log-file=/root/.pfp2/rsync-log-10.51.44_2024-0rsync version 3.2.7 protocol version 31.1 '/root' & echo "${!}" >> /root/.pfp2/rsync-PIDs-10.51.44_2024-05-08] Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others. Web site: https://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, symlinks, symtimes, hardlinks, hardlink-specials, hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs, xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes Optimizations: SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5 Checksum list: xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none Compress list: zstd lz4 zlibx zlib none Daemon auth list: sha512 sha256 sha1 md5 md4rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details.
rsync is a file transfer program capable of efficient remote update via a fast differencing algorithm. ... ... ... Use "rsync --daemon --help" to see the daemon-mode command-line options. Please see the rsync(1) and rsyncd.conf(5) manpages for full documentation. See https://rsync.samba.org/ for updates, bug reports, and answers rsync error: syntax or usage error (code 1) at options.c(2433) [client=3.2.7] notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: All my rsyncs issued; waiting for them to finish. (stats) ERROR on input, no usable data. Check what you're feeding me.notebook-lisirui-cpu1-c7ewc7-notebook-0 INFO: (Reminder) If you suspect errors, check the rsync logs. [/root/.pfp2/rsync-log-10.51.442024-05-08*] and the fpart log: [/root/.pfp2/fpcache/FPART.log.10.51.44_2024-05-08] Your fpcache files were written in [/root/.pfp2/fpcache] and take up [40K /root/.pfp2]. You rsync'ed about [ bytes = 0.00000 MB] via all [1] rsyncs on this host.
If you used the multihost mode, the whole pfp2 log is here:[/root/.pfp2/pfp-log-10.51.44_2024-05-08] (mostly text but identifies as 'binary' due to text coloration strings): If you used the singlehost mode and want to log the output, capture it by prefixing the pfp2 command with 'script' ('man script' for details).
The host-specific command used for this parsyncfp2 run (in case the host dies midway or for debugging purposes) was:[/usr/bin/parsyncfp2 --NP=1 --interface=eth0 --verbose=3 --startdir=/root one /root/two]
— Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp2/issues/18, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3Y3U3S2UMVU7B3LKL2DZBGIM5AVCNFSM6AAAAABHMDVFWWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4DINJXG43DGNQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
--
Harry Mangalam
Issue:
Parsyncfp2 encounters an error when parsing the rsync command. From the logs, it seems that there is a problem with the target address parsed on line 1770 of the code, but I'm unable to fix it.
System Version:
Command Run:
parsyncfp2 --NP=1 --interface=eth0 --verbose=3 --startdir=/root one /root/twoOutput: