hjmangalam
commented
4 years ago Hi Ryan,
Sorry for the delay. I've tried to predicate your problem on a couple of machines using different interfaces and networks and I'm afraid I can't.
The child rsyncs always end before the parent pfp. If I kill the parent pfp manually (via ^C) or explicit kill from another term, all the rsyncs die with it: [[ INFO: Starting rsync for chunkfile [/home/hjm/.parsyncfp/fpcache/f.5]..
INFO: Starting rsync for chunkfile [/home/hjm/.parsyncfp/fpcache/f.6]..
INFO: Starting rsync for chunkfile [/home/hjm/.parsyncfp/fpcache/f.7]..
| Elapsed | 1m | [ wlp3s0] MB/s | Running || Susp'd |
Chunks [2020-05-18]Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo] X11 forwarding request failed on channel 0 12.31.59 0.05 1.01 1.28 / 0.00 8 <> 0 [8] of [24] 12.32.03 0.12 1.01 1.12 / 0.00 8 <> 0 [14] of [24] 12.32.06 0.17 1.01 1.14 / 0.00 8 <> 0 [14] of [24] 12.32.09 0.22 1.25 1.11 / 0.00 8 <> 0 [14] of [24] 12.32.12 0.27 1.25 1.12 / 0.00 8 <> 0 [14] of [24] 12.32.15 0.32 1.15 1.12 / 0.00 8 <> 0 [14] of [24] ^C rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3] rsync: [sender] write error: Broken pipe (32) rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3] rsync: [sender] write error: Broken pipe (32) rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3] rsync: [sender] write error: Broken pipe (32)rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [send er=3.1.3] rsync: [sender] write error: Broken pipe (32)
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3] rsync: [sender] write error: Broken pipe (32) rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3] rsync: [sender] write error: Broken pipe (32) rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3]rsync error: received SIGINT, SIGTER M, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.3] rsync: [sender] write error: Broken pipe (32) ]]
this is the way these processes are suposed to work and in my hands that's the way they do work. Could the processes that you're detecting be from another pfp? I can't match the log IDs.
The fpart error is explained by it's author like this: [[ Hmmm... That error can only be triggered when using arbitrary values (fpart's option -a), which asks fpart not to crawl a FS but instead read lines containing something like :
size path
values. The error is then triggered when sscanf() fails reading a line. Is pfp using that option ? It might be interesting to check the exact log line with a tool that displays special characters (e.g. 'cat -bet logfile') to see if there is a whitespace or something else. Finally, it is possible to build fpart with the '--enable-debug' option. It will display info while crawling the filesystem. It may help us better understand what's happening. ]]
And that option is used when fpart is taking lists of files to generate chunks. It's quite possible that there's something in your list that it does't like, such as a weird file name (like '^s' or one of the many wocko names that are possible to create via random mouse events. (I have several like that).
You can recompile fpart to enable the extended debugging if you want to try to track that down. It is supposed to print out the offending name, so the fact that it isn't implies that it's a non-printable character or whitespace.
Let me know if you see the pfp errors in other contexts or if you verify that the surviving rsyncs are children of the parent pfp that exited.
harry
--
Harry Mangalam
novosirj
Here's some output from our most recent run on one of our three campuses; I mentioned in Issue 34 that there seems to be a bug where sometimes PFP exits while rsync processes are still running, which is incorrect. You can see in the below output that PFP exists, and then there are these lines:
What is happening during that time is a
pgrep -x rsyncloop to ensure that there are no longer any rsyncs running that was added by my colleague Bill Abbott, probably to deal with this problem. As you can see, it was several hours before all of thersyncprocesses completed. Here's the full output:While this
rsyncloop was running, I checked for running rsync processes:As you can see, they are numerous.
We still have an error message in the
fpart.log-- it's not clear to me whether it's related, or how I would go about figuring out what is upsetting it:But we have all of the cache files from the run, so we can look them over and run tests in the interim (the transfer finished very fast this week so I have till Tuesday night to tinker).