Documentation for local mounts

[martinvelez]

In my case, I do not need to ssh into remote machines. I am trying to use this tool to copy data between two local NFS mounts. For example, I am trying to copy data from foo to bar.

martin.velez@example.com:~$ time sudo ./parsyncfp --maxload=5.5 --NP=4 --startdir=/mnt/ foo /mnt/bar

Is this tool appropriate for that? Can I disable that ssh check?

[hjmangalam]

On Monday, May 6, 2019 3:54:24 PM PDT Martin Velez wrote:

In my case, I do not need to ssh into remote machines. I am trying to use this tool to copy data between two local NFS mounts.

Is this tool appropriate for that? Can I disable that ssh check?

You should be able to just allow the ssh check to go forward - it's just adding your own public ssh key to your ~/.ssh/authorized_keys, so that should make no difference, functionally.

While I use it routinely to rsync large dir trees back and forth over locally mounted parallel filesystems, I have seen instances where using it over locally mounted NFS filesystems seems to cause it to choke.

Please try it and see how it behaves and report back. It may be that the NFS buffering exposes some inherent logic flaw that I don't understand.

hjm

Harry Mangalam, Info[1]

---

[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

[martinvelez]

To clarify:

• foo (source) is a FreeNAS filesystem mounted via NFS.
• bar (destination) is a Lustre filesystem mounted via NFS.

I tried it last night. It only transferred 11 / 4092 folders. I got Argument list too long errors. What causes that? Is there a way to fix it?

martin.velez@example:~$ time sudo ./parsyncfp --startdir=/mnt foo /mnt/bar
 INFO: parsyncfp would like to append 'ForwardX11Trusted yes' & 'ForwardX11 yes'
        to your ~/.ssh/config.
        Skipping this may result in a lot of odd ssh warnings being emitted during
        the run if you don't have ssh set correctly for the remote system, but the
        transfer should still work.)

If this mod of your ~/.ssh/config file is OK, hit [Enter].  Otherwise hit [s] to skip.

sh: 1: cannot create /home/martin.velez/.ssh/config: Directory nonexistent
chmod: cannot access /home/martin.velez/.ssh/config: No such file or directory
 INFO: Your ~/.ssh/config file is set correctly.
 WARN: About to remove all the old cached chunkfiles from [/home/martin.velez/.parsyncfp/fpcache].
  Enter ^C to stop this.
  If you specified '--nowait', cache will be cleared in 3s regardless.
  Otherwise, hit [Enter] and I'll clear them.
Press [ENTER] to continue.

 INFO: The fpart chunk files [/home/martin.velez/.parsyncfp/fpcache/f*] are cleared .. continuing.
 INFO: Forking fpart.  Check [/home/martin.velez/.parsyncfp/fpcache/fpart.log.02.50.51_2019-05-07] for errors if it hangs.
 INFO: Starting the 1st [6] rsyncs ..
        | Elapsed |   1m   |    [   eth0]   MB/s  | Running || Susp'd  |      Chunks       [2019-05-07]
  Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   | [UpTo] of [ToDo]
02.51.08    0.07     0.00     133.87 / 0.00             6    <>   0          [6] of [324]
02.51.11    0.12     0.48       1.36 / 0.00             6    <>   0          [6] of [830]
02.51.14    0.17     1.25      24.91 / 0.00             6    <>   0          [6] of [1696]
02.51.17    0.22     1.25       7.18 / 0.00             6    <>   0          [6] of [2653]
02.51.20    0.27     1.63      68.60 / 0.00             6    <>   0          [9] of [3568]
02.51.23    0.32     1.74      67.58 / 0.00             6    <>   0          [12] of [4437]
02.51.26    0.37     1.74       2.69 / 0.00             6    <>   0          [12] of [5313]
02.51.30    0.43     2.08      28.97 / 0.00             6    <>   0          [13] of [6230]
02.51.33    0.48     2.08      47.77 / 0.00             6    <>   0          [15] of [7226]
02.51.36    0.53     2.47      52.53 / 0.00             6    <>   0          [18] of [8167]
02.51.39    0.58     2.68      15.22 / 0.00             6    <>   0          [18] of [9149]
02.51.43    0.65     2.68      94.44 / 0.00             6    <>   0          [24] of [10212]
02.51.46    0.70     3.18      17.40 / 0.00             6    <>   0          [24] of [11017]
02.51.49    0.75     3.41      43.08 / 0.00             6    <>   0          [26] of [12058]
02.51.53    0.82     3.41      70.81 / 0.00             5    <>   0          [30] of [12927]
02.51.56    0.87     3.54      64.43 / 0.00             6    <>   0          [32] of [14051]
02.51.59    0.92     3.33      62.13 / 0.00             6    <>   0          [36] of [15202]
02.52.03    0.98     3.33      35.77 / 0.00             5    <>   0          [38] of [16367]
02.52.06    1.03     3.39     104.80 / 0.00             5    <>   0          [42] of [17596]
02.52.10    1.10     3.44      96.51 / 0.00             6    <>   0          [44] of [18865]
02.52.13    1.15     3.72      67.11 / 0.00             6    <>   0          [48] of [20224]
        | Elapsed |   1m   |    [   eth0]   MB/s  | Running || Susp'd  |      Chunks       [2019-05-07]
  Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   | [UpTo] of [ToDo]
02.52.17    1.22     3.72     106.09 / 0.00             6    <>   0          [50] of [21520]
02.52.20    1.27     3.66     109.77 / 0.00             4    <>   0          [54] of [22898]
02.52.24    1.33     4.09      35.23 / 0.00             5    <>   0          [56] of [24215]
02.52.27    1.38     4.09     138.17 / 0.00             5    <>   0          [60] of [25602]
02.52.31    1.45     4.16     119.43 / 0.00             6    <>   0          [62] of [26920]
02.52.34    1.50     4.39      22.97 / 0.00             5    <>   0          [63] of [28240]
02.52.38    1.57     4.39      48.21 / 0.00             4    <>   0          [66] of [29645]
02.52.42    1.63     4.44     124.34 / 0.00             6    <>   0          [69] of [31022]
02.52.45    1.68     4.57     126.95 / 0.00             6    <>   0          [72] of [32446]
02.52.49    1.75     4.68       1.35 / 0.00             6    <>   0          [72] of [33720]
02.52.52    1.80     4.68     116.63 / 0.00             6    <>   0          [75] of [35204]
02.52.56    1.87     4.79      91.55 / 0.00             6    <>   0          [78] of [36653]
02.53.00    1.93     4.72      66.25 / 0.00             6    <>   0          [81] of [38087]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.45    4.68     1.24      35.25 / 0.00             3    <>   0          [84] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.48    4.73     1.24      46.57 / 0.00             6    <>   0          [87] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.52    4.80     1.78      48.44 / 0.00             5    <>   0          [90] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.55    4.85     1.80      53.06 / 0.00             6    <>   0          [93] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.59    4.92     1.82      47.66 / 0.00             6    <>   0          [96] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.03    4.98     1.82      51.00 / 0.00             5    <>   0          [100] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.06    5.03     2.15      37.64 / 0.00             6    <>   0          [102] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.10    5.10     2.06      63.50 / 0.00             6    <>   0          [106] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
        | Elapsed |   1m   |    [   eth0]   MB/s  | Running || Susp'd  |      Chunks       [2019-05-07]
  Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   | [UpTo] of [ToDo]
sh: 1: ls: Argument list too long
02.56.13    5.15     2.06      35.14 / 0.00             6    <>   0          [108] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.17    5.22     2.37     128.42 / 0.00             6    <>   0          [112] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.20    5.27     2.42      33.67 / 0.00             6    <>   0          [114] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.24    5.33     2.31     114.27 / 0.00             6    <>   0          [118] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.28    5.40     2.31     105.53 / 0.00             6    <>   0          [120] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.31    5.45     2.53      15.30 / 0.00             6    <>   0          [121] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.34    5.50     2.48      42.67 / 0.00             4    <>   0          [124] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.38    5.57     2.48     106.17 / 0.00             5    <>   0          [126] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.41    5.62     2.61      19.61 / 0.00             5    <>   0          [127] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.45    5.68     2.56      49.69 / 0.00             5    <>   0          [130] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.49    5.75     2.59     117.46 / 0.00             6    <>   0          [133] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.52    5.80     2.59      85.25 / 0.00             6    <>   0          [133] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.55    5.85     2.62      47.25 / 0.00             3    <>   0          [136] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.59    5.92     2.57     136.76 / 0.00             6    <>   0          [140] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.03    5.98     2.57      91.77 / 0.00             6    <>   0          [142] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.06    6.03     2.69       1.04 / 0.00             6    <>   0          [142] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.09    6.08     2.87      91.61 / 0.00             5    <>   0          [144] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.13    6.15     2.87      95.08 / 0.00             4    <>   0          [146] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.16    6.20     2.96      38.93 / 0.00             6    <>   0          [148] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.20    6.27     3.21      38.89 / 0.00             6    <>   0          [150] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.23    6.32     3.21      91.42 / 0.00             6    <>   0          [152] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
        | Elapsed |   1m   |    [   eth0]   MB/s  | Running || Susp'd  |      Chunks       [2019-05-07]
  Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   | [UpTo] of [ToDo]
sh: 1: ls: Argument list too long
02.57.27    6.38     3.35       0.96 / 0.00             6    <>   0          [152] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.30    6.43     3.56      27.52 / 0.00             6    <>   0          [154] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.34    6.50     3.52     113.40 / 0.00             6    <>   0          [158] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.37    6.55     3.52      53.30 / 0.00             5    <>   0          [159] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.41    6.62     3.64      19.94 / 0.00             6    <>   0          [160] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.44    6.67     3.51     104.16 / 0.00             6    <>   0          [164] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.48    6.73     3.51      73.16 / 0.00             6    <>   0          [166] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.51    6.78     3.87       0.96 / 0.00             6    <>   0          [166] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.55    6.85     3.96      88.17 / 0.00             5    <>   0          [169] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.58    6.90     3.96      74.64 / 0.00             5    <>   0          [171] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.02    6.97     4.04      16.97 / 0.00             6    <>   0          [172] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.05    7.02     4.12      69.73 / 0.00             6    <>   0          [176] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.09    7.08     3.95      95.61 / 0.00             6    <>   0          [178] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.12    7.13     3.95     140.59 / 0.00             6    <>   0          [182] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.16    7.20     4.11      50.70 / 0.00             6    <>   0          [184] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.19    7.25     4.18      50.04 / 0.00             6    <>   0          [184] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.23    7.32     4.18     134.90 / 0.00             6    <>   0          [188] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.26    7.38     4.41      32.33 / 0.00             6    <>   0          [190] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.30    7.43     4.38      45.05 / 0.00             6    <>   0          [190] of [0]
sh: 1: ls: Argument list too long
 INFO: Done.  Please check the target to make sure
  expected files are where they're supposed to be.
 INFO:
    The parsyncfp cache dir takes up [1.4G      /home/martin.velez/.parsyncfp]
    Don't forget to delete it, but wait until you are sure that your job
    completed correctly, so you don't need the log files anymore.

Reminder: check the parsyncfp log [/home/martin.velez/.parsyncfp/rsync-logfile-02.50.51_2019-05-07_189]
    and the fpart log [/home/martin.velez/.parsyncfp/fpcache/fpart.log.02.50.51_2019-05-07]
    if there were errors. Use '--verbose=1' for less output.

      Thanks for using parsyncfp. Tell me how to make it better.
                       <hjmangalam@gmail.com>

real    7m42.134s
user    0m19.844s
sys     0m19.428s

[hjmangalam]

On Tuesday, May 7, 2019 9:28:09 AM PDT Martin Velez wrote:

To clarify:

• foo (source) is a FreeNAS filesystem mounted via NFS.
• bar (destination) is a Lustre filesystem mounted via NFS.

I tried it last night. It only transferred 11 / 4092 folders. I got Argument list too long errors. What causes that? Is there a way to fix it?

Yes. In fact this is a fairly frequent problem that I should address programmatically. It results when your chunk size is too small, resulting in a huge list of f.* files to transfer (in your ~/.parsyncfp dir.. Try doubling or tripling the chunk size you're using and let me know if that improves things.

hjm

martin.velez@example:~$ time sudo ./parsyncfp --startdir=/mnt foo
/mnt/bar INFO: parsyncfp would like to append 'ForwardX11Trusted yes' &
'ForwardX11 yes' to your ~/.ssh/config.
        Skipping this may result in a lot of odd ssh warnings being
emitted during the run if you don't have ssh set correctly for the
remote system, but the transfer should still work.)

If this mod of your ~/.ssh/config file is OK, hit [Enter].  Otherwise hit
[s] to skip.

sh: 1: cannot create /home/martin.velez/.ssh/config: Directory
nonexistent chmod: cannot access /home/martin.velez/.ssh/config: No
such file or directory INFO: Your ~/.ssh/config file is set correctly.
 WARN: About to remove all the old cached chunkfiles from
[/home/martin.velez/.parsyncfp/fpcache]. Enter ^C to stop this.
  If you specified '--nowait', cache will be cleared in 3s regardless.
  Otherwise, hit [Enter] and I'll clear them.
Press [ENTER] to continue.

 INFO: The fpart chunk files [/home/martin.velez/.parsyncfp/fpcache/f*]
are cleared .. continuing. INFO: Forking fpart.  Check
[/home/martin.velez/.parsyncfp/fpcache/fpart.log.02.50.51_2019-05-07]
for errors if it hangs. INFO: Starting the 1st [6] rsyncs ..

        | Elapsed |   1m   |    [   eth0]   MB/s  | Running || Susp'd  | 
        |     Chunks       [2019-05-07]
  Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   |
[UpTo] of [ToDo] 02.51.08    0.07     0.00     133.87 / 0.00            
6    <>   0          [6] of [324] 02.51.11    0.12     0.48       1.36 /
0.00             6    <>   0          [6] of [830] 02.51.14    0.17    
1.25      24.91 / 0.00             6    <>   0          [6] of [1696]
02.51.17    0.22     1.25       7.18 / 0.00             6    <>   0     
    [6] of [2653] 02.51.20    0.27     1.63      68.60 / 0.00           
 6    <>   0          [9] of [3568] 02.51.23    0.32     1.74      67.58
/ 0.00             6    <>   0          [12] of [4437] 02.51.26    0.37 
   1.74       2.69 / 0.00             6    <>   0          [12] of
[5313] 02.51.30    0.43     2.08      28.97 / 0.00             6    <>  
0          [13] of [6230] 02.51.33    0.48     2.08      47.77 / 0.00   
         6    <>   0          [15] of [7226] 02.51.36    0.53     2.47  
   52.53 / 0.00             6    <>   0          [18] of [8167] 02.51.39
   0.58     2.68      15.22 / 0.00             6    <>   0          [18]
of [9149] 02.51.43    0.65     2.68      94.44 / 0.00             6   
<>   0          [24] of [10212] 02.51.46    0.70     3.18      17.40 /
0.00             6    <>   0          [24] of [11017] 02.51.49    0.75  
  3.41      43.08 / 0.00             6    <>   0          [26] of
[12058] 02.51.53    0.82     3.41      70.81 / 0.00             5    <> 
 0          [30] of [12927] 02.51.56    0.87     3.54      64.43 / 0.00 
           6    <>   0          [32] of [14051] 02.51.59    0.92    
3.33      62.13 / 0.00             6    <>   0          [36] of [15202]
02.52.03    0.98     3.33      35.77 / 0.00             5    <>   0     
    [38] of [16367] 02.52.06    1.03     3.39     104.80 / 0.00         
   5    <>   0          [42] of [17596] 02.52.10    1.10     3.44     
96.51 / 0.00             6    <>   0          [44] of [18865] 02.52.13  
 1.15     3.72      67.11 / 0.00             6    <>   0          [48]
of [20224]
        | Elapsed |   1m   |    [   eth0]   MB/s  | Running || Susp'd  | 
        |     Chunks       [2019-05-07]
  Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   |
[UpTo] of [ToDo] 02.52.17    1.22     3.72     106.09 / 0.00            
6    <>   0          [50] of [21520] 02.52.20    1.27     3.66    
109.77 / 0.00             4    <>   0          [54] of [22898] 02.52.24 
  1.33     4.09      35.23 / 0.00             5    <>   0          [56]
of [24215] 02.52.27    1.38     4.09     138.17 / 0.00             5   
<>   0          [60] of [25602] 02.52.31    1.45     4.16     119.43 /
0.00             6    <>   0          [62] of [26920] 02.52.34    1.50  
  4.39      22.97 / 0.00             5    <>   0          [63] of
[28240] 02.52.38    1.57     4.39      48.21 / 0.00             4    <> 
 0          [66] of [29645] 02.52.42    1.63     4.44     124.34 / 0.00 
           6    <>   0          [69] of [31022] 02.52.45    1.68    

[jxiong]

You probably need to specify --chunkksize to a larger value.

[martinvelez]

I tried 3 values: --chunkksize 10G, --chunkksize 100G and --chunkksize 1T. I still see those errors.

[hjmangalam]

On Wednesday, June 5, 2019 3:52:50 PM PDT Martin Velez wrote:

I tried 3 values: --chunkksize 10G, --chunkksize 100G and --chunkksize 1T. I still see those errors.

Could you send me the command you used and (assuming that you're using ~/.parsyncfp) attach the file:

~/.parsyncfp/fpcache/fpart.log ... (whatever is the one that failed most recently.

Obviously, look at it to make sure it doesn't contain any proprietary info, but it should contain only the stats from the fpart chunking operation.

gzip it if it's ginormous.

hjm

Harry Mangalam, Info[1]

---

[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

[martinvelez]

Since I opened this ticket, I tried https://github.com/jbd/msrsync and it helped me speed up copying large directories.

[hjmangalam]

Hi Martin,

Glad that worked for you. I haven't tried it, but I will. Can you say what you liked about it vs parsyncfp or fpsync or any of the others? Was it easier to use? faster? less overhead? better docs?

Harry

On Tue, Jul 7, 2020 at 7:04 PM Martin Velez notifications@github.com wrote:

Since I opened this ticket, I tried https://github.com/jbd/msrsync and it helped me speed up my file transfer.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/21#issuecomment-655235982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YYSIYDXJ3W2JBNBCFTR2PHY7ANCNFSM4HLEAAKA .

--

Harry Mangalam

[martinvelez]

It was easier to use. I was able to use it with directories of any size without having to find the right parameters by trial-and-error. The only parameter I tuned was the number of parallel processes. I was never able to compare speed.