Using rsync to copy large file sets, but getting some errors.

[dwhagar]

Here is the exact command output along with the logs.

cyclops@Thom:~$ rsync Desktop/Star\ Trek-\ Discovery /Volumes/Keybase\ \(cyclops\)/private/titanshadow\,shortcut53/
building file list ... 
17 files to consider
Star Trek- Discovery/
Star Trek- Discovery/Season 01/
Star Trek- Discovery/Season 01/Star Trek Discovery - S01E10 - Despite Yourself - WEBDL-720p.mkv
       1.05G 100%    3.76MB/s    0:04:26 (xfer#1, to-check=14/17)
Star Trek- Discovery/Season 01/Star Trek Discovery - S01E11 - The Wolf Inside - WEBDL-720p.mkv
rsync: writefd_unbuffered failed to write 32768 bytes [sender]: Broken pipe (32)
rsync: write failed on "/Volumes/Keybase (cyclops)/private/titanshadow,shortcut53/Star Trek- Discovery/Season 01/Star Trek Discovery - S01E11 - The Wolf Inside - WEBDL-720p.mkv": Input/output error (5)
rsync error: error in file IO (code 11) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/receiver.c(268) [receiver=2.6.9]
rsync: connection unexpectedly closed (896 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) [generator=2.6.9]
rsync: connection unexpectedly closed (64 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) [sender=2.6.9]

my log id: fd82e66793987aba3a257a1c

[strib]

You are hitting timeouts. Essentially, macOS only gives us so much time to respond to a file system event, so we buffer your data in memory and the flush it out to the disk in batches. If we can't flush it fast enough, the in-memory buffer grows. Eventually it reaches a limit of the amount of memory we allow ourselves to use, and if it doesn't drain fast enough, then the file system call needs to time out, leading to the error you see.

Most of the time people don't run into this, unless they are using a device with a combination of low memory (RAM) and a slow disk. Is that the case here? How much RAM do you have? Are you use a spinning (i.e., non-SSD) disk?

[dwhagar]

Well it is a spinning disk and an old computer, so that could be it. I only have 8Gb the max for this 2008 MacBook that I'm using (I know it is ancient). I do have to be patient with it, so I think that is probably the issue at hand. I know that with ExpanDrive (also uses Fuse or a Fuse like system to mount cloud disks), I do run into problems with very large file uploads. I prefer to use something like CyberDuck or some non-OS-methods to upload at times. I did notice as well that Keybase was using a lot of RAM (almost twice as much as it started with) this morning. So I quit it and then opened it back up, this cleared the memory issue.

It only took me about 2-3 runs through rsync to make sure everything was uploaded and it did give me a method to make sure all the files were accurate, if anyone reads this later hoping for a solution, I found it helpful to make sure rsync had the -t and -W (might be -w) options to update timestamps and write whole-files. I don't know how much it really helped.

I would be interested in there being some kind of SFTP-like interface option within keybase to manipulate files, to avoid this kind of an issue for large files on slow systems.

Anyway, finally got everything uploaded.

[strib]

We do have a keybase fs set of commands that doesn't go through the file system interface, which you might find helpful. There's a cp but not a full rsync equivalent.

[dwhagar]

Oh, I did not realize that is what that command did. I had assumed the using the keybase fs commands to copy or move would move things between different storage areas within Keybase itself. I would also suggest having an rsync-like command, perhaps part of the copy command.

I will give the command a try, don't mind dropping to the command line to do stuff.