Diagnosing uninterrupted copies when connections lost

[dsoprea]

I need help to figure what's going on with my file copies to Amazon. I've been struggling with this for a week and the results oscillate between unreliable/inconsistent and termination due to failure.

My Amazon account is current FUSE-mounted. A moment ago, I terminated an rsync of a very large file because it was flying at 37MB/S and yet not showing in the Amazon Drive console (webpage). When I then listed the file in the mount, it also showed no trace of it. I just unmounted and remounted a half-hour ago.

By the way, when I unmounted, there was still an instance of the acd_cli process that I had to kill by hand. Yesterday, I found five or six when I had called umount and yet couldn't remount because the directory (which appeared to be empty and unmounted) and still indicated as busy and unable to be mounted. It seems like unmounting often doesn't do so cleanly.

This is what I'm currently dealing with:

I interrupted the copy (notice the speeds):

# rsync -a --progress --append --partial /massive_zfs/image_archives/done/images.201* /mnt/amazon_drive/Tower\ Image\ Backups/
sending incremental file list
images.2011.408ee332b82a6a4a2e99fe0ebcc91122d55ed139.20161013-022245.tar.xz
 17,804,391,676 100%   59.23MB/s    0:01:27 (xfr#1, to-chk=5/7)
images.2012.f6114b7714b654cfe63013459504a8cbba137150.20161013-092411.tar.xz
 68,335,370,240  65%   36.96MB/s    0:15:57  ^C
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.0]
rsync: write failed on "/mnt/amazon_drive/Tower Image Backups/images.2011.408ee332b82a6a4a2e99fe0ebcc91122d55ed139.20161013-022245.tar.xz": Bad address (14)
rsync error: error in file IO (code 11) at receiver.c(389) [receiver=3.1.0]

Listed on my local mount:

# ls -la /mnt/amazon_drive/Tower\ Image\ Backups
total 0
drwxrwxrwx 1 root root           0 Oct 16 17:34 .
drwxrwxrwx 1 root root           0 Oct 12 09:19 ..
-rw-rw-rw- 1 root root 12353536000 Oct 16 21:04 images.2011.408ee332b82a6a4a2e99fe0ebcc91122d55ed139.20161013-022245.tar.xz

The "2012" file is listed nowhere. The "2011" file is about five gigabytes smaller than I would expect (its velocity was also way too high). I've been observing this on most copies over the past week.

What's going on, here?

Thanks.

[dsoprea]

As far as the lingering processes are concerned, here's an example from right-now. After I received a bunch of _acdcli "sync" errors being printed to my console while I wasn't doing anything with it, I just wanted to unmount it until I could discuss it more under an issue. To my knowledge, it doesn't matter how long you wait after an unmount. They'll still hang around:

# acd_cli umount /mnt/amazon_drive/
# ps aux | grep acd_cli
root     10063  0.0  0.2 110524 32876 pts/19   S    21:24   0:01 /usr/bin/python3 /usr/local/bin/acd_cli mount /mnt/amazon_drive
root     10089  0.0  0.2 110516 32872 pts/19   S    21:25   0:01 /usr/bin/python3 /usr/local/bin/acd_cli mount /mnt/amazon_drive
root     10338  0.0  0.0  15936   944 pts/19   S+   22:26   0:00 grep --color=auto acd_cli