Does `writelto` use `rsync`?

[retokromer]

I am updating the instructions (WIP). And I have a question to ask: is writelto still using twice rsync? In my opinion, it’s using once gcp – or I am missing something?

[dericed]

It appears to use gcp then calls migratefiles which in this case would run rsync. So if all goes well then the rsync use doesn't do anything, but if gcp fails than rsync should attempt to fix it.

[retokromer]

Thank you! I will delve into this.

[BleakleyMcD]

@dericed That is certainly how it seems to work for us. migratefiles does seems to have a problem with .DS_Store, which even if we remove keeps popping back up as it is known to do, causing rsync to exit with 1 even though all pertinent files have transferred without issue.

/Users/medialab
≈:≈ writelto -v -e N -t LT004B /Volumes/UtilitySAN/LTO/LTO/
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/objects' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/objects'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/objects/2016_129_7_1_PM_MUHAMMAD.mov' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/objects/2016_129_7_1_PM_MUHAMMAD.mov'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/objects/2016_129_7_1_PM_MUHAMMAD.mov.qctools.xml.gz' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/objects/2016_129_7_1_PM_MUHAMMAD.mov.qctools.xml.gz'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/metadata' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/metadata'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/metadata/checksumchecks.log' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/metadata/checksumchecks.log'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/metadata/dfxml.xml' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/metadata/dfxml.xml'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_1_PM_MUHAMMAD/metadata/checksum.md5' -> '/Volumes/LT004B/LT004B/2016_129_7_1_PM_MUHAMMAD/metadata/checksum.md5'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD/objects' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD/objects'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD/objects/2016_129_7_2_PM_MUHAMMAD.mov' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD/objects/2016_129_7_2_PM_MUHAMMAD.mov'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD/metadata' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD/metadata'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD/metadata/checksumchecks.log' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD/metadata/checksumchecks.log'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD/metadata/dfxml.xml' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD/metadata/dfxml.xml'
'/Volumes/UtilitySAN/LTO/LTO//LT004B/2016_129_7_2_PM_MUHAMMAD/metadata/checksum.md5' -> '/Volumes/LT004B/LT004B/2016_129_7_2_PM_MUHAMMAD/metadata/checksum.md5'
  [writelto] 2017-05-12T11:42:47 - Running: /usr/local/bin/migratefiles -o /Volumes/LT004B /Volumes/UtilitySAN/LTO/LTO//
The input is: /Volumes/UtilitySAN/LTO/LTO//
The target is: /Volumes/LT004B/
  [migratefiles] 2017-05-12T11:42:48 - Running: rsync -rtvP --log-file=/Users/medialab/migrate.log --log-file-format=20170512-114248 '%f' %l '/Volumes/LT004B/' /Volumes/UtilitySAN/LTO/LTO// /Volumes/LT004B/
sending incremental file list
./
.DS_Store
          6,148 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=17/19)
LT004B/
LT004B/LT004B_checksums.txt
      3,669,924 100%   28.69MB/s    0:00:00 (xfr#2, to-chk=15/19)

sent 3,677,633 bytes  received 75 bytes  7,355,416.00 bytes/sec
total size is 187,117,706,843  speedup is 50,878.89
Removing hidden files from tape.
The input is: /Volumes/UtilitySAN/LTO/LTO//
The target is: /Volumes/LT004B/
  [migratefiles] 2017-05-12T11:43:05 - Running: rsync -rtvP --log-file=/Users/medialab/migrate.log --log-file-format=20170512-114305 '%f' %l '/Volumes/LT004B/' /Volumes/UtilitySAN/LTO/LTO// /Volumes/LT004B/
sending incremental file list
./
.DS_Store
          6,148 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=17/19)
rsync: mkstemp "/Volumes/LT004B/.DS_Store.DFqfim" failed: Operation not permitted (1)

sent 6,775 bytes  received 135 bytes  13,820.00 bytes/sec
total size is 187,117,706,843  speedup is 27,079,262.93
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1249) [sender=3.1.2]
  2017-05-12T11:43:05 -  [migratefiles]Error: Running: "rsync -rtvP --log-file=/Users/medialab/migrate.log --log-file-format=20170512-114305 '%f' %l '/Volumes/LT004B/' /Volumes/UtilitySAN/LTO/LTO// /Volumes/LT004B/" gave an Error Code - 23
  2017-05-12T11:43:05 -  [migratefiles]The process ran into a critical error and can not proceed.
rsync exited with 0 on the first pass and 1 on the second pass.
Done writing but not ejecting LT004B.

[retokromer]

Hmm… should we add to the scripts, when running under macOS, the following command?

defaults write com.apple.desktopservices DSDontWriteNetworkStores true

[retokromer]

for the initial issue, see now https://github.com/amiaopensource/ltopers/issues/105