rsync sparse files handling and rsync error notification.

[rstech209]

By experimenting with profile-sync-daemon (overlay mode is not enabled), I realized that some files in my Firefox profile were corrupted (places.sqlite for example). This didn't seem to be too much of a problem for Firefox. 1) In fact, I noticed that the synchronization was very slow (30s to transfer the 100M of my profile) and I started to look for the cause. rsync encountered these kind of errors:

nov. 20 20:27:37 PClinux profile-sync-daemon[26364]: rsync: read errors mapping "/home/rodolphe/.mozilla/firefox/qi68mwcn.default-backup/places.sqlite": Input/output error (5)
nov. 20 20:27:37 PClinux profile-sync-daemon[26364]: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
nov. 20 20:32:00 PClinux profile-sync-daemon[26491]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [sender=3.1.2]

It seemed to me that such a problem (rsync command which generates errors) should be easily detected, without necessarily having to look into the logs. So I forwarded the rsync command lines with an error test and an alert message:

          rsync -aX --delete-after --inplace --no-whole-file --exclude .flagged "$ DIR /" "$ BACKUP /"
          [[$? -ne 0]] && notify-send -i "error" "profile-sync-daemon resync error"

Same thing for 'sync' and 'unsync' operations . On the other hand, notify-sendcommand (or alert or something similar) adds a dependency. What other solution for handling rsync errors might be possible?

2) By searching the issue, I noticed that at the first synchronization, the sparse files were copied like "standard" files.

I tested the sparseness of files (to compare backup directory with "volatile" directory in RAM) with :

find /home/rodolphe/.mozilla/firefox/qi68mwcn.default-backup/ -type f -printf "%S\t%p\n" 2>/dev/null | awk '$1 > 0 && $1 < 1.0 {print}'
find /home/rodolphe/.mozilla/firefox/qi68mwcn.default/ -type f -printf "%S\t%p\n" 2>/dev/null | awk '$1 > 0 && $1 < 1.0 {print}'

I also tried stat command to see block used by sparse files.

The problem seems to be fixed by changing the following line:

... 
        # keep user from launching browser while rsync is active
          rsync -aX --sparse --no-whole-file "$ BACKUP /" "$ TMP"
...

For other synchronizations (resync), the --inplace option seems OK.

Since these tests, I have rebuilt my profile and I no longer see any problem. I don't believe psd is the cause of a profile corruption issue, but have you ever seen anything similar that can result from dealing with sparse files?

[rstech209]

Well, concerning sparse files in a Firefox profile (point 2), I have to admit that I haven't seen anything in Mozzilla's technical literature referring to it ... The advice for copying profiles never takes this point into consideration. To determine the sparsness of the profile files, I used this link, established by Linux gurus!... And I can see some sparses files in a Firefox profile ... Maybe we shouldn't take this into account too much, maybe not ??

This allowed me to do some tests (overlay mode is not enabled) to copy the profile by comparing the performance of cp and rsync (only for first copy):

time cp -a --reflink=auto "$BACKUP"/* "$TMP"

real 0m0,219s
user 0m0,028s
sys 0m0,190s

versus

time rsync -aX --sparse --no-whole-file "$BACKUP/" "$TMP" (equivalent results for time rsync -aX --inplace --no-whole-file "$BACKUP/" "$TMP")

real 0m0,535s
user 0m0,592s
sys 0m0.338s 

No contest! => 'cp' allows to speed up the copy by a factor 2 (and more, depending on the tests) and as a bonus, native support of sparse files. Here, my profile was "only" 70M, but for a much larger volume of data (with several profiles for example) 'cp' could improve the start of the user session. Don't you think it would be a good idea to replace rsync by cp in this case ?

[graysky2]

I believe you'll need to clear the caches between tests or else the 2nd test may artificially complete faster. This can be done by sysctl vm.drop_caches=3.

[rstech209]

The difference is less, but still significant:

sync; sleep 5; echo 3 | sudo tee /proc/sys/vm/drop_caches
time cp -a --reflink=auto "$BACKUP"/* "$TMP"

real    0m3,714s
user    0m0,053s
sys 0m0,357s

sync; sleep 5; echo 3 | sudo tee /proc/sys/vm/drop_caches
time rsync -aX --sparse --no-whole-file "$BACKUP/" "$TMP"

real    0m5,500s
user    0m0,716s
sys 0m0,589s

But of course, resynchronization (destination directory not erased) is more efficient with rsync

sync; sleep 5; echo 3 | sudo tee /proc/sys/vm/drop_caches
time rsync -aX --inplace --no-whole-file "$BACKUP/" "$TMP"

real    0m1,313s
user    0m0,110s
sys 0m0,137s

Seeing the transfer times ... fortunately there are caches !