Backup takes extremely long

[M-Welsch]

Describe the bug

Backup takes longer than expected

Expected behavior

A weekly backup should take a few hours at most (gut feeling).

Actual behavior

Some backups take a whole day.

What happens if we don't solve it (aka why is it important)

long on-times with

• greater chance of being hacked
• greater change of mechanical impact during run or lightning strike

To Reproduce

see https://github.com/M-Welsch/backup-server/issues/46#issuecomment-2227293940. Further investigation necesssary

Additional context, Environment

Add any other context about the problem here.

Describe/define the problem

Problem seems to be that full backups are performed instead of increments. BaSe transfers much more data than necessary

Develop Interim Containment Plan (if necessary)

As of now two possiblities:

• get rid of increments cp -al ...
• or change the rsync command to rsync -av ... or rsync -hv ... not sure which

Determine Root Causes and Escape Points

Pointer to the solution

Actions to prevent recurrence or solve systematic problems

[M-Welsch]

TL;DR

there seems to be a problem with cp -al ... in combination with rsync -avh ...

• full backups have been executed even if there were only small changes
• reproducible when cp -al .. then rsync -avh ... regardless of smb or ssh (maybe even rsync daemon)
• not reproduced when
  • no cp -al, but directly overwrting with rsync -avh
  • cp -al .., then rsync -av .. (no -h parameter .. or no -a. There was something with the params)

need to verify

Details

surprise, no further details :)

[M-Welsch]

Summary

rsync preserves timestamps and everything. It seems to copy whole files if the timestamp doesn't match.

Details

$mkdir backup
$rsync_sink cp ~/Base/Literatur/. backup -r
rsync -avh ~/Base/Literatur/. backup --stats  # transfers everything
rsync -avh ~/Base/Literatur/. backup --stats  # transfers nothing

why? Let's see ...

after running rsync -avh .. for the first time, the timestamps have been updated. cp -a ... preserves everything. New try:

$cp ~/Base/Literatur/. backup -ra
$rsync -avh ~/Base/Literatur/. backup --stats
Number of regular files transferred: 0
Total file size: 39,25M bytes

Able to reproduce, let's see if that is the actual issue. Running a backup until we reach relevant data and see if there's a mismatch in permissions, user, time or whatever ...

hdd/nextcloud/martha/files/The Thing/2sort/2023-04-22-13-57-47-318.jpg

letting rsync run over this directory

timestamps have an hour offset, but are otherwise in sync (because server has a different time zone .. didnt fix that)