search

bit-team / backintime

Back In Time - An easy-to-use backup tool for GNU Linux using rsync in the back
https://backintime.readthedocs.io
GNU General Public License v2.0
2.01k stars 197 forks source link

bit ver 1.2 seems slow and does not respect older snapshots #989

Closed priand1 closed 5 years ago

priand1 commented 5 years ago

Hello All, according to this issue

https://github.com/bit-team/backintime/issues/988

bit 1.2 seems to run very slow but after testing it today I can not confirm this. It rather looks like the according fields in the status bar do not get updated or at least not as often as necessary.

I let it run for 30 min and while the status bar told me

0% | Sent: 2.51G | Speed: 1.56 MB/s

the available space on the external HD I was using for backup was reduced by nearly 20GB.

BIT also was doing a complete backup instead of an incremental one and did not seem to recognize or respect my older snapshots.

Regards Andreas

priand1 commented 5 years ago

Hello again, just want to let you know that bit is working as it is supposed to. The above mentioned symptoms only showed during the first run of v 1.2 when there were older snapshots present. When I do a fresh backup with a test profile or on the second run of bit 1.2 with older snapshots with existing profiles it works as it should:

  1. The status bar is updated correctly and shows the correct progress percentage, GB sent and speed
  2. bit 1.2 does an incremental backup if there is at least one previous snapshot present created by bit 1.2 itself and not a previous version.

Looks like I have been a bit overzealous and my testing was not thorough enough. Hence I am closing this now.

Sorry for the noise and keep up the good work gentlemen.

colinl commented 5 years ago

I am still not clear, if I upgrade to this version will it add a new copy of all my files the first time I run it? I have a number of very large files which rarely change and to add a new copy of all of those will give me a problem.

priand1 commented 5 years ago

I don*t know what distro you are running but I am running openSUSE Tumbleweed with / on btrfs and /home on xfs. Maybe it had to do with that, I don't know. I would definitely give it a try.

I can tell you what I did to test it in the hope it helps you:

  1. With your current version

    • create two folders somewhere called "test" and "test-bit"
    • copy a few GB worth of files to "test"
    • create a test profile, point it to "test" as source and "test-bit" as target and take a snapshot of it
    • after that add some new files to "test"

  2. Install bit 1.2.0

    • load your test profile and let bit 1.2 create a new snapshot of "test" with the newly added files
    • observe how it behaves

Other than that I had a problem similar to yours. The first new snapshot with bit 1.2 was a full backup of 1.1TB and only 800MB space left on the target drive. In the end I just deleted all old snapshots and let bit create a new one. It took roughly 3.5 hours to do so. I went having a beer with friends in the meantime. When I came back it was done.

Good luck!

ghost commented 5 years ago

@FreigeistZ Thanks for this. TBH I come to the same conclusion as you in smaller size VM tests. Thing however is that some do have to consider data preservation, and to just deleted all old snapshots like you mentioned isn't a solution (for everybody). That leaves a new strain of snapshots of course, but then, like @colinl mentions, available space most of the time becomes a PITA. Furthermore, can you tell us what your speeds where during that 3.5 hr backup, because my GUI didn't show anything, and diskio showed never before seen immense fluctuations. Was it in your case similar to the < 1.2 BiT version? I suspect it was, since rsync as far as I know hasn't changed, and when I use it cli it's all good, but that's just gut feeling. Really hope the devs can point us to how to make the transition to 1.2 more compatible and smooth.

priand1 commented 5 years ago

Furthermore, can you tell us what your speeds where during that 3.5 hr backup, because my GUI didn't show anything, and diskio showed never before seen immense fluctuations. Was it in your case similar to the < 1.2 BiT version? I suspect it was, since rsync as far as I know hasn't changed, and when I use it cli it's all good, but that's just gut feeling.

The full backup I took was started at

Apr 30 19:14:02 grilltomate python3[11905]: backintime (atze/1): INFO: Take a new snapshot. Profile: 1 Main profile

and ended at

Apr 30 23:02:18 grilltomate python3[11905]: backintime (atze/1): INFO: Save config file Apr 30 23:02:18 grilltomate python3[11905]: backintime (atze/1): INFO: Save permissions Apr 30 23:02:41 grilltomate python3[11905]: backintime (atze/1): INFO: Create info file Apr 30 23:02:44 grilltomate python3[11905]: backintime (atze/1): INFO: Unlock

It took, to be more precise, 3h48m or 228m. The total size was 1.12T. This translates to roughly 81,87 MB/s to 85.x MB/s - depending if I use MiB or MB for calculation - on average which is consistent with what versions < 1.2 were showing.

I am using plasma5 and have ksysguard running during the backups and it has, just as every version of bit, always shown large swings or fluctuations concerning the transfer rate. My transfer rates have been fluctuating, and they still are, between 120 MB/s max and 1 MB/s min depending on the number of files and their size. IOW when very large files are transferred the transfer rate is between 100 - 120 MB/s and when a large number of very small files is transfered the transfer rate can go as low as 1 MB/s.

ghost commented 5 years ago

Cheers @FreigeistZ for that, that helps, if not for showing me I'm not of (completely) my rocker. I also saw that by now Germar is on the case in #988 so I'm absolutely positive all hick ups will be solved. Apart from the permissions issue they mention he has a very valid point about the "re-evaluation/ calculation" that 1.2 seems to do the 1st time. I suppose that kind of caught me off guard...