backup freezes

[fsironman]

I try to backup a directory and after about 5,3 GB into the backup it just gets stuck.

I tried to check with lsof if they are in use but no.

The 2nd time I tried it, it got stuck after 1079 and 1115 files which was around 5,4 GB

I tried to backup only a subset of the Folder (~500 files) it got stuck on and this was successful.

Then I tried to backup again and after 6,5 GB processed and 5,38 GB deduplicated data it got stuck again.

RAM usage is fine.

Server1 (source) Ubuntu 14.04 LTS Kernel 3.13.0-68 borg 0.30.0 python 3.4.3

Server2 (target) Openmediavault 2.1.25 Kernel 3.16.0-0.bpo.4-amd64 borg 0.30.0 python 3.2.3

[ThomasWaldmann]

Please use borg create -v --progress ... and retry, so you better see what's happening.

[fsironman]

This time it stopped a bit later. As you can see in the concy disk io and network graph its basicly just halts

[ThomasWaldmann]

Can you completely copy the folder to another place?

Just to try whether the problem is caused outside of borg.

I also see you use snapraid. Try same files not on snapraid.

[fsironman]

no luck:

borg create -v --progress [...]/Sync/borg::test2 /mnt/usb1/syncthing
reading files cache N mnt/usb1/syncthing                                
processing files
16.10 GB O 16.10 GB C 7.65 GB D 2491 N mnt/usb1/syn...0160130_214059.mp4

it seems that on every try it can read some more

[fsironman]

stopped again

borg create -v --progress [...]/Sync/borg::test2 /mnt/usb1/syncthing
reading files cache N mnt/usb1/syncthing                                
processing files
19.29 GB O 19.29 GB C 9.12 GB D 2593 N mnt/usb1/syn...y Fukkk Offf).flac

however it seems to make some checkpoints

borg list [...]/Sync/borg
no01                                 Tue, 2016-02-02 10:44:22
no02                                 Tue, 2016-02-02 10:44:50
no03                                 Tue, 2016-02-02 10:46:24
no04                                 Tue, 2016-02-02 10:46:35
no05                                 Tue, 2016-02-02 10:50:07
no06.checkpoint                      Tue, 2016-02-02 11:26:44
no06                                 Tue, 2016-02-02 15:15:39
no07.checkpoint                      Tue, 2016-02-02 16:16:11
test1                                Tue, 2016-02-02 17:00:02
test2.checkpoint                     Tue, 2016-02-02 17:05:23
test2.checkpoint.1                   Tue, 2016-02-02 17:19:21

[ThomasWaldmann]

maybe also consider a hardware or filesystem or OS issue.

a while ago, I've tested borg with ~7.5TB of data and the only issue I discovered was ... a hardware issue.

[fsironman]

I did some further testing:

Server1 (source) Ubuntu 14.04 LTS Kernel 3.13.0-68 borg 0.30.0 python 3.4.3

Server2 (target) Openmediavault 2.1.25 Kernel 3.16.0-0.bpo.4-amd64 borg 0.30.0 python 3.2.3

Any new repository I init on server2 und fill from server1 halts regardless of the data I backup at around 5 GB

I copied some raw data to the server2 and was putting it in a new repository also on server2. No issues.

I also mounted server2 over sshfs on server1 and did a backup. No issues.

I think the problem is related to the transmit of the backup data over the network.

How can I help debugging this issue ?

[ThomasWaldmann]

Could you reproduce locally? Please stop shortening your commandlines, it must be pretty clear whether you are working locally or with a remote repository. Also, for each try, specify your setup so we do not have to guess what it might be.

Do not try to debug with a remote repository (ssh:) if you can reproduce locally! If you can, please remove all references to remote/ssh backups from above posts. Also, add version info to top post. If you can not, please try with python 3.4+ on the server (for a easy/quick try use the borg binary releases on the server, they contain python 3.5.1).

Does /mnt/usb1 mean just a simple ext4 formatted usb disk mounted locally or anything fancy mounted there?

Did you try to copy all the files out of the syncthing folder to another (non-syncthing) folder? Did that work? Did you try again to backup the non-syncthing folder?

[fsironman]

Does /mnt/usb1 mean just a simple ext4 formatted usb disk mounted locally or anything fancy mounted there?

yes normal ext4 drive with a copy of the syncthing data

Did you try to copy all the files out of the syncthing folder to another (non-syncthing) folder? Did that work? Did you try again to backup the non-syncthing folder?

I did this the issue still occured.

I used the 0.30.0 binary on server2 and was not able to reproduce the Issue anymore. Guess python 3.2 on was the problem.

As you are dropping the support for in anyways with 1.0 theres no point looking further into it.

[ThomasWaldmann]

Oh, that's interesting - more py 3.2 brokenness.

I had already documented that we need at least 3.2.2 because the older ones did hang, but if you say even 3.2.3 is having troubles, it is good to get rid of it for borg 1.0. :)

Thanks for your investigations.