search

backuppc / rsync-bpc

Rsync-bpc is a customized version of rsync that is used as part of BackupPC
GNU General Public License v3.0
28 stars 17 forks source link

rsync_bpc segfault after upgrading to 3.0.9.9/3.0.9.10 #5

Closed stroetgen closed 6 years ago

stroetgen commented 6 years ago

After upgrading rsync_bpc to 3.0.9.9 (and also 3.0.9.10) I find a segfault just for one singe Solaris machine. Backup is broken for this machine.

BackupPC is 4.1.5, BackupPC-XS is 0.57.

backuppc log:

XferLOG file /var/lib/backuppc/pc/ubsun02/XferLOG.138.z created 2017-12-06 10:55:53 
Backup prep: type = incr, case = 4, inPlace = 0, doDuplicate = 0, newBkupNum = 138, newBkupIdx = 30, lastBkupNum = 137, lastBkupIdx = 29 (FillCycle = 0, noFillCnt = 1)
Running: /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 138 --bpc-bkup-comp 3 --bpc-bkup-prevnum 137 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225396 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/bin/rsync --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /
incr backup started for directory /etc
Xfer PIDs are now 868
This is the rsync child about to exec /usr/local/bin/rsync_bpc
rsync_bpc exited with fatal status 0 (139) (This is the rsync child about to exec /usr/local/bin/rsync_bpc)

syslog: Dec 6 10:55:54 backup kernel: [525132.318971] rsync_bpc[868]: segfault at 14 ip 0000000000430c47 sp 00007fff081d2ae0 error 4 in rsync_bpc[400000+7c000]

On the solaris machine

/usr/bin/rsync --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 /etc/ /tmp/test

runs without any problem.

Also no problem for any other machine (including some more solaris machines).

craigbarratt commented 6 years ago

Could you compile rsync_bpc with -g and run the rsync_bpc command via gdb to get a stack trace? You could create a new host (copying ubsun02 config using the copy-host feature in CGI) to do a test backup there if you want. To do that you would run rsync_bpc from the command line under gdb.

Alternatively (and likely easier), you could re-compile rsync_bpc 3.0.9.10 with the address sanitizer options (in particular, add "-fsanitize=address -fsanitize=undefined" to CFLAGS in Makefile after running configure and before running make), eg:

CFLAGS=-g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -fsanitize=address -fsanitize=undefined

The address sanitizer should provide more detailed information before exiting.

stroetgen commented 6 years ago

I run bpc_rsync with installed rsync (/usr/bin/rsync) and find this:

backuppc@backup:/$ gdb --args /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 141 --bpc-bkup-comp 3 --bpc-bkup-prevnum 140 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225656 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /tmp/test
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/rsync_bpc...done.
(gdb) run
Starting program: /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 141 --bpc-bkup-comp 3 --bpc-bkup-prevnum 140 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225656 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /tmp/test

Program received signal SIGSEGV, Segmentation fault.
hashtable_find (tbl=0x0, key=0, allocate_if_missing=allocate_if_missing@entry=1) at hashtable.c:59
59      hashtable.c: Permission denied.

I compiled rsync (Ubuntu 16.04 - rsync_3.1.1-3ubuntu1.debian) from source. rsync_bpc runs ( using /usr/local/bin/rsync) without segfault:

backuppc@backup:/$ gdb --args /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 141 --bpc-bkup-comp 3 --bpc-bkup-prevnum 140 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225656 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/local/bin/rsync --super --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /tmp/test
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/rsync_bpc...done.
(gdb) run
Starting program: /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 141 --bpc-bkup-comp 3 --bpc-bkup-prevnum 140 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225656 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/local/bin/rsync --super --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /tmp/test
xferPids 19540,19545
__bpc_progress_fileCnt__ 1
__bpc_progress_fileCnt__ 21
__bpc_progress_fileCnt__ 41
__bpc_progress_fileCnt__ 61
__bpc_progress_fileCnt__ 81
__bpc_progress_fileCnt__ 101
__bpc_progress_fileCnt__ 121
__bpc_progress_fileCnt__ 141
__bpc_progress_fileCnt__ 161
__bpc_progress_fileCnt__ 181
__bpc_progress_fileCnt__ 201
__bpc_progress_fileCnt__ 221
__bpc_progress_fileCnt__ 241
__bpc_progress_fileCnt__ 261
__bpc_progress_fileCnt__ 281
__bpc_progress_fileCnt__ 301
__bpc_progress_fileCnt__ 321
__bpc_progress_fileCnt__ 341
__bpc_progress_fileCnt__ 361
__bpc_progress_fileCnt__ 381
__bpc_progress_fileCnt__ 401
__bpc_progress_fileCnt__ 421
__bpc_progress_fileCnt__ 441
__bpc_progress_fileCnt__ 461
__bpc_progress_fileCnt__ 481
__bpc_progress_fileCnt__ 501
__bpc_progress_fileCnt__ 521
__bpc_progress_fileCnt__ 541
__bpc_progress_fileCnt__ 561
__bpc_progress_fileCnt__ 581
log: recv >f..t...... r--r--r--        0,       0      1348 mnttab
IOdone: new .mnttab.000000
__bpc_progress_fileCnt__ 601
__bpc_progress_fileCnt__ 621
__bpc_progress_fileCnt__ 641
__bpc_progress_fileCnt__ 661
__bpc_progress_fileCnt__ 681
__bpc_progress_fileCnt__ 701
__bpc_progress_fileCnt__ 721
__bpc_progress_fileCnt__ 741
__bpc_progress_fileCnt__ 761
__bpc_progress_fileCnt__ 781
__bpc_progress_fileCnt__ 801
__bpc_progress_fileCnt__ 821
__bpc_progress_fileCnt__ 841
__bpc_progress_fileCnt__ 861
__bpc_progress_fileCnt__ 881
__bpc_progress_fileCnt__ 901
IOrename: 14 .mnttab.000000mnttab
__bpc_progress_fileCnt__ 921
__bpc_progress_fileCnt__ 941
__bpc_progress_fileCnt__ 961
__bpc_progress_fileCnt__ 981
__bpc_progress_fileCnt__ 1001
__bpc_progress_fileCnt__ 1021
__bpc_progress_fileCnt__ 1041
__bpc_progress_fileCnt__ 1061
__bpc_progress_fileCnt__ 1081
__bpc_progress_fileCnt__ 1101
__bpc_progress_fileCnt__ 1121
__bpc_progress_fileCnt__ 1141
__bpc_progress_fileCnt__ 1161
__bpc_progress_fileCnt__ 1181
__bpc_progress_fileCnt__ 1201
__bpc_progress_fileCnt__ 1221
IOdone: same rc0.d/K00ANNOUNCE
[...]
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 1 filesNew, 1348 sizeNew, 449 sizeNewComp, 225657 inode

Number of files: 2047
Number of files transferred: 2
Total file size: 10837275 bytes
Total transferred file size: 352264 bytes
Literal data: 648 bytes
Matched data: 700 bytes
File list size: 49455
File list generation time: 0.002 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5612
Total bytes received: 54239

sent 5612 bytes  received 54239 bytes  119702.00 bytes/sec
total size is 10837275  speedup is 181.07
G bpc_attrib_dirWrite: can't open/create raw /var/lib/backuppc/pc/ubsun02/141/f%2fetc/ftmp/ftest/attrib.19540 for writing
G bpc_attribCache_dirWrite: failed to write attributes for dir f%2fetc/ftmp/ftest/attrib
DoneGen: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 1545 filesTotal, 10837275 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 225656 inode
[Inferior 1 (process 19540) exited normally]

After compiling with address sanitizer the log shows:


Backup prep: type = incr, case = 4, inPlace = 0, doDuplicate = 0, newBkupNum = 142, newBkupIdx = 30, lastBkupNum = 141, lastBkupIdx = 29 (FillCycle = 0, noFillCnt = 3)
Running: /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 142 --bpc-bkup-comp 3 --bpc-bkup-prevnum 141 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225656 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /
incr backup started for directory /etc
Xfer PIDs are now 24038
This is the rsync child about to exec /usr/local/bin/rsync_bpc
rsync_bpc exited with fatal status 0 (139) (This is the rsync child about to exec /usr/local/bin/rsync_bpc)

Does this help in any way?

craigbarratt commented 6 years ago

Thanks for the additional information. I was hoping to get a stack trace, but at least you found the line where the error occurred.

I'm trying to recreate this. What is the version of /usr/bin/rsync on the client that causes rsync_bpc to crash?

I'll also add a potential fix for you to try.

craigbarratt commented 6 years ago

I haven't been able to re-create this so far. But I pushed a change ba50ec9 that checks the hlink_nlinks hash is non-NULL before accessing it. Can you try that change, either by apply that diff, or building rsync_bpc with the latest 3.0.9 branch, eg:

git clone https://github.com/backuppc/rsync-bpc
cd rsync-bpc
git checkout 3.0.9
./configure
make
sudo make install
stroetgen commented 6 years ago

I just tried the change, backup works without any problem.

Thank you very much!