performance: holland closes invalid file descriptors

[mikegriffin]

Using a new cent7 kick, install holland and mariadb from stock channels:

yum install mariadb-server mariadb holland-mysqldump strace holland mc mysqldump -f /etc/holland/backupsets/default.conf systemctl start mariadb holland bk strace -f /usr/bin/mysqldump --flush-privileges --routines --events --max-allowed-packet=128M --lock-tables mysql 2>&1 | grep -c EBADF strace -f holland bk default 2>&1 | grep -c EBADF

You'll see the number of EBADF errors scales with ulimit:

[root@mg-cent7 ~]# ulimit -n 33333 [root@mg-cent7 ~]# strace -f holland bk default 2>&1 | grep -c EBADF 99974

This can also be seen by profiling with perf-record:

[root@mg-cent7 ~]# perf record -g holland bk 2> /dev/null [root@mg-cent7 ~]# perf report -g --stdio | grep close 17.09% 6.75% holland libpthread-2.17.so [.] __libc_close

[mikegriffin]

This seems to be a bug in 1.1:

cent6# holland --version | grep Backup Holland Backup v1.0.6 cent6# strace -ff holland bk 2>&1 | grep -c EBADF 0 cent6# yum install holland ---> Package holland.noarch 0:1.0.6-3.fc13 will be updated ---> Package holland.noarch 0:1.1.21-1.el6 will be an update cent6# strace -ff holland bk 2>&1 | grep -c EBADF 666652

cent8# yum install epel-release cent8# yum install mariadb-server mariadb holland-mysqldump strace cent8# systemctl start mariadb cent8# holland mc mysqldump -f /etc/holland/backupsets/default.conf cent8# holland --version 1.2.1 cent8# strace -ff holland bk 2>&1 | grep -c EBADF 2 cent8# ulimit -n 33333 cent8# strace -ff holland bk 2>&1 | grep -c EBADF 2 cent8# strace -ff holland bk 2>&1 | grep EBADF [pid 29595] ioctl(-1, TIOCGPGRP, 0x7ffef0ae5594) = -1 EBADF (Bad file descriptor) [pid 29599] fstat(-1, 0x7ffcdacbbde0) = -1 EBADF (Bad file descriptor)

[soulen3]

This seems to be related to how python2 is executing shell commands. You can test this by setting file-per-database = no and the number of errors drop considerably.

[root@ee4ae50b5338 /]# strace -f holland bk default 2>&1 | grep -c EBADF
133298
[root@ee4ae50b5338 /]# sed -i 's/file-per-database = yes/file-per-database = no/g' /etc/holland/backupsets/default.conf
[root@ee4ae50b5338 /]# strace -f holland bk default 2>&1 | grep -c EBADF
66650

The errors seems to happen while holland is waiting on external commands to complete.

As far as I can tell, this only effect python2 and doesn't prevent a successful backup. As python2 is EOL, I'm not planning on fixing it. Let me know if this is breaking something and I'm missing it.

[soulen3]

Appears to be this bug in the subprocess library: https://bugs.python.org/issue35757

[soulen3]

Confirmed that using the subprocess32 library will fix this