Open Sean58238 opened 5 years ago
Confirm via SHOW ENGINES that RocksDB is enabled ... 2019-07-29 21:33:54 4450 [Note] Plugin 'FEDERATED' is disabled. 2019-07-29 21:33:54 4450 [Note] Plugin 'ROCKSDB' is disabled. ...
Next, attach gdb to mysqld while the test is running to get a stack trace from the crash.
We used rocksDB engine and innoDB , both of them met this issue , so we think it maybe a problem about MySQL. We found this issue happened on branch "fb-mysql-5.6.35" , we also test for "fb-mysql-8.0.13" and "fb-prod201903" , this issue not happened. We attach gbd to mysqld , it seems create LWP failed.
Error Report for Facebook MySQL 5.6's Crashing during sysbench Testing
installation environment
CentOS Linux release 7.6.1810 (Core) Linux version 4.19.0-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Mon Oct 22 10:40:32 EDT 2018 MemTotal: 196719476 kB
Facebook MySQL's version:
branch: fb-mysql-5.6.35 commit: df34666abb743015c6039e4a440cfbe10f645114 We also tried for later commits (822bfe2cb935051b626db579d9e94731ae59f244), and searched for correlated crash information. However, we had not found any solution to this problem.
Facebook MySQL's installation process
git clone https://github.com/facebook/mysql-5.6.git cd mysql-5.6 git submodule init git submodule update
cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_SSL=system \ -DWITH_ZLIB=bundled -DMYSQL_MAINTAINER_MODE=0 -DENABLED_LOCAL_INFILE=1 \ -DENABLE_DTRACE=0 -DCMAKE_CXX_FLAGS="-march=native"
make -j make install
mkdir /data mkdir /data/mysql
vim /etc/my.cnf [client] default-character-set=utf8 port=3306 user=mysql socket=/data/mysql/mysql.sock
cd /usr/local/mysql ./scripts/mysql_install_db --defaults-file=/etc/my.cnf
chgrp -R mysql /data/mysql/ chown -R mysql /data/mysql/
mysqld_safe --defaults-file=/etc/my.cnf
Testing tools (sysbench)
version: sysbench 1.0.9 we use sysbench to do Testing
Testing command
first step is to prepare
./bin/sysbench \ --num-threads=100 \ --test=/home/XXX/sysbench/tests/include/oltp_legacy/oltp.lua \ --oltp-table-size=2180000 \ --oltp-tables-count=10 \ --oltp-point-selects=1 \ --oltp-simple-ranges=0 \ --oltp-sum-ranges=0 \ --oltp-order-ranges=0 \ --oltp-index-updates=1 \ --oltp-non-index-updates=0 \ --report-interval=1 \ --mysql-user=root \ --mysql-password='' \ --mysql-host='XXX' \ --mysql-port=3306 \ --mysql-db=summertest \ --max-time=900 \ --max-requests=0 \ --oltp_secondary=on \ --oltp_auto_inc=off prepare
prepare statement can run successfully
2nd step is to insert ./src/sysbench \ --num-threads=100 \ --test=/home/XXX/sysbench/tests/include/oltp_legacy/insert.lua \ --oltp-table-size=2180000 \ --oltp-tables-count=10 \ --percentile=99 \ --report-interval=1 \ --mysql-host='XXX' \ --mysql-user=root \ --mysql-password='' \ --mysql-db=summertest \ --max-time=300 \ --max-requests=0 \ --mysql-port=3306 \ --oltp_secondary=on \ --oltp_auto_inc=off \ run
but the insert statement will cause Facebook MySQL to have error
Error log for Facebook's MySQL (from /var/log/error.log)
Output for sysbench
Additional information
We have ensured that there was enough memory for the program and unlimited memory for each process. We installed such Facebook MySQL version on 2 CentOS machines and 1 ubuntu machine, all of them had such issue. The crash did not always happen. Sometimes it ran successfully, but other times it crashed. Even if we adjusted key_buffer_size, read_buffer_size, and other buffer size, the crashing still happened. If we adjusted the number of threads to be smaller, or the table size to be smaller, the possibility of crashing would decrease.