Freyayan
commented
7 years ago 不过系统日志中也出现下面报错，不知是不是磁盘坏块了
Nov 10 10:47:56 smartd[4475]: Device: /dev/sdb [SAT], 1016 Currently unreadable (pending) sectors (changed +1) Nov 10 10:47:56 smartd[4475]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 200 to 193 Nov 10 10:47:58 smartd[4475]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 200 to 206 Nov 10 10:47:59 smartd[4475]: Device: /dev/sdf [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 200 to 193 Nov 10 11:17:56 smartd[4475]: Device: /dev/sdb [SAT], 1017 Currently unreadable (pending) sectors (changed +1) Nov 10 11:17:56 smartd[4475]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 193 to 181 Nov 10 11:17:58 smartd[4475]: Device: /dev/sdf [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 193 to 200 Nov 10 11:18:00 smartd[4475]: Device: /dev/sdi [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 200 to 206 Nov 10 11:32:48 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:32:48 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:32:48 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:32:48 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:32:48 kernel: sd 0:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 10 11:32:48 kernel: sd 0:0:1:0: [sdb] Sense Key : Medium Error [current] Nov 10 11:32:48 kernel: Info fld=0x40f91bc Nov 10 11:32:48 kernel: sd 0:0:1:0: [sdb] Add. Sense: Unrecovered read error Nov 10 11:32:48 kernel: sd 0:0:1:0: [sdb] CDB: Read(10): 28 00 04 0f 91 a2 00 00 20 00 Nov 10 11:32:48 kernel: end_request: critical medium error, dev sdb, sector 68129202 Nov 10 11:33:26 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:33:26 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:33:26 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:33:26 kernel: mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) Nov 10 11:33:26 kernel: sd 0:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 10 11:33:26 kernel: sd 0:0:1:0: [sdb] Sense Key : Medium Error [current] Nov 10 11:33:26 kernel: Info fld=0x40f91bc Nov 10 11:33:26 kernel: sd 0:0:1:0: [sdb] Add. Sense: Unrecovered read error Nov 10 11:33:26 kernel: sd 0:0:1:0: [sdb] CDB: Read(10): 28 00 04 0f 91 a2 00 00 20 00 Nov 10 11:33:26 kernel: end_request: critical medium error, dev sdb, sector 68129202 Nov 10 11:47:56 smartd[4475]: Device: /dev/sdb [SAT], 1018 Currently unreadable (pending) sectors (changed +1) Nov 10 11:47:56 smartd[4475]: Device: /dev/sdb [SAT], SMART Prefailure Attribute: 5 Reallocated_Sector_Ct changed from 56 to 55 Nov 10 11:47:56 smartd[4475]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 181 to 200 Nov 10 11:47:57 smartd[4475]: Device: /dev/sdb [SAT], ATA error count increased from 26 to 28 Nov 10 11:47:58 smartd[4475]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 206 to 200
wodesuck
你好， 我在用sysbench进行otlp.lua进行性能测试的时候，发现一个备机几点挂掉，前面sysbench准备数据是ok的，但是连接测试时出现备机节点mysql挂掉。
[root@hongxin59 tools]# ps -ef |grep phxsql root 41128 1 2 Nov09 ? 00:45:02 /cache1/phxsql/sbin/phxbinlogsvr_phxrpc root 41141 1 9 Nov09 ? 02:21:49 /cache1/phxsql/sbin/phxsqlproxy_phxrpc /cache1/phxsql/etc/phxsqlproxy.conf daemon root 41142 1 8 Nov09 ? 02:20:27 /cache1/phxsql/sbin/phxsqlproxy_phxrpc /cache1/phxsql/etc/phxsqlproxy.conf daem
sysbench测试脚本： sysbench --test=oltp --test=/cache1/sysbench/sysbench/tests/db/oltp.lua --mysql-host=180.101.24.57 --mysql-user=root --mysql-port=54321 --mysql-db=sbtest --oltp-tables-count=10 --oltp-table-size=1000000 --num-threads=200 --max-requests=100000 --report-interval=1 --max-time=600 --oltp-read-only=off run>>./sysbench200oltp.log
日志： 2016-11-10 10:41:43 42526 [ERROR] Error reading packet from server: Lost connection to MySQL server during query (server_errno=20 13) 2016-11-10 10:41:43 42526 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.000003' at pos ition 2662 2016-11-10 10:41:43 42526 [Warning] Storing MySQL user name or password information in the master info repository is not secure a nd is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START S LAVE Syntax' in the MySQL Manual for more information. 2016-11-10 11:32:48 42526 [ERROR] InnoDB: Tried to read 16384 bytes at offset 161677312. Was only able to read 0. 2016-11-10 11:32:48 7fa152dac700 InnoDB: Operating system error number 5 in a file operation. InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2016-11-10 11:32:48 42526 [ERROR] InnoDB: File (unknown): 'read' returned OS error 105. Cannot continue operation 161110 11:32:48 mysqld_safe Number of processes running now: 0 161110 11:32:48 mysqld_safe mysqld restarted 2016-11-10 11:32:48 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 2016-11-10 11:32:48 0 [Note] /cache1/phxsql/sbin/mysqld (mysqld 5.6.31-77.0-log) starting as process 31753 ... 2016-11-10 11:32:48 31753 [Note] InnoDB: Using atomics to ref count buffer pool pages 2016-11-10 11:32:48 31753 [Note] InnoDB: The InnoDB memory heap is disabled 2016-11-10 11:32:48 31753 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-11-10 11:32:48 31753 [Note] InnoDB: Memory barrier is not used 2016-11-10 11:32:48 31753 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-11-10 11:32:48 31753 [Note] InnoDB: Using CPU crc32 instructions 2016-11-10 11:32:48 31753 [Note] InnoDB: Initializing buffer pool, size = 200.0M 2016-11-10 11:32:48 31753 [Note] InnoDB: Completed initialization of buffer pool 2016-11-10 11:32:48 31753 [Note] InnoDB: Opened 3 undo tablespaces 2016-11-10 11:32:48 31753 [Note] InnoDB: Highest supported file format is Barracuda. 2016-11-10 11:32:48 31753 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3664293153 2016-11-10 11:32:48 31753 [Note] InnoDB: Database was not shutdown normally! 2016-11-10 11:32:48 31753 [Note] InnoDB: Starting crash recovery. 2016-11-10 11:32:48 31753 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-11-10 11:32:48 31753 [Note] InnoDB: Restoring possible half-written data pages 2016-11-10 11:32:48 31753 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 3666856472 InnoDB: Transaction 17522 was in the XA prepared state. InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 0 row operations to undo InnoDB: Trx id counter is 17920 2016-11-10 11:32:50 31753 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 490071419, file name mysql-bin.000005 InnoDB: Starting in background the rollback of uncommitted transactions 2016-11-10 11:33:22 31753 [Note] InnoDB: 128 rollback segment(s) are active. 2016-11-10 11:33:22 7f083ca65700 InnoDB: Rollback of non-prepared transactions completed 2016-11-10 11:33:22 31753 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.31-77.0 started; log sequence number 366685 6472 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_cache_size=256m' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_read_status_frequency=1' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_commit_sync=0' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_read_block_size=128K' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_write_status_frequency=1' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_log_dir=//home/root/data/phxsql/pe rcona.workspace/tokudb' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_directio=1' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_fsync_log_period=1000' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_row_format=tokudb_fast' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_read_buf_size=128K' 2016-11-10 11:33:22 31753 [Warning] /cache1/phxsql/sbin/mysqld: unknown variable 'loose_tokudb_data_dir=/home/root/data/phxsql/pe rcona.workspace/tokudb' 2016-11-10 11:33:22 31753 [Note] get last uuid f8ec778f-a625-11e6-8e48-90e2ba253bd0 done, last gtid [] 2016-11-10 11:33:22 31753 [Note] get last send gtid svrid [f8ec778f-a625-11e6-8e48-90e2ba253bd0] ret 0, last gtid [] 2016-11-10 11:33:22 31753 [Note] is_gtid_in_binlog_file read [/cache1/phxdata/percona.workspace/binlog/mysql-bin.000005] ret 0 2016-11-10 11:33:23 31753 [Note] [/cache1/phxdata/percona.workspace/binlog/mysql-bin.000005] read event null 2016-11-10 11:33:23 31753 [Note] truncate from [/cache1/phxdata/percona.workspace/binlog/mysql-bin.000005], ret 1 2016-11-10 11:33:23 31753 [Note] truncate_while_init ret 0, m_last_gtid_in_binlog 1fd15947-a623-11e6-8e36-001b21b92df0:7462 2016-11-10 11:33:23 31753 [Note] repl_phx_report_binlog_before_recovery:90 server_uuid [f8ec778f-a625-11e6-8e48-90e2ba253bd0] log _bin_index [/cache1/phxdata/percona.workspace/binlog/mysql-bin.index] ret 0 2016-11-10 11:33:23 31753 [Note] Recovering after a crash using /cache1/phxdata/percona.workspace/binlog/mysql-bin 2016-11-10 11:33:24 31753 [Note] Starting crash recovery... 2016-11-10 11:33:24 7f0877269720 InnoDB: Starting recovery for XA transactions... 2016-11-10 11:33:24 7f0877269720 InnoDB: Transaction 17522 in prepared state after recovery 2016-11-10 11:33:24 7f0877269720 InnoDB: Transaction contains changes to 5 rows 2016-11-10 11:33:24 7f0877269720 InnoDB: 1 transactions in prepared state after recovery 2016-11-10 11:33:24 31753 [Note] Found 1 prepared transaction(s) in InnoDB 2016-11-10 11:33:26 31753 [ERROR] InnoDB: Tried to read 16384 bytes at offset 161677312. Was only able to read 0. 2016-11-10 11:33:26 7f084adac700 InnoDB: Operating system error number 5 in a file operation. InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2016-11-10 11:33:26 31753 [ERROR] InnoDB: File (unknown): 'read' returned OS error 105. Cannot continue operation 161110 11:33:26 mysqld_safe mysqld from pid file /cache1/phxdata/percona.workspace/data/percona.pid ended
请问这是什么原因，有什么解决办法？？