huhongbo / devops

6 stars 0 forks source link

influxdb/mysql/tokudb在单表插入性能和读取性能比较 #23

Open huhongbo opened 9 years ago

aricy commented 9 years ago

InfluxDB是一个开源分布式的时序、事件和指标数据库。InfluxDB使用Go语言编写,无需外部依赖。其设计目标是实现分布式和水平伸缩扩展。

在InfluxDB数据库中,一切都是基于时间序列的,同时你仍然可以执行那些标准的函数,比如求最小数min、求最大数max、求和sum、统计个数count、求平均值mean、求中位数median、求百分比percentile等等。

InfluxDB数据库可以在任意时间间隔收集数据并完成计算汇总,它可以跟踪上百个指标到上百万个指标。

InfluxDB数据库的数据模型支持任意的事件数据。

InfluxDB数据库内建了基于HTTP的API,因此通过JavaScript读写数据非常方便。

InfluxDB数据库自带了管理工具,以可视化的方式管理数据库、用户及权限等。

aricy commented 9 years ago

这个好像跟我们常见的数据库测试不太一样,用的是GO语言编写的,而且都是基于WEB操作的 influxdb

aricy commented 9 years ago

现在的PerconaDB这个与MariDB挺好的,是不是考虑测试这两种数据库的性能。

aricy commented 9 years ago

通过PerconaDB与Mysql的测试性能优势比较明显,后续再测试MariDB等。 Mysql Version : 5.6.22 Percona Version : 5.6.22-71.0 end_percona_tpcc_all

aricy commented 9 years ago

下面是percona、mysql、maridb三者的TPCC对比 end_percoan_mysql_maridb_all

aricy commented 9 years ago

版本信息: Mysql : 5.6.22 Percona : 5.6.22-71.0 MariDB : 10.0.16 image 综合比较,MariDB在TPCC和Tpmc上较其他两种数据库好

aricy commented 9 years ago

通过对MariDB参数的调整,将压力提升至较高级别后,TPCC与Tpmc(Tpmc: 14373.767)有很大的提升,TPCC有较之前默认参数的3倍多。 maridb_op

aricy commented 9 years ago

而同样参数测试Percona后发现已经无法在该压力下测试,表现为重启mysqld进程,后台error显示OUT OF Memory,但改小内存后还是同样的报错, key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=32 max_threads=2102 thread_count=32 connection_count=32 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8918013 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. 日志信息如下: wangbin1:/mysql/tpcc-mysql # nohup sh tpcc_mysql_2.sh & [2] 16058 wangbin1:/mysql/tpcc-mysql # nohup: ignoring input and appending output to `nohup.out'

wangbin1:/mysql/tpcc-mysql # /usr/local/perconamysql/bin/mysqld_safe: line 173: 15928 Aborted nohup /usr/local/perconamysql/bin/mysqld --basedir=/usr/local/perconamysql --datadir=/usr/local/perconamysql/data --plugin-dir=/usr/local/perconamysql/lib/mysql/plugin --user=mysql --log-error=/usr/local/perconamysql/data/wangbin1.err --pid-file=/usr/local/perconamysql/data/wangbin1.pid --socket=/tmp/mysql.sock --port=3306 < /dev/null >> /usr/local/perconamysql/data/wangbin1.err 2>&1 >> /usr/local/perconamysql/data/wangbin1.err 2>&1 150204 15:57:56 mysqld_safe Number of processes running now: 0 150204 15:57:56 mysqld_safe mysqld restarted


option h with value 'localhost' option P with value '3306' option d with value 'tpcc1000' option u with value 'root' option w with value '100' option c with value '32' option r with value '60' option l with value '3600'

[server]: localhost [port]: 3306 [DBname]: tpcc1000 [user]: root [pass]: ``` [rampup]: 60 (sec.) [measure]: 3600 (sec.) ``` RAMP-UP TIME.(60 sec.) error at thread_main error at thread_main error at thread_main 可能也是碰到BUG了,无法继续测试。
aricy commented 9 years ago

主进程一重启,子进程已经无法获取事务信息

MEASURING START.

10, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 20, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 30, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 40, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 50, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 60, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 70, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 80, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 90, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 100, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 110, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 120, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 130, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 140, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 150, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000 160, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000, 0(0):0.000|0.000

aricy commented 9 years ago

option h with value 'localhost' option P with value '3306' option d with value 'tpcc1000' option u with value 'root' option w with value '100' option c with value '32' option r with value '60' option l with value '3600'

[server]: localhost [port]: 3306 [DBname]: tpcc1000 [user]: root [pass]: ``` [rampup]: 60 (sec.) [measure]: 3600 (sec.) ``` ![image](https://cloud.githubusercontent.com/assets/10055186/6055001/2e37ac42-ad31-11e4-8918-82167571b319.png) ![image](https://cloud.githubusercontent.com/assets/10055186/6055025/d542d5fc-ad31-11e4-950b-fbfa96d2e4da.png)
huhongbo commented 9 years ago

influxdb是一种时间序列的nosql数据库,采用常见的json和rest接口,你可以学习下json和rest的方法,他有天生的对于时间序列存储的优势,所以想看看和传统的sql数据库的性能差别

aricy commented 9 years ago

下面为ONESQL与MariaDB的测试,使用tpcc-mysql测试工具测试 测试使用root用户登录,tcc参数均为默认,结果如下: tpcc_mariadb_onesql

aricy commented 9 years ago

下面测试案例分3种场景,下面均关闭binlog下测试 1、将tcc_max_transaction_concurrency改为64,其他参数不动 2、将tcc_async_commit_mode参数改为1 3、使用test用户登录 结果如下: tpcc-onesql

aricy commented 9 years ago

由这两个测试看出: 1、第一个测试,ONESQL默认参数下,在稳定后MariaDB性能比ONESQL要好。 2、第二个测试,在调整成并发64(默认为40)后,性能与MariaDB有所接近;打开ASYNC提交参数后,ONESQL性能稍有提高,不过还是较MariaDB差些;而使用test用户登录执行后,性能有明显下降,与楼的文档中“队 列补丁设置了对“ root”用户发起的 SQL 操作不进行任何限流的机制”相一致,test用户登录执行TPCC,通过"show processlist和show engine innodb status\G"查看后,事务锁基本没有,或非常少,且通过工具oneagent采集后,可以看到: root用户登录: image test用户登录: image 处理效率test明显比root用户差许多。

aricy commented 9 years ago

image

基本与TPCC测试结果一致。

aricy commented 9 years ago

测试中进程信息: db_process 测试中CPU信息: pre_cpu_info