session-replay-tools / mysql-sgt-replay-module

A tcpcopy module for replaying MySQL sessions with skip-grant-table enabled.
Other
12 stars 5 forks source link

MYSQL【mysql-sgt-replay-module】离线回放不成功 #4

Closed wangjuan08 closed 4 weeks ago

wangjuan08 commented 7 years ago

你好,我们在使用mysql流量回放时,测试机器上没有流量。 以下是相关信息,辛苦帮忙看下 Tcpcopy版本:1.0 工作模式:离线回放

机器分配: 10.48.166.122 安装了tcpcopy-client,不是线上数据库机器, 编译: ./configure --offline --with-debug --set-protocol-module=mysql-sgt-replay-module make make install 配置: conf/plugin.conf及plugin.conf.default配置user usercount@pwd; 权限同线上用户账号 启动命令: tcpcopy -i /home/work/ent1.cap -x 58885-10.135.3.139:58885 -s 10.135.3.163 -c 10.48.166.x 说明:在ent1.cap抓包时,脚本方式命令行登陆线上数据库。

10.135.3.139 目标机器
测试数据库以skip-grant-tables方式启动 路由配置 route add -net 10.48.166.0 netmask 255.255.255.0 gw 10.135.3.163 image

10.135.3.163 辅助机器 ./configure --with-resp-payload --with-debug make make install 启动命令: intercept -i eth0 -F 'tcp and src port 58885'

日志信息及抓包信息: Tcpcopy.log: 2017/02/23 20:11:36 +576 [notice] tcpcopy version:1.0.0 2017/02/23 20:11:36 +576 [notice] tcpcopy internal version:6 2017/02/23 20:11:36 +576 [notice] target:58885-10.135.3.139:58885 2017/02/23 20:11:36 +576 [notice] TC_OFFLINE mode 2017/02/23 20:11:36 +576 [notice] TC_COMBINED mode 2017/02/23 20:11:36 +576 [notice] TC_PLUGIN mode 2017/02/23 20:11:36 +576 [notice] epoll mode 2017/02/23 20:11:36 +576 [notice] TC_DETECT_MEMORY is true 2017/02/23 20:11:36 +576 [notice] session timeout:120 2017/02/23 20:11:36 +576 [notice] keepalive timeout:240 2017/02/23 20:11:36 +576 [notice] min sess pool size:592 2017/02/23 20:11:36 +576 [notice] sess pool size:1024 2017/02/23 20:11:36 +576 [notice] set global port for tcpcopy 2017/02/23 20:11:36 +576 [notice] raw_clt_tf_ip:10.48.166.x 2017/02/23 20:11:36 +576 [debug] clt ip addr:10.48.166.1 … 2017/02/23 20:11:36 +576 [debug] clt ip addr:10.48.166.254 2017/02/23 20:11:36 +576 [notice] parallel connections per target:2 2017/02/23 20:11:36 +576 [notice] accelerated 1 times,interval:0 ms 2017/02/23 20:11:36 +576 [notice] s parameter:10.135.3.163 2017/02/23 20:11:36 +576 [notice] set only ip for tcpcopy 2017/02/23 20:11:36 +576 [debug] pool:0x22fa370, add timer:0x22fa414 2017/02/23 20:11:36 +576 [debug] pool:0x22fa370, add timer:0x22fa464 2017/02/23 20:11:36 +576 [info] pool:0x23033a0,block:0x23073b0,len:44 occupy 2017/02/23 20:11:36 +576 [info] pool:0x23033a0,block:0x23073b0,len:44 occupy 2017/02/23 20:11:36 +576 [info] connect to remote server(10.135.3.163:36524) 2017/02/23 20:11:36 +576 [info] connect to remote server(10.135.3.163:36524) 2017/02/23 20:11:36 +576 [notice] open pcap success:/home/work/ent1.cap 2017/02/23 20:11:36 +576 [notice] send the first packets here 2017/02/23 20:11:36 +576 [debug] fir auth packet is null … 2017/02/23 20:11:41 +365 [info] pl:43133728, p:41469 2017/02/23 20:11:41 +365 [debug] pool:0x2922b20, add timer:0x2922c94 2017/02/23 20:11:41 +365 [info] nev:43134076,p:41469 2017/02/23 20:11:41 +365 [debug] pool:0x2922b20, add timer:0x2922ce4 2017/02/23 20:11:41 +365 [notice] session key:20504979897761, p:41469 2017/02/23 20:11:41 +365 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 48,seq=3768542767,ack=0 2017/02/23 20:11:41 +365 [info] ln:43134396, pkt:43134324, save:3768542767,p:41469 2017/02/23 20:11:41 +365 [info] slide_win_packs size:1, p:41469 2017/02/23 20:11:41 +365 [debug] proc clt pack:41469 2017/02/23 20:11:41 +365 [debug] record rtt base:1487834411298,p:41469 2017/02/23 20:11:41 +365 [debug] syn port:41469 2017/02/23 20:11:41 +365 [debug] to bak:10.48.166.18:41469-->10.135.3.139:58885,len 48,seq=3768542767,ack=0 2017/02/23 20:11:41 +365 [info] empty slide,p:41469 2017/02/23 20:11:41 +365 [debug] pool:0x22fa370, up timer:0x22fa50c 2017/02/23 20:11:41 +392 [debug] del timer:0x22fa50c 2017/02/23 20:11:42 +040 [debug] del timer:0x22fa50c 2017/02/23 20:11:42 +040 [debug] pool:0x22fa370, up timer:0x22fa50c 2017/02/23 20:11:42 +051 [debug] del timer:0x22fa50c 2017/02/23 20:11:42 +051 [debug] pool:0x22fa370, up timer:0x22fa50c 2017/02/23 20:11:42 +061 [debug] del timer:0x22fa50c 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 40,seq=3768542768,ack=1280173344 2017/02/23 20:11:42 +061 [info] ln:43134500, pkt:43134436, save:3768542768,p:41469 2017/02/23 20:11:42 +061 [info] rtt:0,p:41469 2017/02/23 20:11:42 +061 [info] internal:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:2, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] fir auth packet is null 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 40,seq=3768542768,ack=1280173434 2017/02/23 20:11:42 +061 [info] ln:43134604, pkt:43134540, save:3768542768,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:3, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] fir auth packet is null 2017/02/23 20:11:42 +061 [debug] fir auth packet is null 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 150,seq=3768542768,ack=1280173434 2017/02/23 20:11:42 +061 [info] ln:43134644, pkt:43134828, save:3768542768,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:4, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 77,seq=3768542878,ack=1280173445 2017/02/23 20:11:42 +061 [info] ln:43134684, pkt:43134996, save:3768542878,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:5, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] fir auth packet is null 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 57,seq=3768542915,ack=1280173579 2017/02/23 20:11:42 +061 [info] ln:43135172, pkt:43135092, save:3768542915,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:6, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 45,seq=3768542932,ack=1280173657 2017/02/23 20:11:42 +061 [info] ln:43135276, pkt:43135212, save:3768542932,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:7, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 40,seq=3768542937,ack=1280173657 2017/02/23 20:11:42 +061 [info] ln:43135380, pkt:43135316, save:3768542937,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:8, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] recv clt:10.48.166.18:41469-->10.126.124.199:58885,len 40,seq=3768542938,ack=1280173658 2017/02/23 20:11:42 +061 [info] ln:43135484, pkt:43135420, save:3768542938,p:41469 2017/02/23 20:11:42 +061 [info] slide_win_packs size:9, p:41469 2017/02/23 20:11:42 +061 [debug] proc clt pack:41469 2017/02/23 20:11:42 +061 [debug] fir auth packet is null 2017/02/23 20:11:42 +061 [debug] fir auth packet is null 2017/02/23 20:11:42 +061 [debug] fir auth packet is null … 日志间歇性地提示fir auth packet is null,然后是一些info的信息 还有一些warn信息: 2017/02/23 20:15:16 +136 [debug] del timer:0x22fa50c 2017/02/23 20:15:16 +136 [debug] pool:0x22fa370, up timer:0x22fa50c 2017/02/23 20:15:16 +136 [warn] sig 15 received 2017/02/23 20:15:16 +136 [notice] active:6,rel:295,obs del:0,tw:5 2017/02/23 20:15:16 +136 [notice] conns:0,resp:0,c-resp:0 2017/02/23 20:15:16 +136 [notice] resp fin:0,resp rst:0 2017/02/23 20:15:16 +136 [notice] send:601,send content:0 2017/02/23 20:15:16 +136 [notice] send syn:301, fin:0,reset:300 2017/02/23 20:15:16 +136 [notice] reconnect:0,for no syn:0 2017/02/23 20:15:16 +136 [notice] retransmit:0 2017/02/23 20:15:16 +136 [notice] recv packs after retransmission:0 2017/02/23 20:15:16 +136 [notice] syn cnt:301,all clt:1844592,clt cont:915603 2017/02/23 20:15:16 +136 [notice] total cont retransmit:0, frag:0 2017/02/23 20:15:16 +136 [notice] total captured packets:2928866 2017/02/23 20:15:16 +136 [warn] many connections can't be established 2017/02/23 20:15:16 +136 [info] session table, size:65536, total:6 2017/02/23 20:15:16 +136 [info] sess packs in swin:1,p:32809 2017/02/23 20:15:16 +136 [debug] sess post disp:32809 2017/02/23 20:15:16 +136 [debug] send passive rst:32809 2017/02/23 20:15:16 +136 [debug] to bak:10.48.166.42:32809-->10.135.3.139:58885,len 40,seq=1885657975,ack=0

10.48.166.122[tcpcopy-client]上抓包如下: Tcpdump –i any port 58885 10.126.215.x 是线上数据库机器 image 测试数据库抓包为 image 辅助机器上的intercept.log信息 image 问题:

  1. 多次进行抓包,并进行回放,一直回放不成功。请辛苦帮忙看下
  2. 每次抓包过程中都有命令行登陆数据库的操作,还是提示fir auth packet is null,总觉得认证信息特别不好捕获。之前在线下也搭建一套,在您的建议下,可以正常回放,但是也是认证的问题造成回放成功与否不能保证,时好时坏。 由于项目本身性质,需要使用线上机器搭建一套,环境一直不行,困扰好久,一直没有找到解决办法。 您在使用mysql回放时,有更好的捕获认证信息的方式吗
  3. 因为线上数据库本身流量很大,为了减少对线上影响,我们采取了离线模式的。不知道我们的理解对不对,或者实时模式也是可以用的。
wangbin579 commented 7 years ago

不好意思,现在才看到这个问题。

目前你的主要问题比较多: 1、intercept configure的时候按照( https://github.com/session-replay-tools/mysql-sgt-replay-module ),不需要设置--with-resp-payload,那是为常规mysql回放用的

2、路由生效的问题 从测试服务器抓包来看,syn数据包到达了测试机器,测试机器的tcp也回复了第二次握手数据包,但很快就收到了rst数据包,rst数据包和第二次握手数据包之间的时间间隔很短,不太可能是tcpcopy发送的。 建议在10.135.3.163上面,看看能不能抓到路由过来的第二次握手数据包,如果抓不到,说明路由设置没有生效或者被环境给干掉了