search

scylladb / seastar

High performance server-side application framework
http://seastar.io
Apache License 2.0
8.28k stars 1.54k forks source link

Memcached is not working with DPDK on Intel servers #25

Open slivne opened 9 years ago

slivne commented 9 years ago

DPDK 1.8.0 seastar : 89763c95c92b44d3482e7500ddd5c14b8c47b76en also seastar: 29366cb07683e7b12a849e3058681b8b911942c1 (same results)

seastar httpd with dpdk does work on the servers

''' sudo build/release/apps/httpd/httpd --network-stack nave --dhcp 0 --host-ipv4-addr 192.168.20.101 --netmask-ipv4-addr 255.255.255.0 --gw-ipv4-addr 192.168.20.185 --smp 1 --dpdk-pmd ''' from packet 75 it seems that seastar has lost network connectivity

after memaslap ends send pings confirms this

I'll attach a wireshark capture [shlomi@dpdk2 ~]$ sudo tshark -i enp129s0 -r /tmp/memached_capture.cap Running as user "root" and group "root". This could be dangerous. 1 0.000000000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 2 0.000032000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 3 1.000040000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 4 1.000068000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 5 2.000068000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 6 2.000098000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 7 3.000085000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 8 3.000112000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 9 4.000116000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 10 4.000145000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 11 5.000132000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 12 5.000158000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 13 6.000160000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 14 6.000192000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 15 7.000172000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 16 7.000201000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 17 8.000211000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages 18 8.000240000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages 19 8.279197000 192.168.20.185 -> 192.168.20.101 TCP 74 51693→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032647 TSecr=0 WS=128 20 8.279263000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51693 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 21 8.279337000 192.168.20.185 -> 192.168.20.101 TCP 54 51693→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 22 8.280717000 192.168.20.185 -> 192.168.20.101 TCP 74 51694→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032649 TSecr=0 WS=128 23 8.280774000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51694 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 24 8.280822000 192.168.20.185 -> 192.168.20.101 TCP 54 51694→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 25 8.281946000 192.168.20.185 -> 192.168.20.101 TCP 74 51695→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032650 TSecr=0 WS=128 26 8.281992000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51695 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 27 8.282038000 192.168.20.185 -> 192.168.20.101 TCP 54 51695→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 28 8.283147000 192.168.20.185 -> 192.168.20.101 TCP 74 51696→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032651 TSecr=0 WS=128 29 8.283192000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51696 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 30 8.283233000 192.168.20.185 -> 192.168.20.101 TCP 54 51696→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 31 8.284336000 192.168.20.185 -> 192.168.20.101 TCP 74 51697→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032652 TSecr=0 WS=128 32 8.284380000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51697 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 33 8.284430000 192.168.20.185 -> 192.168.20.101 TCP 54 51697→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 34 8.285531000 192.168.20.185 -> 192.168.20.101 TCP 74 51698→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032654 TSecr=0 WS=128 35 8.285576000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51698 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 36 8.285618000 192.168.20.185 -> 192.168.20.101 TCP 54 51698→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 37 8.286725000 192.168.20.185 -> 192.168.20.101 TCP 74 51699→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032655 TSecr=0 WS=128 38 8.286761000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51699 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 39 8.286802000 192.168.20.185 -> 192.168.20.101 TCP 54 51699→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 40 8.287924000 192.168.20.185 -> 192.168.20.101 TCP 74 51700→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032656 TSecr=0 WS=128 41 8.287966000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51700 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 42 8.288007000 192.168.20.185 -> 192.168.20.101 TCP 54 51700→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 43 8.289107000 192.168.20.185 -> 192.168.20.101 TCP 74 51701→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032657 TSecr=0 WS=128 44 8.289152000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51701 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 45 8.289194000 192.168.20.185 -> 192.168.20.101 TCP 54 51701→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 46 8.290280000 192.168.20.185 -> 192.168.20.101 TCP 74 51702→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032658 TSecr=0 WS=128 47 8.290325000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51702 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 48 8.290366000 192.168.20.185 -> 192.168.20.101 TCP 54 51702→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 49 8.291402000 192.168.20.185 -> 192.168.20.101 TCP 74 51703→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032659 TSecr=0 WS=128 50 8.291442000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51703 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 51 8.291483000 192.168.20.185 -> 192.168.20.101 TCP 54 51703→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 52 8.292512000 192.168.20.185 -> 192.168.20.101 TCP 74 51704→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032661 TSecr=0 WS=128 53 8.292557000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51704 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 54 8.292598000 192.168.20.185 -> 192.168.20.101 TCP 54 51704→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 55 8.293634000 192.168.20.185 -> 192.168.20.101 TCP 74 51705→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032662 TSecr=0 WS=128 56 8.293679000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51705 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 57 8.293720000 192.168.20.185 -> 192.168.20.101 TCP 54 51705→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 58 8.294760000 192.168.20.185 -> 192.168.20.101 TCP 74 51706→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032663 TSecr=0 WS=128 59 8.294805000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51706 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 60 8.294846000 192.168.20.185 -> 192.168.20.101 TCP 54 51706→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 61 8.295890000 192.168.20.185 -> 192.168.20.101 TCP 74 51707→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032664 TSecr=0 WS=128 62 8.295926000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51707 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 63 8.295967000 192.168.20.185 -> 192.168.20.101 TCP 54 51707→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 64 8.297002000 192.168.20.185 -> 192.168.20.101 TCP 74 51708→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032665 TSecr=0 WS=128 65 8.297047000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51708 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128 66 8.297089000 192.168.20.185 -> 192.168.20.101 TCP 54 51708→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0 67 8.297145000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 68 8.297168000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 69 8.297184000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 70 8.297198000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 71 8.297215000 192.168.20.101 -> 192.168.20.185 MEMCACHE 62 STORED 72 8.297222000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 73 8.297224000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 74 8.297225000 192.168.20.101 -> 192.168.20.185 MEMCACHE 62 STORED 75 8.297235000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024 76 8.297243000 192.168.20.185 -> 192.168.20.101 TCP 54 51694→11211 [ACK] Seq=1106 Ack=9 Win=29312 Len=0 77 8.297457000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\020\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 78 8.297467000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 79 8.297475000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 80 8.297484000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 81 8.297493000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 82 8.297516000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 83 8.297529000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 84 8.297540000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 85 8.497421000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 86 8.497432000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 87 8.497437000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 88 8.497441000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 89 8.498412000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 90 8.498419000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 91 8.498424000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 92 8.498429000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020p\024\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 93 8.498434000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\020\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 94 8.498440000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 95 8.498445000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 96 8.900453000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 97 8.900467000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 98 8.900472000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 99 8.900477000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 100 8.901460000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 101 8.901475000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 102 8.901480000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 103 8.901484000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\024\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 104 8.901489000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\020\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 105 8.901493000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 106 8.901498000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 107 9.707471000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 108 9.707485000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 109 9.707490000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 110 9.707494000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 111 9.707499000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 112 9.707503000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 113 9.707508000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 114 9.707522000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 115 9.707527000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 116 9.707533000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 117 9.707538000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 118 9.707543000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 119 9.707555000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 120 9.707560000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 121 11.319491000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 122 11.319506000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 123 11.319510000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 124 11.319515000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 125 11.319521000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 126 11.319525000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 127 11.319530000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 128 11.319545000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 129 11.319550000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 130 11.319557000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 131 11.319562000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 132 11.319567000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 133 11.319578000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 134 11.319583000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 135 11.535564000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020P\021\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 136 11.535583000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\360\021\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 137 11.535589000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\220\022\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 138 11.535594000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \0200\023\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 139 11.535599000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\320\023\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 140 11.535605000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020p\024\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 141 11.535610000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\020\025\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 142 11.535621000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020P\026\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h 143 14.543471000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 144 14.543487000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 145 14.543492000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 146 14.543497000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 147 14.543503000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 148 14.543507000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 149 14.543513000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 150 14.543527000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 151 14.543531000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 152 14.543536000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 153 14.543543000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 154 14.543547000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 155 14.543560000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 156 14.543565000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 157 20.991488000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 158 20.991503000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 159 20.991508000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 160 20.991513000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 161 20.991518000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 162 20.991522000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 163 20.991527000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 164 29.301230000 192.168.20.185 -> 192.168.20.101 TCP 54 51693→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 165 29.301290000 192.168.20.185 -> 192.168.20.101 TCP 54 51694→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 166 29.301312000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51695→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 167 29.301327000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51696→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 168 29.301341000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51697→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 169 29.301345000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51698→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 170 29.301348000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51699→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 171 29.301386000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51700→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 172 29.301388000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51701→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 173 29.301389000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51702→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 174 29.301391000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51703→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 175 29.301395000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51704→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 176 29.301608000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51705→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 177 29.301609000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51706→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 178 29.301611000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51707→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 179 29.301611000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51708→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0 180 33.871513000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 181 33.871530000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 182 33.871537000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 183 33.871545000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 184 33.871551000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 185 33.871558000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 186 33.871565000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 187 45.301172000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=1/256, ttl=64 188 46.300493000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=2/512, ttl=64 189 47.300532000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=3/768, ttl=64 190 48.300682000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=4/1024, ttl=64 191 49.300541000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=5/1280, ttl=64 192 50.300491000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=6/1536, ttl=64 193 50.319452000 IntelCor_2d:3a:00 -> IntelCor_27:d3:ea ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 194 51.300489000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=7/1792, ttl=64 195 51.321471000 IntelCor_2d:3a:00 -> IntelCor_27:d3:ea ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 196 52.300480000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=8/2048, ttl=64 197 52.323450000 IntelCor_2d:3a:00 -> IntelCor_27:d3:ea ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 198 53.300465000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=9/2304, ttl=64 199 54.300466000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 200 55.303461000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 201 56.305472000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 202 59.663499000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 203 60.665480000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185 204 61.667452000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185

slivne commented 9 years ago

https://drive.google.com/open?id=0B3W1iB9JfvzyejBxTlZ5XzgwaVE&authuser=0

asias commented 9 years ago

I used tcp_server rxrx test + debug print. DPDK were dropping some packets. Only seq=2925 and seq=11401 of the data packets are received and passed from eth to ip/tcp. We even did not see the other data packets at layer 2.

[asias@dpdk1 seastar]$  /home/asias/src/seastar/build/release/tests/tcp_server --smp 1 --network-stack native --host-ipv4-addr 192.168.20.101 --gw-ipv4-addr 192.168.20.185 --collectd 0 --tso on --memory 4096M --ufo off --dpdk-pmd
Seastar TCP server listening on port 10000 ...         
dpdk_pool 2560
dpdk_pool 2560
============== got eth: p.len=56
============== got eth: p.len=74
--------------ip rx p.len=60
to l4
tcp_received p.len()=40
tcp_received src_port=39231
============== got eth: p.len=56
--------------ip rx p.len=42
1: ip_len=40
to l4
tcp_received p.len()=20
tcp_received src_port=39231
============== got eth: p.len=58
--------------ip rx p.len=44
to l4
tcp_received p.len()=24
tcp_received src_port=39231
Got data for us
============== got eth: p.len=1230
--------------ip rx p.len=1216
to l4
tcp_received p.len()=1196
tcp_received src_port=39231
ooo port=39231, nr=0, seq=2925, seq_abs=4089585917
============== got eth: p.len=946
--------------ip rx p.len=932
to l4
tcp_received p.len()=912
tcp_received src_port=39231
ooo port=39231, nr=1, seq=11401, seq_abs=4089594393
dpdk_pool 2561                                                                                                    
dpdk_pool 2561
[asias@dpdk2 ~]$ go run client.go -host 192.168.20.101 -test rxrx  -conn 1
========== rxrx ============
Server:  192.168.20.101:10000
Connections:  1

[asias@dpdk2 ~]$ sudo tshark -i enp129s0 -c 100
Running as user "root" and group "root". This could be dangerous.
Capturing on 'enp129s0'
1 0.000000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
2 0.000087 IntelCor_27:d3:ea -> IntelCor_2d:3a:00 ARP 60 192.168.20.101 is at 68:05:ca:27:d3:ea
3 0.000103 192.168.20.185 -> 192.168.20.101 TCP 74 39231→10000 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=411871771 TSecr=0 WS=128
4 0.000175 192.168.20.101 -> 192.168.20.185 TCP 62 10000→39231 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
5 0.000243 192.168.20.185 -> 192.168.20.101 TCP 54 39231→10000 [ACK] Seq=1 Ack=1 Win=29312 Len=0
6 0.000336 192.168.20.185 -> 192.168.20.101 TCP 58 39231→10000 [PSH, ACK] Seq=1 Ack=1 Win=29312 Len=4
7 0.000374 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
8 0.000378 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=1465 Ack=1 Win=29312 Len=1460
9 0.000380 192.168.20.185 -> 192.168.20.101 TCP 1230 39231→10000 [PSH, ACK] Seq=2925 Ack=1 Win=29312 Len=1176
10 0.000401 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=4101 Ack=1 Win=29312 Len=1460
11 0.000403 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=5561 Ack=1 Win=29312 Len=1460
12 0.000404 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=7021 Ack=1 Win=29312 Len=1460
13 0.000406 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=8481 Ack=1 Win=29312 Len=1460
14 0.000407 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=9941 Ack=1 Win=29312 Len=1460
15 0.000409 192.168.20.185 -> 192.168.20.101 TCP 946 39231→10000 [PSH, ACK] Seq=11401 Ack=1 Win=29312 Len=892
16 0.000454 192.168.20.101 -> 192.168.20.185 TCP 60 10000→39231 [ACK] Seq=1 Ack=5 Win=3737600 Len=0
17 0.000492 192.168.20.101 -> 192.168.20.185 TCP 60 [TCP Dup ACK 16#1] 10000→39231 [ACK] Seq=1 Ack=5 Win=3737600 Len=0
18 0.000512 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=12293 Ack=1 Win=29312 Len=1460
19 0.000577 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=13753 Ack=1 Win=29312 Len=1460
20 0.000585 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=15213 Ack=1 Win=29312 Len=1460
21 0.200614 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
22 0.603570 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
23 1.408608 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
24 3.020611 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
25 6.244586 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
asias commented 9 years ago

Added more debug print in dpdk_qp::process_packets().

We did not see other data packets in dpdk_qp::process_packets() neither.

And we do did drop packet because of ip or tcp csum error. 

            if (m->ol_flags & (PKT_RX_IP_CKSUM_BAD | PKT_RX_L4_CKSUM_BAD)) {
                // Packet with bad checksum, just drop it.
                print("dpdk_qp drop pkt\n");
                continue;
            }
EAL:   0000:03:00.1 not managed by UIO driver, skipping
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1584 rte_i40e_pmd
EAL:   0000:05:00.0 not managed by VFIO driver, skipping
EAL:   0000:05:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:81:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1584 rte_i40e_pmd
EAL:   0000:81:00.0 not managed by VFIO driver, skipping
EAL:   PCI memory mapped at 0x7ffef13ff000
EAL:   PCI memory mapped at 0x7ffff7fea000
PMD: eth_i40e_dev_init(): FW 4.1 API 1.1 NVM 04.01.00 eetrack 80001121
PMD: eth_i40e_dev_init(): Failed to stop lldp
PMD: i40e_pf_parameter_init(): Max supported VSIs:130
PMD: i40e_pf_parameter_init(): PF queue pairs:64
PMD: i40e_pf_parameter_init(): Max VMDQ VSI num:63
PMD: i40e_pf_parameter_init(): VMDQ queue pairs:4
[New Thread 0x7ffef0fff700 (LWP 144203)]
ports number: 1
Port 0: max_rx_queues 316 max_tx_queues 316
Port 0: using 1 queue
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
Port 0 init ... done:
Creating mbuf pool 'dpdk_net_pktmbuf_pool0' [3072 mbufs] ...
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
PMD: i40e_dev_tx_queue_setup(): Using full-featured tx path

Checking link status
Created DPDK device
.done
Port 0 Link Up - speed 40000 Mbps - full-duplex
Seastar TCP server listening on port 10000 ...
^^^^^^^^^^^^^^^^ dpdk_qp p.len=74
dpdk_qp to l2
============== got eth: p.len=74
--------------ip rx p.len=60
to l4
tcp_received p.len()=40
tcp_received src_port=39235
^^^^^^^^^^^^^^^^ dpdk_qp p.len=56
dpdk_qp to l2
============== got eth: p.len=56
--------------ip rx p.len=42
1: ip_len=40
to l4
tcp_received p.len()=20
tcp_received src_port=39235
^^^^^^^^^^^^^^^^ dpdk_qp p.len=58
dpdk_qp to l2
============== got eth: p.len=58
--------------ip rx p.len=44
to l4
tcp_received p.len()=24
tcp_received src_port=39235
Got data for us
^^^^^^^^^^^^^^^^ dpdk_qp p.len=1230
dpdk_qp to l2
============== got eth: p.len=1230
--------------ip rx p.len=1216
to l4
tcp_received p.len()=1196
tcp_received src_port=39235
ooo port=39235, nr=0, seq=2925, seq_abs=2182892504
^^^^^^^^^^^^^^^^ dpdk_qp p.len=946
dpdk_qp to l2
============== got eth: p.len=946
--------------ip rx p.len=932
to l4
tcp_received p.len()=912
tcp_received src_port=39235
ooo port=39235, nr=1, seq=11401, seq_abs=2182900980
dpdk_pool 2561
^^^^^^^^^^^^^^^^ dpdk_qp p.len=56
dpdk_qp to l2
============== got eth: p.len=56
dpdk_pool 2561
dpdk_pool 2561
gleb-cloudius commented 9 years ago

The problem is that memcached send packets with 9 fragments each, but this HW is only capable of sending packets with 8 frags ot less if TSO is not enabled and it is not enabled for DPDK yet. I haven't found how many frags are supported with TSO, but definitely greater than 8.

nyh commented 9 years ago

On Thu, Feb 12, 2015 at 11:17 AM, Gleb Natapov notifications@github.com wrote:

The problem is that memcached send packets with 9 fragments each, but this HW is only capable of sending packets with 8 frags ot less if TSO

Interesting. While we can - and probably should - fix memcached to send fewer fragments, someone can come along and write a new application that again tries to send 9 fragments. So definitely Seastar itself needs to realize it was given too many fragments and merge some of them (by copying) until we have fewer than the limit. Yesterday there was also a discussion on how such merging for tiny fragments is also needed for performance.

Nadav Har'El nyh@cloudius-systems.com

gleb-cloudius commented 9 years ago

On Thu, Feb 12, 2015 at 01:33:37AM -0800, nyh wrote:

On Thu, Feb 12, 2015 at 11:17 AM, Gleb Natapov notifications@github.com wrote:

The problem is that memcached send packets with 9 fragments each, but this HW is only capable of sending packets with 8 frags ot less if TSO

Interesting. While we can - and probably should - fix memcached to send fewer fragments, someone can come along and write a new application that again tries to send 9 fragments. So definitely Seastar itself needs to realize it was given too many fragments and merge some of them (by copying) until we have fewer than the limit. Yesterday there was also a discussion on how such merging for tiny fragments is also needed for performance.

The fix is to use TSO if it was not clear from my previous message :) TSO is needed anyway for reasonable TCP performance.

        Gleb.
nyh commented 9 years ago

On Thu, Feb 12, 2015 at 11:36 AM, Gleb Natapov notifications@github.com wrote:

The fix is to use TSO if it was not clear from my previous message :) TSO is needed anyway for reasonable TCP performance.

It's not clear TSO would be a complete solution because:

  1. It's not clear to me that the 8-fragment limit of the NIC has a 8-fragment limit is only when TSO is not enabled. (sounds reasonable, but it needs checking...).
  2. If there's such a thing of a limit on the number of fragments, Seastar needs to be aware of it. If it's not 8 and it's 64, Seastar still needs to do something different if someone tries to send 65 fragments...

Nadav Har'El nyh@cloudius-systems.com

gleb-cloudius commented 9 years ago

On Thu, Feb 12, 2015 at 01:41:19AM -0800, nyh wrote:

On Thu, Feb 12, 2015 at 11:36 AM, Gleb Natapov notifications@github.com wrote:

The fix is to use TSO if it was not clear from my previous message :) TSO is needed anyway for reasonable TCP performance.

It's not clear TSO would be a complete solution because:

  1. It's not clear to me that the 8-fragment limit of the NIC has a 8-fragment limit is only when TSO is not enabled. (sounds reasonable, but it needs checking...). Spec says this specifically that's why I wrote what I wrote. This is not a guess work.
  2. If there's such a thing of a limit on the number of fragments, Seastar needs to be aware of it. If it's not 8 and it's 64, Seastar still needs to do something different if someone tries to send 65 fragments...

Of course, currently we linearize if there is more then 32 frags.

        Gleb.