search

iqiyi / dpvs

DPVS is a high performance Layer-4 load balancer based on DPDK.
Other
3k stars 723 forks source link

How to configure two-arm DR #153

Open tiepnv-viosoft opened 6 years ago

tiepnv-viosoft commented 6 years ago

Hello,

I'm planning to compare performance between the two-arm DR mode and the simple two-arm FNAT mode. I have set up the simple two-arm FNAT mode successfully but I couldn't finish the two-arm DR mode. Here is my DPVS configuration:

# on DPVS

# add LAN IP for DPVS, it must be different from VIP
./dpip addr add 192.168.100.1/24 dev dpdk0
# add VIP and the route will generate automatically.
./dpip addr add 10.0.0.100/32 dev dpdk1

# route for LAN network, just a hint.
#./dpip route add 192.168.100.0/24 dev dpdk0
./dpip route add 10.0.0.0/16 dev dpdk1

# add service <VIP:vport> to forwarding, scheduling mode is RR.
./ipvsadm -A -t 10.0.0.100:80 -s rr

# add two RS for service, forwarding mode is DR
./ipvsadm -a -t 10.0.0.100:80 -r 192.168.100.2 -g
./ipvsadm -a -t 10.0.0.100:80 -r 192.168.100.3 -g

For each RS, I ran the following commands:

ip addr add 10.0.0.100/32 dev lo
sysctl -w net.ipv4.conf.lo.arp_ignore=1

Could you help me set up the two-arm DR mode?

Thanks, TiepNV

beacer commented 6 years ago

More details needed, is the packet forwarded to RS ? can you show ipvsadm -ln , dpip addr show, dpip route show. and try capture the packets on RS ?

tiepnv-viosoft commented 6 years ago

Hi beacer,

Thanks for your reply. The packet was forwarded to the RS. Here is a snippet of the tcpdump's output on the RS:

23:29:20.435972 ARP, Request who-has 192.168.100.2 tell 192.168.100.1, length 46
    0x0000:  ffff ffff ffff 3cfd fea0 ce20 0806 0001  ......<.........
    0x0010:  0800 0604 0001 3cfd fea0 ce20 c0a8 6401  ......<.......d.
    0x0020:  0000 0000 0000 c0a8 6402 0000 0000 0000  ........d.......
    0x0030:  0000 0000 0000 0000 0000 0000            ............
23:29:20.436006 ARP, Reply 192.168.100.2 is-at c6:c8:d4:1d:0d:9e (oui Unknown), length 28
    0x0000:  3cfd fea0 ce20 c6c8 d41d 0d9e 0806 0001  <...............
    0x0010:  0800 0604 0002 c6c8 d41d 0d9e c0a8 6402  ..............d.
    0x0020:  3cfd fea0 ce20 c0a8 6401                 <.......d.
23:29:20.436061 IP 10.0.0.48.48956 > 10.0.0.100.http: Flags [S], seq 3129311084, win 29200, options [mss 1460,sackOK,TS val 472098791 ecr 0,nop,wscale 7], length 0
    0x0000:  c6c8 d41d 0d9e 3cfd fea0 ce20 0800 4500  ......<.......E.
    0x0010:  003c d88b 4000 4006 4d9d 0a00 0030 0a00  .<..@.@.M....0..
    0x0020:  0064 bf3c 0050 ba85 7f6c 0000 0000 a002  .d.<.P...l......
    0x0030:  7210 03d3 0000 0204 05b4 0402 080a 1c23  r..............#
    0x0040:  a7e7 0000 0000 0103 0307                 ..........
23:29:20.436148 IP 10.0.0.100.http > 10.0.0.48.48956: Flags [S.], seq 1344142166, ack 3129311085, win 28960, options [mss 1460,sackOK,TS val 60477253 ecr 472098791,nop,wscale 7], length 0
    0x0000:  3cfd fea0 ce20 c6c8 d41d 0d9e 0800 4500  <.............E.
    0x0010:  003c 0000 4000 4006 2629 0a00 0064 0a00  .<..@.@.&)...d..
    0x0020:  0030 0050 bf3c 501d fb56 ba85 7f6d a012  .0.P.<P..V...m..
    0x0030:  7120 14c2 0000 0204 05b4 0402 080a 039a  q...............
    0x0040:  cf45 1c23 a7e7 0103 0307                 .E.#......
23:29:21.432646 IP 10.0.0.48.48956 > 10.0.0.100.http: Flags [S], seq 3129311084, win 29200, options [mss 1460,sackOK,TS val 472099041 ecr 0,nop,wscale 7], length 0
    0x0000:  c6c8 d41d 0d9e 3cfd fea0 ce20 0800 4500  ......<.......E.
    0x0010:  003c d88c 4000 4006 4d9c 0a00 0030 0a00  .<..@.@.M....0..
    0x0020:  0064 bf3c 0050 ba85 7f6c 0000 0000 a002  .d.<.P...l......
    0x0030:  7210 02d9 0000 0204 05b4 0402 080a 1c23  r..............#
    0x0040:  a8e1 0000 0000 0103 0307                 ..........
23:29:21.432674 IP 10.0.0.100.http > 10.0.0.48.48956: Flags [S.], seq 1344142166, ack 3129311085, win 28960, options [mss 1460,sackOK,TS val 60477502 ecr 472098791,nop,wscale 7], length 0
    0x0000:  3cfd fea0 ce20 c6c8 d41d 0d9e 0800 4500  <.............E.
    0x0010:  003c 0000 4000 4006 2629 0a00 0064 0a00  .<..@.@.&)...d..
    0x0020:  0030 0050 bf3c 501d fb56 ba85 7f6d a012  .0.P.<P..V...m..
    0x0030:  7120 14c2 0000 0204 05b4 0402 080a 039a  q...............
    0x0040:  d03e 1c23 a7e7 0103 0307                 .>.#......
23:29:22.428985 IP 10.0.0.100.http > 10.0.0.48.48956: Flags [S.], seq 1344142166, ack 3129311085, win 28960, options [mss 1460,sackOK,TS val 60477752 ecr 472098791,nop,wscale 7], length 0
    0x0000:  3cfd fea0 ce20 c6c8 d41d 0d9e 0800 4500  <.............E.
    0x0010:  003c 0000 4000 4006 2629 0a00 0064 0a00  .<..@.@.&)...d..
    0x0020:  0030 0050 bf3c 501d fb56 ba85 7f6d a012  .0.P.<P..V...m..
    0x0030:  7120 14c2 0000 0204 05b4 0402 080a 039a  q...............
    0x0040:  d138 1c23 a7e7 0103 0307                 .8.#......
23:29:23.436678 IP 10.0.0.48.48956 > 10.0.0.100.http: Flags [S], seq 3129311084, win 29200, options [mss 1460,sackOK,TS val 472099542 ecr 0,nop,wscale 7], length 0
    0x0000:  c6c8 d41d 0d9e 3cfd fea0 ce20 0800 4500  ......<.......E.
    0x0010:  003c d88d 4000 4006 4d9b 0a00 0030 0a00  .<..@.@.M....0..
    0x0020:  0064 bf3c 0050 ba85 7f6c 0000 0000 a002  .d.<.P...l......
    0x0030:  7210 00e4 0000 0204 05b4 0402 080a 1c23  r..............#
    0x0040:  aad6 0000 0000 0103 0307

And this is some information of the DPVS:

./ipvsadm -ln --stats
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  10.0.0.100:80                       7       26        0     1560        0
  -> 192.168.100.2:80                    3       18        0     1080        0
  -> 192.168.100.3:80                    4        8        0      480        0
./dpip addr show
inet 10.0.0.100/32 scope global dpdk1
     valid_lft forever preferred_lft forever
inet 192.168.100.1/24 scope global dpdk0
     valid_lft forever preferred_lft forever
./dpip route show
inet 192.168.100.1/32 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto 
inet 10.0.0.100/32 via 0.0.0.0 src 0.0.0.0 dev dpdk1 mtu 1500 tos 0 scope host metric 0 proto auto 
inet 192.168.100.0/24 via 0.0.0.0 src 192.168.100.1 dev dpdk0 mtu 1500 tos 0 scope link metric 0 proto auto 
inet 10.0.0.0/16 via 0.0.0.0 src 0.0.0.0 dev dpdk1 mtu 1500 tos 0 scope link metric 0 proto auto

Is there any wrong in my configuration?

beacer commented 6 years ago

After inbound packets forwarded to RS, outbound packets should get back to Client from RS directly. Outbound packets do not go through DPVS, RS should have route to Client or Gateway. Thus no OutPkts in statistics is correct for DR mode. http://www.linuxvirtualserver.org/VS-DRouting.html

tiepnv-viosoft commented 6 years ago

Thank you very much, beacer!

cxd0513 commented 6 years ago

hi， @tiepnv-viosoft ，can i ask you some questions?i use ubuntu 16.04,too. how can i increase my node numbers?

tiepnv-viosoft commented 6 years ago

Sorry @cxd0513 , I don't understand your question. Could you explain more detail?

cxd0513 commented 6 years ago

sorry about my last question . Now i can build dpdk successfully .when i build dpvs in ubuntu16.04,it show this question: image can you see it before？thank you.

tiepnv-viosoft commented 6 years ago

I built successfully with the following modification:

sed -i 's#$(CC) $(CFLAGS) -o $@ $^ $(LIBS) $(DPVSDEPS)#$(CC) $(CFLAGS) -o $@ $^ $(DPVSDEPS) $(LIBS)#g' tools/ipvsadm/Makefile
sed -i 's#CFLAGS += $(LIBS) $(DEFS)#CFLAGS += $(DEFS)#g' tools/dpip/Makefile
sed -i 's#gcc $(CFLAGS) -o $@ $^#gcc $(CFLAGS) -o $@ $^ $(LIBS)#g' tools/dpip/Makefile

(Run above commands in your dpvs directory)

beacer commented 6 years ago

Tks @tiepnv-viosoft. It seems just need swap the sequence of target and source when build. And pls modify Makefile.in instead of Makefile, the later is auto generated. So that the changes can be committed to upstream. @cxd0513 if success, pls help to raise an Pull Request to submit the changes.

cxd0513 commented 6 years ago

thank you very much. @tiepnv-viosoft