search

WALL-E / tcp-ip-labs

tcp ip labs
2 stars 1 forks source link

SO_REUSEADDR和SO_RESUSEPORT #8

Open WALL-E opened 7 years ago

WALL-E commented 7 years ago

SO_REUSEPORT

This socket option allows multiple sockets to listen on the same IP address and port combination. The kernel then load balances incoming connections across the sockets.

SO_REUSEADDR

KERNEL_SOURCE/include/net/inet_hashtables.h

/* There are a few simple rules, which allow for local port reuse by
 * an application.  In essence:
 *
 *  1) Sockets bound to different interfaces may share a local port.
 *     Failing that, goto test 2.
 *  2) If all sockets have sk->sk_reuse set, and none of them are in
 *     TCP_LISTEN state, the port may be shared.
 *     Failing that, goto test 3.
 *  3) If all sockets are bound to a specific inet_sk(sk)->rcv_saddr local
 *     address, and none of them are the same, the port may be
 *     shared.
 *     Failing this, the port cannot be shared.
 *
 * The interesting point, is test #2.  This is what an FTP server does
 * all day.  To optimize this case we use a specific flag bit defined
 * below.  As we add sockets to a bind bucket list, we perform a
 * check of: (newsk->sk_reuse && (newsk->sk_state != TCP_LISTEN))
 * As long as all sockets added to a bind bucket pass this test,
 * the flag bit will be set.
 * The resulting situation is that tcp_v[46]_verify_bind() can just check
 * for this flag bit, if it is set and the socket trying to bind has
 * sk->sk_reuse set, we don't even have to walk the owners list at all,
 * we return that it is ok to bind this socket to the requested local port.
 *
 * Sounds like a lot of work, but it is worth it.  In a more naive
 * implementation (ie. current FreeBSD etc.) the entire list of ports
 * must be walked for each data port opened by an ftp server.  Needless
 * to say, this does not scale at all.  With a couple thousand FTP
 * users logged onto your box, isn't it nice to know that new data
 * ports are created in O(1) time?  I thought so. ;-)   -DaveM
 */
  1. tcp_tw_reuse 和SO_REUSEADDR 是两个完全不同的东西
  2. SO_REUSEPORT linux 3.7才支持，用于绑定相同ip:port，像nginx 那样 fork方式也能实现
WALL-E commented 7 years ago

SO_REUSEPORT 实验

服务端代码

#!/usr/bin/python

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
s.bind(("0.0.0.0", 1919))
s.listen(1)

while 1:
    clientsock, clientaddr = s.accept()
    print "Got connection from", clientsock.getpeername()
    clientsock.close()

连接4个终端，可以登录同一个IP地址。

  • 【终端1】 172.28.32.101
  • 【终端2】 172.28.32.101
  • 【终端3】 172.28.32.101
  • 【终端4】 172.28.32.101
  1. 【终端1】，【终端2】，【终端3】 
    ./serv.py
  2. 【终端4】 
    for i in {1..30}; do sock 172.28.32.101 1919; done

结论：

  1. 三个serv.py进程都可以启动
  2. 查看socket状态，3个进程监听了同一个端口 
    [root@vagrant-172-28-32-101 ~]# ss -ltnp|more
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port
LISTEN     0      128          *:22                       *:*                   users:(("sshd",pid=1009,fd=3))
LISTEN     0      100    127.0.0.1:25                       *:*                   users:(("master",pid=1787,fd=13))
LISTEN     0      1            *:1919                     *:*                   users:(("serv.py",pid=2534,fd=3))
LISTEN     0      1            *:1919                     *:*                   users:(("serv.py",pid=2532,fd=3))
LISTEN     0      1            *:1919                     *:*                   users:(("serv.py",pid=2530,fd=3))
LISTEN     0      128         :::22                      :::*                   users:(("sshd",pid=1009,fd=4))
LISTEN     0      100        ::1:25                      :::*                   users:(("master",pid=1787,fd=14))
  3. 三个服务器都可以接受请求，并且可以负载均衡。
WALL-E commented 7 years ago

nginx的fork模式是怎么实用这个选项的？

Ref

WALL-E commented 7 years ago

Kernel Git commit

2013-01-23 13:44:10 -0500 Merge branch 'soreuseport'

The TCP implementation has a problem in that the request sockets for a listener are attached to a listener socket. If a SYN is received, a listener socket is chosen and request structure is created (SYN-RECV state). If the subsequent ack in 3WHS does not match the same port by so_reusport, the connection state is not found (reset) and the request structure is orphaned. This scenario would occur when the number of listener sockets bound to a port changes (new ones are added, or old ones closed). We are looking for a solution to this, maybe allow multiple sockets to share the same request table...

目前，目前不知道是否已经修复，只能下载内核代码看看啦

git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

Solve #11

WALL-E commented 7 years ago

Resthub生产环境可以使用reuseport指令，出现风险为小概率事件。