UDP 报文增多时出现 `socket: too many open files`

[sbilly]

环境描述

客户端配置 /etc/v2ray/config.json 使用 VMESS+WS+TLS+CDN

{
  "inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 1080,
      "protocol": "socks",
      "sniffing": {
        "enabled": false,
        "destOverride": ["http", "tls"]
    },
      "settings": {
        "auth": "noauth",
        "udp": true
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "vmess",
      "settings": {
        "vnext": [
          {
            "address": "${EDGE_HOSTNAME}",
            "port": 443,
            "users": [
              {
                "id": "${UUID}",
                "alterId": 32
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "wsSettings": {
          "connectionReuse": true,
          "path": "/ws/",
          "headers": null
        }
      },
      "mux": {"enabled": true}
    },
    {
      "protocol": "freedom",
      "settings": {},
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "adblock"
    }
  ],
  "routing": {
    "domainStrategy": "IPOnDemand",
    "rules": [
      {
        "type": "field",
        "domain": [
          "googeadsserving.cn"
        ],
        "outboundTag": "adblock"
      },
      {
        "type": "field",
        "ip": [
          "geoip:private",
          "geoip:cn"
        ],
        "outboundTag": "direct"
      }
    ]
  }
}

客户端启动脚本和相关描述

# 内核开启转发功能
sysctl -w net.ipv4.ip_forward=1

# 确保 tun2socks 使用的代理服务器直接访问
ip route add x.x.x.x/32 via 192.168.158.2

# 启动 tun2socks 并使用 v2ray 代理并指定 v2ray 配置
./build/tun2socks -tunName tun1 -tunAddr 240.0.0.2 -tunGw 240.0.0.1 -tunMask 255.255.255.0 -proxyType v2ray -vconfig "/etc/v2ray/config.json" -applog

# 给 tun1 配置 IP 地址并且使之上线
ip addr add 240.0.0.1 dev tun1
ip link set dev tun1 up

# 调整缺省路由表
ip route add 0.0.0.0/1 via 240.0.0.1
ip route add 128.0.0.0/1 via 240.0.0.1

配置后通过 curl ip.sb 能看到已经是通过代理服务器访问。

现象描述

当流量增长的时候（特别是 UDP 流量） go-tun2socks 报错

2019/02/16 10:19:13 [unknown process] is connecting udp:x.x.x.x:xxxx
2019/02/16 10:19:13 [unknown process] is connecting udp:x.x.x.x:xxxx
2019/02/16 10:19:13 [unknown process] is connecting udp:x.x.x.x:xxxx
2019/02/16 10:19:13 [Warning] failed to handler mux client connection > v2ray.com/core/proxy/vmess/outbound: failed to find an available destination > v2ray.com/core/common/retry: [v2ray.com/core/transport/internet/websocket: failed to dial WebSocket > v2ray.com/core/transport/internet/websocket: failed to dial to (wss://xxx.xxx/ws/):  > dial tcp x.x.x.x:443: socket: too many open files] > v2ray.com/core/common/retry: all retry attempts failed
2019/02/16 10:19:13 [unknown process] is connecting udp:x.x.x.x:xxxx

已经开启了 mux 和 connectionReuse 都没有什么缓解。

[eycorsican]

目测是出现死循环了，首先你的系统 DNS 看起来没有加路由到原来网关，然后你的代理服务器中的地址似乎用了域名，导致：

1. 某个软件要解析域名，发 DNS 请求
2. DNS 请求发到 TUN，被代理
3. 发起代理连接，解析代理服务器域名
4. 解析代理服务器域名的 DNS 请求发到 TUN，然后被代理
5. 接上面 3，死循环

[eycorsican]

另外 tun2socks 不需要这个的：

# 内核开启转发功能
sysctl -w net.ipv4.ip_forward=1

[eycorsican]

最新的 master 在 Linux 上支持了 Dynamic Routing：加上 -gateway 参数，可能对你的使用情况有帮助，不过暂时得自行编译。

[sbilly]

已经更新到了 commit 7c1edc107ca2c5afb16fd89d512ac008c8658087

./build/tun2socks -tunName tun1 -tunAddr 240.0.0.2 -tunGw 240.0.0.1 -tunMask 255.255.255.0 -proxyType v2ray -vconfig "/etc/v2ray/config.json" -gateway
192.168.158.2
2019/02/16 13:05:29 openning tun device
2019/02/16 13:05:29 ICMP packets will be delayed for 10ms
2019/02/16 13:05:29 Dynamic routing is enabled
2019/02/16 13:05:29 Running tun2socks
2019/02/16 13:05:29 [Warning] v2ray.com/core: V2Ray 4.16 started

应该不是 DNS 导致的，我自己有个应用的确是 UDP 的，分析如下：

# DNS
cat /etc/resolv.conf | grep nameserver
nameserver 127.0.0.53

# 给 tun1 配置 IP 地址并且使之上线
ip addr add 240.0.0.1 dev tun1
ip link set dev tun1 up

# 调整缺省路由表
ip route add 0.0.0.0/1 via 240.0.0.1
ip route add 128.0.0.0/1 via 240.0.0.1

# 确保 tun2socks 使用的代理服务器直接访问
ip route add x.x.x.x/32 via 192.168.158.2

# 查看当前路由表
ip route
0.0.0.0/1 via 240.0.0.1 dev tun1
default via 192.168.158.2 dev ens33 proto dhcp src 192.168.158.221 metric 100
x.x.x.x via 192.168.158.2 dev ens33
128.0.0.0/1 via 240.0.0.1 dev tun1
192.168.158.0/24 dev ens33 proto kernel scope link src 192.168.158.221
192.168.158.2 dev ens33 proto dhcp scope link src 192.168.158.221 metric 100

在命令行中用 dig www.sina.com.cn 进行 DNS 查询时使用 tcpdump -i tun1 port 53 -n 也看不到任何 DNS 报文，curl ip.sb 已经看到是通过代理访问。

[sbilly]

通过 -gateway 192.168.158.2 指定启动 Dynamic Routing 以后，确实看不到 socket: too many open files 的错误了，但看到很多内网地址路由添加的错误 adding route for udp:172.18.0.1:xxxx failed: exit status 2，同时导致我远程连接的 ssh 也断了。

2019/02/16 13:22:45 new proxy connection for target: udp:192.168.31.150:56044
2019/02/16 13:22:45 adding route for udp:192.168.31.150:56044 failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:192.168.31.150:56044
2019/02/16 13:22:45 adding route for udp:172.17.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.17.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.17.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.17.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.18.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.18.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.18.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.18.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.19.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.19.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.19.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.19.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.20.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.20.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.20.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.20.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.21.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.21.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.21.0.1:xxxx failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.21.0.1:xxxx
2019/02/16 13:22:45 adding route for udp:172.17.0.1:56043 failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.17.0.1:xxxxx
2019/02/16 13:22:45 adding route for udp:172.17.0.1:56043 failed: exit status 2
2019/02/16 13:22:45 new proxy connection for target: udp:172.17.0.1:xxxxx
2019/02/16 13:22:45 adding route for udp:172.17.0.1:56043 failed: exit status 2 

同时在 ip route 的时候看到增加了很多主机路由，这又是为何呢？

123.126.157.222 via 192.168.158.2 dev ens33 
140.205.94.189 via 192.168.158.2 dev ens33 
140.205.220.96 via 192.168.158.2 dev ens33 
153.36.233.68 via 192.168.158.2 dev ens33 

[eycorsican]

动态路由就是动态添加路由到路由表里的一个功能，至于这些错误，我不是太清楚，你手动执行条命令试试？ ip route add 172.17.0.1/32 via gateway

[sbilly]

手工执行也报错

ip route add 172.17.0.1/32 via 192.168.52.32                                                                                                   
Error: Nexthop has invalid gateway.

[sbilly]

我原本就是期望让所有的流量走我指定的代理，所以我之前只加了到代理服务器的主机路由。这样加了主机路由以后就不符合我的预期了。

[eycorsican]

那你应该把 routing 里的 direct 规则删掉，用asis 策略 ，把freedom outbound 删掉，代理地址也用IP不要用域名，全局代理，保证不会出错

[sbilly]

先在 /etc/hosts 中增加了 hosts 避免查询 DNS，然后使用下面配置使用 ./build/tun2socks -tunName tun1 -tunAddr 240.0.0.2 -tunGw 240.0.0.1 -tunMask 255.255.255.0 -proxyType v2ray -vconfig "/etc/v2ray/config.json" 启动 go-tun2socks 后没出现错误。

{
  "inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 1080,
      "protocol": "socks",
      "sniffing": {
        "enabled": false,
        "destOverride": ["http", "tls"]
    },
      "settings": {
        "auth": "noauth",
        "udp": true
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "vmess",
      "settings": {
        "vnext": [
          {
            "address": "${EDGE_HOSTNAME}",
            "port": 443,
            "users": [
              {
                "id": "${UUID}",
                "alterId": 32
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "wsSettings": {
          "connectionReuse": true,
          "path": "/ws/",
          "headers": null
        }
      },
      "mux": {"enabled": true}
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "adblock"
    }
  ]
}

因为使用 CDN 必须用带 hostname 访问，因此只能在配置中保留 hostname 。有没有什么办法不修改 hosts 文件呢？

[eycorsican]

https://v2ray.com/chapter_02/transport/websocket.html

用IP做地址，域名写到 WS 的 Host 头里

[sbilly]

多谢～～

仅仅指定 WS 的 Host 头不工作。因为配置中的 address 使用了 IP 地址，因此为了继续使用 CDN 和 SSL 必须在 tlsSettings和 wsSettings 的 Host 中指定域名。下面的配置彻底 OK 了：

{
  "inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 1080,
      "protocol": "socks",
      "sniffing": {
        "enabled": false,
        "destOverride": ["http", "tls"]
    },
      "settings": {
        "auth": "noauth",
        "udp": true
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "vmess",
      "settings": {
        "vnext": [
          {
            "address": "${V2RAY_EDGE_IP}",
            "port": 443,
            "users": [
              {
                "id": "${V2RAY_USER_UUID}",
                "alterId": 32
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "wsSettings": {
          "connectionReuse": true,
          "path": "/ws/",
          "headers": {
            "Host": "${V2RAY_EDGE_HOSTNAME}"
          }
        },
        "tlsSettings": {
          "serverName": "${V2RAY_EDGE_HOSTNAME}"
        }
      },
      "mux": {"enabled": true}
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "adblock"
    }
  ]
}

[itshaadi]

@sbilly

仅仅指定 WS 的 Host 头不工作。因为配置中的 address 使用了 IP 地址，因此为了继续使用 CDN 和 SSL 必须在 tlsSettings和 wsSettings 的 Host 中指定域名。下面的配置彻底 OK 了：

so based on google translate what I figured out, is that if I specify my domain name in tlsSettings and wsSettings even though my server's public IP is in the address section. it would still be safe. since everything is behinde a CDN. is that so? do I still have to use -gateway option?

[eycorsican]

Basically, using an IP in the address field is to prevent infinite loop.

But for the WSS+CDN setup to work, we need to tell the remote server the domain name we are using, remote server would need this information when dealing with SNI or to find the right virtual server. We specify this in wssettings and tlssettings.

Having such a setup, if you just want everything to be proxied by the server, you don't need -gateway anymore.

[sbilly]

@sbilly

仅仅指定 WS 的 Host 头不工作。因为配置中的 address 使用了 IP 地址，因此为了继续使用 CDN 和 SSL 必须在 tlsSettings和 wsSettings 的 Host 中指定域名。下面的配置彻底 OK 了：

so based on google translate what I figured out, is that if I specify my domain name in tlsSettings and wsSettings even though my server's public IP is in the address section. it would still be safe. since everything is behinde a CDN. is that so? do I still have to use -gateway option?

As @eycorsican said. You will not need -gateway in this case. And it is as secure as using hostname in address field.

[sbilly]

@eycorsican 请问 -gateway 选项自动增加的主机路由是的 IP 地址是因为命中了 routing 策略而触发增加的吗？例如，我之前的配置中包含了 "geoip:private" 和 "geoip:cn" 这两个策略，所导致命中了这两个策略的私网地址和 geoip 判断为 cn 的 ip 地址都会自动增加主机路由。

之前的 routing 配置

  "routing": {
    "domainStrategy": "IPOnDemand",
    "rules": [
      {
        "type": "field",
        "domain": [
          "googeadsserving.cn"
        ],
        "outboundTag": "adblock"
      },
      {
        "type": "field",
        "ip": [
          "geoip:private",
          "geoip:cn"
        ],
        "outboundTag": "direct"
      }
    ]
  }

[eycorsican]

是的，针对匹配了 direct tag 的 IP

[sbilly]

@eycorsican 👍

这个可以结合 iptables / ip route 之类的实现很灵活的策略

[itshaadi]

Basically, using an IP in the address field is to prevent infinite loop.

so ideally the IP is still hidden? if someone inspected the packets they wouldn't find the IP right? because it would just show the domain (since it's behind a CDN)... if so, then my IP just got walled using this method. I don't see why?