search

darkk / redsocks

transparent TCP-to-proxy redirector
http://darkk.net.ru/redsocks
3.33k stars 862 forks source link

Performance / Improvement #130

Open mail3dexter opened 6 years ago

mail3dexter commented 6 years ago

I am currently evaluating SOCKS client which would work well with Dante SOCKS server. These are the clients I am interested into after exploring mane things like features, open source code, support, etc.

  1. Dante Client
  2. Redsocks

I am more leaned towards redsocks due to its simplicity working wise. However what I can see from my load tests is that I am getting 10x performance with redsocks as compared to 50x performance with Dante SOCKS client. I am comparing both with HAProxy load tests.

Is there a way to optimize redsocks so that it can go multiprocess because currently, with my tests, I am able to choke redsocks without using full resources.

darkk commented 6 years ago

What do you use as a baseline while saying 10x and 50x, what is 1x? What metrics do you measure? How do you measure it?

What sort of network and machine are you speaking about? What version of Linux kernel does it run?

mail3dexter commented 6 years ago

So my baseline metrics are gathered from HAProxy Load Tests I am measuring RPS, Bandwidth, RPS per CPU, etc using tools like ab, wrk, custom http client, etc which can do different IO size with concurrent connections, and with or without connection pooling.

I am using a 40Gbps NIC which is rarely used as this baremetal is used only for load testing.

These are my OS details: 

root@elb-none-479194:~# cat /etc/*rel*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@elb-none-479194:~# uname -a
Linux elb-none-479194 4.4.0-75-generic #96-Ubuntu SMP Thu Apr 20 09:56:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

This box is a 48 Core CPU and with 3-4 concurrent load testing tool, I am able to choke redsocks to 100% CPU

darkk commented 6 years ago

Do I understand you correctly that

?

mail3dexter commented 6 years ago

So consider I am using 87 clients which has 2 Core CPU 4 GB RAM. I have 70+ remote destinations which runs ngix and serves files of different sizes which would act as IO size.

There are 3 tests which has below mentioned data flow: Client -> HA Proxy -> Backend Server Client -> Redsocks -> Dante Server -> Backend Server Client (with Dante Client) -> Dante Server -> Backend Server

With 1 connection per client (87 clients) and IO size 128B (this is to mainly calculate RPS), I am getting ~136K RPS in HA proxy, ~56K RPS with redsocks and ~57K in Dante Client

With 16 connection per client (87 clients) and IO size 128B (this is to mainly calculate RPS), I am getting ~360K RPS in HA proxy, ~86K RPS with redsocks and ~89K in Dante Client

With 1 connection per client (87 clients) and IO size 16KB (this is to mainly calculate bandwidth), I am getting ~111K RPS in HA Proxy, ~11600MBps BW in HA Proxy ~9.5K RPS with redsocks, ~159MBps BW in redsocks, ~44K in Dante Client and ~734MBps BW in Dante Client

With 16 connection per client (87 clients) and IO size 16KB (this is to mainly calculate bandwidth), I am getting ~170K RPS in HA proxy,~17500MBps BW in HA Proxy ~9.5K RPS with redsocks ~1140MBps BW in redsocks, ~44K in Dante Client and ~1300MBps BW in Dante Client

Also, under these tests, redsocks sometimes fails on a few of the client boxes with this error

Jul  9 14:14:31 vm-1003 redsocks[1297]: redsocks started
Jul  9 14:14:31 vm-1003 redsocks[1297]: [1.2.3.4:16998->4.3.2.1:80]: accepted
Jul  9 14:14:32 vm-1003 redsocks[1297]: [1.2.3.4:16998->4.3.2.1:80]: data relaying started
Jul  9 14:14:33 vm-1003 redsocks[1297]: [1.2.3.4:16998->4.3.2.1:80]: shutdown(client, SHUT_WR): Transport endpoint is not connected
Jul  9 14:14:33 vm-1003 redsocks[1297]: [1.2.3.4:16998->4.3.2.1:80]: both client and server disconnected
Jul  9 14:14:33 vm-1003 redsocks[1297]: [1.2.3.4:16998->4.3.2.1:80]: dropping client

On the Dante Server, we can see that SOCKS connection was properly initiated from client, server went to endpoint and got all the data and then server properly returned data back to redsocks client and properly closed the connection.

Logs on Dante server:

Jul  9 14:14:31 (1531125871.946543) sockd[95992]: info: pass(1): tcp/accept [: 1.2.3.4.39237 5.5.5.5.1234
Jul  9 14:14:32 (1531125872.833943) sockd[96287]: info: pass(23): tcp/connect [: 1.2.3.4.39237 5.5.5.5.1234 -> 5.5.5.5.39237 4.3.2.1.80
Jul  9 14:14:33 (1531125873.142279) sockd[96287]: info: pass(23): tcp/connect -: 1.2.3.4.39237 5.5.5.5.1234 -> 5.5.5.5.39237 4.3.2.1.80 (41)
Jul  9 14:14:33 (1531125873.146601) sockd[96287]: info: pass(23): tcp/connect -: 1.2.3.4.39237 5.5.5.5.1234 -> 5.5.5.5.39237 4.3.2.1.80 (0)
Jul  9 14:14:33 (1531125873.146713) sockd[96287]: info: pass(23): tcp/connect -: 4.3.2.1.80 5.5.5.5.39237 -> 5.5.5.5.1234 1.2.3.4.39237 (358)
Jul  9 14:14:33 (1531125873.150708) sockd[96287]: info: pass(23): tcp/connect ]: 358 -> 1.2.3.4.39237 5.5.5.5.1234 -> 41, 41 -> 5.5.5.5.39237 4.3.2.1.80 -> 358: local client closed.  Session duration: 1s
Jul  9 14:14:33 (1531125873.150826) sockd[96287]: info: pass(1): tcp/accept ]: 358 -> 1.2.3.4.39237 5.5.5.5.1234 -> 41: local client closed.  Session duration: 1s

Is this error occurring when application data is redirected to redsocks OR when redsocks is requesting data from dante socks server ?

mail3dexter commented 6 years ago

Any update?

I have recently started seeing this in redsocks logs: Aug 10 18:15:09 debian redsocks[14918]: redsocks started, conn_max=125000 Aug 10 18:15:09 debian redsocks[14918]: [192.168.12.10:10870->192.168.123.1:80]: shutdown: bufferevent_disable(client, EV_READ) Aug 10 18:15:11 debian redsocks[14918]: [192.168.12.10:10950->192.168.123.1:80]: backpressure: event_del(relay_read) Aug 10 18:15:11 debian redsocks[14918]: [192.168.12.10:10950->192.168.123.1:80]: backpressure: event_add(relay_read) Aug 10 18:15:12 debian redsocks[14918]: [192.168.12.10:10986->192.168.123.1:80]: backpressure: event_del(relay_read) Aug 10 18:15:12 debian redsocks[14918]: [192.168.12.10:10986->192.168.123.1:80]: backpressure: event_add(relay_read)

Is this related to the problem for which logs were provided earlier?

mail3dexter commented 6 years ago

Bump