error_connect_nxdomain around 60k clients

[jcperez85]

Doing load tests to an XMPP server through TCP, when the number of clients connected reaches 60000, logs start showing this error error_connect_nxdomain. This is the test I'm using (XMPP server has been replaced by asterisks):

test-100000.txt

[tisba]

Hey. Looks like you are exhausting ephemeral ports. There is a design limit how many connections you can have per Source IP/Destination IP Port combination. You could distribute your test to multiple tsung nodes or add more source IPs (see tsung documentation).

[weibomobile]

Hey.

The Socket's IP and Port

One socket connection's address have the quadruple :

{SrcIP, SrcPort, DstIP, DstPort}

One socket original address combile with {SrcIP, SrcPort}:

• M avaliable SrcIPs
• N avaliable SrtPorts
• You can craete M * N connections

The available ports range

You should know some details about the ports range in Linux/Unix.

The Port (short int variable) value range is 1 ~ 65535, and the 1~1024 just for the root apps.

You should check the available ports range with sysctl command :

sysctl -a | grep "ip_local_port_range"

You can set ports range:

sysctl -w net.ipv4.ip_local_port_range="1024 65535"

sysctl -p

Now you have 65535-1024 ≈ 64000 availabe ports.

The available Ip & Ports

• You have N available IPs
• Now, the total available ports : N * 64000

Some details you should know

• The one TCP's tcp_snd_buffer & tcp_rcv_buffer is 32KB by default, maybe think one tcp consume 100KB memory
• You want to simulator 10W users, you should have the 100000*100/1000/1000 = 10G memory at least
• Maybe should set the tcp_snd_buffer & tcp_rcv_buffer for 16KB
• Maybe you should use the hibernate option to reduce the memory usage for idle connections

[tisba]

Nice writeup. One should be careful though, because on a usual system there are services running that use ports > 1024. E.g. one of them is the epmd launched by Erlang. This is usually not a problem, but will generate some errors.

[weibomobile]

Maybe you should have a look at Tsung 1.6.1 with IP_TRANSPARENT support when you just only running test in lan-nextwork.

Some one who master the Chinese Language, should visit the post:

http://www.blogjava.net/yongboy/archive/2016/08/16/431601.html