The recommended no. of concurrent workers

[ealashwali]

Hi, I would like to run tls-scan with --concurrency option set to perform the scan as fast as possible. The readme says:

Concurrency should be set based on your system capacity (memory, cpu, network) etc. Default: 1.

If I'm using Ubuntu 18 with 16 GB RAM and Intel Core i7 2.6 GHz processor, can you suggest the suitable value? Can you provide your experience values as guidance for selecting the most suitable value of this parameter?

[prbinu]

Here are few things that need to be considered:

1. Check the port range: % sysctl net.ipv4.ip_local_port_range Increase the range: sysctl -w net.ipv4.ip_local_port_range="1500 64000"

2. Increase file limits (/etc/security/limits.conf)

3. Network bandwidth capacity can become your bottle neck: sudo ethtool eth0 | grep Speed Speed: 1000Mb/s <== This may not be adequate, depending up on your CPU capacity)

You may find other network settings here: https://medium.com/@_wmconsulting/tuning-linux-to-reach-maximum-performance-on-10-gbps-network-card-with-http-streaming-8599c9b4389d

For memory and CPU, track its usage (using top or htop) and increase the concurrency until it reaches the limit. BTW, tls-scan is single-threaded by design, hence to get max performance run multiple instances of tls-scan (in parallel) up to a max of N-1 cores (N = no. of CPU cores). Though I cannot predict which resource hit the limit first, you may start with watching the following resources:

1. Network bandwidth (try nethogs)
2. Port range (ss- s )
3. File descriptor (lsof -a -p $pid )
4. CPU
5. Memory

Hope this helps!

[ealashwali]

Thanks. But how settings concurrency argument benefits the performance if I'm using tls-scan to perform handshakes with domain names (no ciphersuites enumeration here). Since tls-scan is single thread, isn't going to make one TLS connection at a time and wait till it finishes, then do the second?

[prbinu]

No, that is not true. tls-scan is single threaded, but the connections are non-blocking. Hence it can handle large number of concurrent connections.

[ealashwali]

For your previous post,

Speed: 1000Mb/s <== This may not be adequate, depending up on your CPU capacity)

Not adequate even for 10 concurrent workers?

[prbinu]

There are couple of ways to answer this, but in short the answer is yes, it is adequate.

[ealashwali]

I will appreciate your advise in this issue. My network hardware capacity is: 867 Mbps. I have 4 i7 cores and 16 GB RAM. I run nload to monitor the network. I am sorry but could not find a good reference to help me interpret the output. I see the maximum 2.03 Mbit and this value increases when I increase the concurrent workers. Can you please advise me on this? Do you think I can increase till I reach 100s of Mbps? Can you see the screenshot here? https://github.com/ealashwali/public-photos/blob/master/5000_concurrent_edited.png

And this is nethogs output with 5000 workers. https://github.com/ealashwali/public-photos/blob/master/nethogs_5000_concurrent.png I'm just not sure if it is safe to increase? to what extent? What is the total value I should not exceed in given my network card capacity?

[prbinu]

Based on the second screenshot, it seems like you are only consuming < 600KB/sec is approx 5Mb/sec. So NIC is not saturated with 5000 concurrent connections. Since it is a single process, your one CPU would have hit the max. Also make sure you use ethernet cable, not wireless router.

[ealashwali]

Thanks. My CPU usage is less than 40% (30s range). I use wireless at the moment (testing) as I do not have wired card in my device. If I will use wired, it will be USB-to-Eth. I hope Linux nethogs can detect tit? My device manufacturer says the card's capacity is 867 Mbps. It seems only 6 Mbps? https://github.com/ealashwali/public-photos/blob/master/Screenshot%20from%202018-08-22%2000-15-44.png Or is it Ubuntu/Linux system shows the used capacity not the full one? P.S I'm connecting to Wireless.

I have two more questions please: 1) when my device hardware capacity is not coping with the no. of concurrent workers, do I lose connections (dropped) or the requests wait? and it is just that I do not get the expected performance? 2) The link photo is for 10 concurrent workers. Does this seem the optimal no. for my current settings? https://github.com/ealashwali/public-photos/blob/master/nethogs-10-concurrent.png

[prbinu]

Like I said before, there are many reasons that can impact your performance. Unless you are doing research on performance, I would suggest to try few combination and stick with a reasonable setting. Worst case, it takes couple more hours to complete the scan (if the scan is in the order of 100k servers).

Connections are not dropped, but if it takes more time, then it may get timeout errors.