Closed MMquant closed 5 years ago
I found out that OS X allows just 256 TCP connections at once. I increased it by editing open file descriptors soft limit
$ ulimit -S -n
256
$ ulimit -S -n 50000
$ ulimit -S -n
50000
I additionally added following code to dynamically set open file descriptors soft limit from the program:
import resource
new_soft_limit = 50000
(rlimit_nofile_soft, rlimit_nofile_hard) = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (new_soft_limit, rlimit_nofile_hard))
Following code snippet fetches multiple public DNS servers asynchronously. If script is executed in PyCharm it works perfectly and resolves all given resolvers with very few errors (~14 errors in 1078 requests).
However if I run exactly same script in OS X terminal only first ~280 aiodns requests are successful and the rest returns aiodns.DNSError(11, 'Could not contact DNS servers')(~834 errors in 1078 requests).
Copy/paste resolvers_short list from https://pastebin.com/wSYtzebZ
This code is part of my open-source project on https://github.com/MMquant/DNSweeper/blob/master/DNSweeper.py