getting "too many open files" few times

ehsandeep commented 5 years ago

Hi @joohoi,

Thank you for working on this, just noticed this error for few times now with random targets.

ffuf -t 50 -fs 0 -k -mc 200 -w word.txt -u https://test.site.com/FUZZ

Error in runner: Get https://test.site.com//config/locales/ja.yml: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//config/locales/simple_form.en.yml: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//config/newrelic.yml: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php.save: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configs/: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.jsp: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php.bak: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php.dist: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php.old: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//connect.php: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php.txt: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration.php~: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configuration/: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//configure/: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//conflg.php: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//console/login/LoginForm.jsp: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//console/: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files
Error in runner: Get https://test.site.com//content/: dial tcp: lookup test.site.com on 10.1.94.9:53: dial udp 10.1.94.9:53: socket: too many open files

ehsandeep commented 5 years ago

Also point to note, for the same target it works few times and throws error other time.

joohoi commented 5 years ago

Hi, unfortunately that basically means that you are trying to go too fast for your connection, and the file handles for DNS requests pile up. I have a few ideas about how to fix this in future, but at the least I should make the error message more reasonable.

For the time being lowering the number of concurrent requests (-t 40 for example) should fix it for you. As a quick "non-fix" I could make the error message in this situation say something along the lines of:

Looks like your connection cannot keep up with the rate of requests, consider lowering the amount of concurrent requests (-t)

joohoi commented 5 years ago

As an additional note, the reasons why you might not be seeing this on every run with the same wordlist, and against the same host could be:

The amount of concurrent requests you are currently using is right on the limit of what your connection can handle, so the results fluctuate a bit naturally.
The -t value is too high, but your wordlist is short enough that for some of the runs enough of the file handles get closed before running into the limits.

One way to mitigate the issue, would be to change the ulimit setting, but the issue would still appear when using a longer wordlist.

ehsandeep commented 5 years ago

Thanks for all the suggestion, will lower down the -t and just for your information, I was running this with the wordlist of 6k entries on VPS with good bandwidth connection.

joohoi commented 5 years ago

I dug in a bit, and found a culprit of it. Fix will be implemented in the next release. Thanks for opening the issue! In my own tests the speed went up from ~4,5k req/sec to ~7k sec, which I believe is the limit of my connection.

joohoi commented 5 years ago

Aaand the v0.5 is out. You should now be able to crank the -t way higher than 50.

ehsandeep commented 5 years ago

Thank you for the quick fix!

ffuf / ffuf

getting "too many open files" few times #1