[dns] wordlist line count is always higher by one

[sheran]

go run main.go dns -w words.txt -d example.com -r dnsserver:5335 --delay 1000ms
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain:     example.com
[+] Threads:    10
[+] Delay:      1s
[+] Resolver:   dnsserver:5335
[+] Timeout:    1s
[+] Wordlist:   words.txt
===============================================================
Starting gobuster in DNS enumeration mode
===============================================================
Found: webmail.example.com

Progress: 4 / 5 (80.00%)
===============================================================
Finished
===============================================================

cat words.txt
webmail
email
dashboard
admin

My DNS scans usually show one extra line in the wordlist file, which is different from the actual line count. This means my scans always complete "partially". Not a huge deal, but thought to flag this. Switching count to 0 instead of 1 here kinda solves the issue but it breaks tests:

https://github.com/OJ/gobuster/blob/697dd9dd11c078caced3f992307dcd9ac9245c0d/libgobuster/helpers.go#L73

I understand that this may be a design decision to handle working with goroutines as referenced in: https://github.com/OJ/gobuster/blob/697dd9dd11c078caced3f992307dcd9ac9245c0d/libgobuster/helpers_test.go#L172

But at the same time, I have verified with ngrep on my unbound DNS server that all 4 requests are indeed sent out. I don't know if this will vary with bigger wordlist files as I have yet to test them with different files. In the grand scheme of things, this is just an off by one count and I think we should all be able to live with it, but in the interest of accurate reporting, I guess this should be considered for a fix?

[firefart]

does your wordlist contains a new line at the end? to get the line count we simply count the new lines in the file so usually this off by one error is caused by a trailing new line

[sheran]

thanks for the quick response @firefart! Negative, I made sure that there was no empty line with \n on there:

➜  sheran@leonov gobuster git:(master) ✗ cat words.txt|wc -l
       4
➜  sheran@leonov gobuster git:(master) ✗ cat words.txt
webmail
email
dashboard
admin
➜  sheran@leonov gobuster git:(master) ✗ cat words_newline.txt|wc -l
       5
➜  sheran@leonov gobuster git:(master) ✗ cat words_newline.txt
webmail
email
dashboard
admin

➜  sheran@leonov gobuster git:(master) ✗

Here is the output with the wordlist with the blank line:

go run main.go dns -w words_newline.txt -d example.com -r dnsserver:5335 --delay 1000ms
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain:     example.com
[+] Threads:    10
[+] Delay:      1s
[+] Resolver:   dnsserver:5335
[+] Timeout:    1s
[+] Wordlist:   words_newline.txt
===============================================================
Starting gobuster in DNS enumeration mode
===============================================================
Found: webmail.example.com

Progress: 5 / 6 (83.33%)
===============================================================
Finished
===============================================================

[firefart]

@sheran I added a more accurate (but sadly slower) linecounter on the dev branch. It now also ignore comments in lines which where reported before. Can you please check this version?

To install: go install github.com/OJ/gobuster/v3@dev

[sheran]

Yes, the dev branch addresses the behavior that I observed, and is accurate in reporting. I'll take a deeper look at the linecounter code on this branch and see the internals, thank you.

[firefart]

thanks for the feedback. I’m still thinking about dropping support for comments and switch back to the faster counter. With the current one counting rockyou.txt can take over 1 second compared to 0.04 seconds with the old method. Reopening this until I came to a decision

[sheran]

That's a significant drop in speed. I'll do some testing this week as well because I want to understand the original lineCounter in more depth.

[firefart]

I switched back to the fast method and dropped supports for comments in wordlists. The only thing this method can't handle correctly are empty wordlists but that's a corner case I think