search

projectdiscovery / subfinder

Fast passive subdomain enumeration tool.
https://projectdiscovery.io
MIT License
9.71k stars 1.23k forks source link

panic while running job #103

Closed ZeroDot1 closed 6 years ago

ZeroDot1 commented 6 years ago

The error just happened when I checked the URL., I hope the log and the Subfinder version I used can help to solve the problem. subfinder.zip

$ '/home/user/AppImages/subfinder' -v -r 1.1.1.1,8.8.8.8 -d hashpower.co
===============================================
-=Subfinder v1.1 github.com/subfinder/subfinder
===============================================

Running Source: Ask
Running Source: Archive.is
Running Source: Baidu
Running Source: Bing
Running Source: Censys
Running Source: CertDB
Running Source: CertificateTransparency
Running Source: Certspotter
Running Source: Crt.sh
Running Source: Dnsdb
Running Source: DNSDumpster
Running Source: Dogpile
Running Source: Exalead
Running Source: Findsubdomains
Running Source: Googleter
Running Source: Hackertarget
Running Source: Ipv4Info
Running Source: Netcraft
Running Source: PassiveTotal
Running Source: PTRArchive
Running Source: Riddler
Running Source: Securitytrails
Running Source: SSLCertificates
Running Source: Shodan
Running Source: Sitedossier
Running Source: Threatcrowd
Running Source: ThreatMiner
Running Source: Virustotal
Running Source: WaybackArchive
Running Source: Yahoo

Found Wildcard DNS at hashpower.co
 - 185.53.178.6
Running enumeration on hashpower.co

archiveis: Get http://archive.is/*.hashpower.co: dial tcp: lookup archive.is: Temporary failure in name resolution

[DNSDB] www.hashpower.co
riddler: failed to get authentication token

[SECURITYTRAILS] www.hashpower.co
[WAYBACKARCHIVE] hashpower.co
[HACKERTARGET] hashpower.co2018/07/02 19:20:16 panic while running job: runtime error: invalid memory address or nil pointer dereference
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: interface conversion: interface {} is runtime.errorString, not string
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x6bdc30]

goroutine 39 [running]:
github.com/subfinder/subfinder/libsubfinder/helper.(*Pool).subworker.func1(0xc4202c4190)
    /home/user/go/src/github.com/subfinder/subfinder/libsubfinder/helper/pool.go:61 +0x163
panic(0x7148a0, 0x92a970)
    /usr/lib/go/src/runtime/panic.go:502 +0x229
github.com/subfinder/subfinder/libsubfinder/sources/sslcertificates.Query(0xc4201c01a0, 0x2, 0x2, 0x6a6601, 0x940101)
    /home/user/go/src/github.com/subfinder/subfinder/libsubfinder/sources/sslcertificates/sslcertificates.go:35 +0xf0
github.com/subfinder/subfinder/libsubfinder/helper.(*Pool).subworker(0xc4201ee000, 0xc4202c4190)
    /home/user/go/src/github.com/subfinder/subfinder/libsubfinder/helper/pool.go:64 +0x6e
github.com/subfinder/subfinder/libsubfinder/helper.(*Pool).worker(0xc4201ee000, 0x3)
    /home/user/go/src/github.com/subfinder/subfinder/libsubfinder/helper/pool.go:78 +0xb3
created by github.com/subfinder/subfinder/libsubfinder/helper.(*Pool).Run
    /home/user/go/src/github.com/subfinder/subfinder/libsubfinder/helper/pool.go:181 +0x7f
[user@0 ~]$
ZeroDot1 commented 6 years ago 
$ git clone https://github.com/Ice3man543/subfinder.git
Klone nach 'subfinder' ...
remote: Counting objects: 2379, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 2379 (delta 39), reused 77 (delta 30), pack-reused 2279
Empfange Objekte: 100% (2379/2379), 471.49 KiB | 237.00 KiB/s, Fertig.
Löse Unterschiede auf: 100% (1198/1198), Fertig.
[user@0 ~]$ cd subfinder
[user@0 subfinder]$ go get -u github.com/Ice3man543/subfinder
# cd /home/user/go/src/github.com/subfinder/xurls; git pull --ff-only
warning: keine gemeinsamen Commits
Von https://github.com/subfinder/xurls
 + 20b927f...90a0f32 master     -> origin/master  (Aktualisierung erzwungen)
fatal: Verweigere den Merge von nicht zusammenhängenden Historien.
package github.com/subfinder/xurls: exit status 128
Ice3man543 commented 6 years ago

Remove your SubFinder directory completely from the gopath and then try again. If your gopath is '/root/go/' then do rm -rf /root/go/src/subfinder

ehsandeep commented 6 years ago

Hi @Ice3man543,

There is an issue with updating the project using go get -u github.com/subfinder/subfinder

go get -u github.com/subfinder/subfinder
# cd /Users/tech/Documents/go/src/github.com/subfinder/xurls; git pull --ff-only
fatal: refusing to merge unrelated histories
package github.com/subfinder/xurls: exit status 128

rm -rf /root/go/src/subfinder then go get github.com/subfinder/subfinder worked for me.

codingo commented 6 years ago

@Mzack9999 I think this happened when we moved your XURLS fork to be a standardised project. Any ideas?

From what I can see the solution is going to be to remove that directory and re-create it as above.

Perhaps we should put an update.sh script in the main repo to handle this exception (with a prompt) but also to help with the number of go get -u dm's / questions that come with a go project?

eur0pa commented 6 years ago

Can confirm this is happening to me as well on a wildcard domain, throwing panic. Removing completely and re-getting doesn't fix it

Ice3man543 commented 6 years ago

I have no idea what is causing that trouble. Most likely explanation is that changing xurls broke things somehow?

eur0pa commented 6 years ago

No idea either; this is what's panicking on me: subfinder -nW -o test.out -d tesla.services

Ice3man543 commented 6 years ago

@eur0pa reinstall the tool and then try?

eur0pa commented 6 years ago

As I said, rm -rf + go get doesn't fix it

codingo commented 6 years ago

@eur0pa I'm assuming you're working outside of your gopath? If so, removing the main directory is just removing this branch, not the dependencies. You need to remove the path in your gopath, and then do a fresh go get.

We're working on a more permanent solution now. We have test cases / reviewed pull requests in place to prevent this, but this was a change to an outside dependency, and we're going to fix it in a way that it can't happen again.

eur0pa commented 6 years ago

No, I removed ~/go/src entirely

codingo commented 6 years ago

I'll ping you on slack - more information will be useful for us since half the team is travelling at the moment.

Ice3man543 commented 6 years ago

It's a problem with the shitty ssl certificate subdomains extraction code I wrote. Removing it will solve the issue. @eur0pa try without sslcertificate

picatz commented 6 years ago

👋 @Ice3man543 @codingo @eur0pa @ZeroDot1

I'm going to remove sslcertificate ( can be added back later ) to fix this error.

picatz commented 6 years ago

This issue has been fixed in v1.1.3

@ZeroDot1 @eur0pa 🙏 Please let us know if you have any further problems.

ZeroDot1 commented 6 years ago

@picatz Sorry for answering so late, it works. 

[user@0 ~]$ git clone https://github.com/Ice3man543/subfinder.git
Klone nach 'subfinder' ...
remote: Counting objects: 2409, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 2409 (delta 50), reused 101 (delta 38), pack-reused 2279
Empfange Objekte: 100% (2409/2409), 475.94 KiB | 219.00 KiB/s, Fertig.
Löse Unterschiede auf: 100% (1209/1209), Fertig.
[user@0 ~]$ cd subfinder
[user@0 subfinder]$ go get -u github.com/Ice3man543/subfinder
[user@0 subfinder]$ '/home/user/AppImages/subfinder' -v -r 1.1.1.1,8.8.8.8 -d hashpower.co
===============================================
-=Subfinder v1.1.3 github.com/subfinder/subfinder
===============================================

Running Source: Ask
Running Source: Archive.is
Running Source: Baidu
Running Source: Bing
Running Source: Censys
Running Source: CertDB
Running Source: CertificateTransparency
Running Source: Certspotter
Running Source: Crt.sh
Running Source: Dnsdb
Running Source: DNSDumpster
Running Source: Dogpile
Running Source: Exalead
Running Source: Findsubdomains
Running Source: Googleter
Running Source: Hackertarget
Running Source: Ipv4Info
Running Source: Netcraft
Running Source: PassiveTotal
Running Source: PTRArchive
Running Source: Securitytrails
Running Source: Shodan
Running Source: Sitedossier
Running Source: Threatcrowd
Running Source: ThreatMiner
Running Source: Virustotal
Running Source: WaybackArchive
Running Source: Yahoo

Found Wildcard DNS at hashpower.co
 - 185.53.178.6
Running enumeration on hashpower.co

[DNSDB] www.hashpower.co
[WAYBACKARCHIVE] hashpower.co
[VIRUSTOTAL] www.hashpower.co
[FINDSUBDOMAINS] hashpower.co
[SECURITYTRAILS] www.hashpower.co
[HACKERTARGET] hashpower.co
archiveis: Get http://archive.is/*.hashpower.co: dial tcp: lookup archive.is: Temporary failure in name resolution

[Ipv4Info] .hashpower.co
[PASSIVETOTAL] ftp.hashpower.co
[PASSIVETOTAL] mail.hashpower.co
[PASSIVETOTAL] smtp.hashpower.co
[PASSIVETOTAL] www.hashpower.co
baidu: net/http: request canceled (Client.Timeout exceeded while reading body)

Total 5 Unique subdomains found for hashpower.co

.hashpower.co
ftp.hashpower.co
mail.hashpower.co
smtp.hashpower.co
www.hashpower.co

[user@0 subfinder]$