More sources

[shelld3v]

The API request to fetch subdomains: https://jonlu.ca/anubis/subdomains/{domain}

[shelld3v]

Hi, saw you pushed the source, appreciate it!! And here is another source that you need to add too:

ThreatBook: https://api.threatbook.cn/v3/domain/sub_domains?apikey={API-KEY}&resource={DOMAIN} (API key required)

Response:

{
    "data": {
        "domain": "yahoo.com",
        "sub_domains": {
            "total": "10000",
            "data": [
                "news.yahoo.com",
                "research.yahoo.com",
                "my.yahoo.com",
                ...
            ]
        }
    },
    "response_code": 0,
    "verbose_msg": "OK"
}

[shelld3v]

Others:

IPv4Info: Steps:

• GET http://ipv4info.com/search/NF/yahoo.com
• Grep /dns/(.*?)/yahoo.com in the response for the token
• GET http://ipv4info.com/subdomains/{TOKEN}/yahoo.com for first subdomains.
• Now make a loop:
  • In http://ipv4info.com/subdomains/{TOKEN}/yahoo.com, there is the URL for the next page. It's something like this: /subdomains/{ANOTHER_TOKEN}/page{NUMBER}/yahoo.com
  • And on that page, there is the URL for another next page (this is the loop) with a different token and a different page number (Note: the page number is from 1->2->3->...)
  • Break the loop if you can't find any token anymore, because that means out of subdomain

Reference: https://github.com/projectdiscovery/subfinder/blob/master/v2/pkg/subscraping/sources/ipv4info/ipv4info.go

---

Archive.Today: You can't get all subdomains from Archive.Today because of the captcha.

• GET http://archive.is/*.{DOMAIN}

---

NetCraft:

• GET https://searchdns.netcraft.com/?restriction=site+contains&host={DOMAIN}&from=1#then+21+then+41

And

• GET https://searchdns.netcraft.com/?restriction=site+ends+with&host={DOMAIN}&from=1#then+21+then+41

There is a verification check if you send too many requests, but you can bypass it with a different User-Agent for each request

[shelld3v]

Okay, so here are top 10 most valuable sources from subfinder:

1. anubis
2. sublist3r
3. hackertarget
4. bufferover
5. alienvault
6. crtsh
7. threatcrowd
8. certspotterold
9. sitedossier
10. riddler

[shelld3v]

Hey, don't add NetCraft yet since there are some problems in getting the next page! I will tell you soon about this

[hazcod]

@caffix I wonder if we simply can integrate subfinder as a source?

[shelld3v]

subfinder is a tool that fetches subdomains sources, the same as amass, but amass has more sources! So, I don't think that adding subfinder as a source makes sense

[shelld3v]

@hazcod you can try these ADS scripts: https://gist.github.com/sillydadddy/b1726c8e8ce281d55b82d4e2a1a610e8 (amass enum -d example.com -scripts /path/to/those/scripts)

[caffix]

Examples of incorporating other tools using Amass Data Source scripts has been demonstrated in various training videos. One such script is shown below.

` name = "test" type = "api"

function start() setratelimit(1) end

function vertical(ctx, domain) local cmd = outputdir(ctx) .. "/bin/assetfinder --subs-only " .. domain

local data = assert(io.popen(cmd))
for line in data:lines() do
    newname(ctx, line)
end
data:close()

end `

[shelld3v]

Hey @caffix, I saw that ArchiveToday has been deleted because "it does not work outside a browser". But, I'm seeing that subfinder still can find subdomains from http://archive.is!!! (5 xoom.com subdomains)