[link]https://github.com/jaeles-project/gospider[/link]
[short_descr]GoSpider - Fast web spider written in Go[/short_descr]
[tags]directories,endpoints[/tags]
[long_descr]
Features
Fast web crawling
Brute force and parse sitemap.xml
Parse robots.txt
Generate and verify link from JavaScript files
Link Finder
Find AWS-S3 from response source
Find subdomains from response source
Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault
Format output easy to Grep
Support Burp input
Crawl multiple sites in parallel
Random mobile/web User-Agent
[/long_descr]
[link]https://github.com/jaeles-project/gospider[/link] [short_descr]GoSpider - Fast web spider written in Go[/short_descr] [tags]directories,endpoints[/tags] [long_descr] Features Fast web crawling Brute force and parse sitemap.xml Parse robots.txt Generate and verify link from JavaScript files Link Finder Find AWS-S3 from response source Find subdomains from response source Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault Format output easy to Grep Support Burp input Crawl multiple sites in parallel Random mobile/web User-Agent [/long_descr]