devanshbatham / ParamSpider

Mining URLs from dark corners of Web Archives for bug hunting/fuzzing/further probing
MIT License
2.53k stars 426 forks source link

Add setting web request proxy option #92

Closed Vincebye closed 1 year ago

Vincebye commented 1 year ago

https://github.com/devanshbatham/ParamSpider/issues/91

Before

[INFO] Fetching URLs for steampy.com
Error fetching URL https://web.archive.org/cdx/search/cdx?url=steampy.com/*&output=txt&collapse=urlkey&fl=original&page=/. Retrying in 5 seconds...
Error fetching URL https://web.archive.org/cdx/search/cdx?url=steampy.com/*&output=txt&collapse=urlkey&fl=original&page=/. Retrying in 5 seconds...
Error fetching URL https://web.archive.org/cdx/search/cdx?url=steampy.com/*&output=txt&collapse=urlkey&fl=original&page=/. Retrying in 5 seconds...
Failed to fetch URL https://web.archive.org/cdx/search/cdx?url=steampy.com/*&output=txt&collapse=urlkey&fl=original&page=/ after 3 retries.

After

v@DESKTOP-DPIU403:/mnt/e/pentest/paramspider$ paramspider -d steampy.com -p '172.26.240.1:7890'

                                      _    __
   ___  ___ ________ ___ _  ___ ___  (_)__/ /__ ____
  / _ \/ _ `/ __/ _ `/  ' \(_-</ _ \/ / _  / -_) __/
 / .__/\_,_/_/  \_,_/_/_/_/___/ .__/_/\_,_/\__/_/
/_/                          /_/

                              with <3 by @0xasm0d3us

[INFO] Fetching URLs for steampy.com
[INFO] Found 9688 URLs for steampy.com
[INFO] Cleaning URLs for steampy.com
[INFO] Found 246 URLs after cleaning
[INFO] Saved cleaned URLs to results/steampy.com.txt
devanshbatham commented 1 year ago

LGTM!