search

devanshbatham / ParamSpider

Mining URLs from dark corners of Web Archives for bug hunting/fuzzing/further probing
MIT License
2.34k stars 403 forks source link

Some regions cannot access web.archive.org, a proxy option is required #91

Closed Vincebye closed 10 months ago

Vincebye commented 10 months ago

When I scan xxx.com, I am prompted that the URL cannot be accessed

[INFO] Fetching URLs for xxxx.com
Error fetching URL https://web.archive.org/cdx/search/cdx?url=xxxx.com/*&output=txt&collapse=urlkey&fl=original&page=/. Retrying in 5 seconds...
Error fetching URL https://web.archive.org/cdx/search/cdx?url=xxxx.com/*&output=txt&collapse=urlkey&fl=original&page=/. Retrying in 5 seconds...
Error fetching URL https://web.archive.org/cdx/search/cdx?url=xxxx.com/*&output=txt&collapse=urlkey&fl=original&page=/. Retrying in 5 seconds...
Failed to fetch URL https://web.archive.org/cdx/search/cdx?url=xxxx.com/*&output=txt&collapse=urlkey&fl=original&page=/ after 3 retries.

So a proxy option is needed

devanshbatham commented 10 months ago

merged, ty