search

projectdiscovery / katana

A next-generation crawling and spidering framework.
MIT License
12.13k stars 627 forks source link

add -probe feature #289

Closed BrownBearSec closed 1 year ago

BrownBearSec commented 1 year ago

Please describe your feature request:

katana -probe should be able to check if links on a page are live or dead, without going on to crawl those URLs. Eg, Crawling a site example.com, which has the code,

<!DOCTYPE html>
<html>
<head>
    <title>Bingle</title>
    <script src="https://dkasdkasjfkasjf.com/script.js"></script>
    <style>
        body {
            text-align: center;
        }
    </style>
</head>
<body>
    <h1>Bingle</h1>
</body>
</html>

Should return something like {"timestamp":"XXXXXXXX","endpoint":"https://dkasdkasjfkasjf.com/script.js","source":"https://example.com/","status": "FAILED"} if the script link was dead.

Describe the use case of this feature:

Useful for broken link hijacking and subdomain takeovers.

ocervell commented 1 year ago

I would even add more info to the JSON, such as the status code, the content length, number of words, lines in the response, to match for instance httpx output. Currently we have to re-run httpx on all endpoints to get those info.