[BUG] Directories are not enumerated if directory has no trailing slash

[wilco375]

Describe the bug When Feroxbuster discovers a link with no trailing slash, it is not further enumerated.

To Reproduce

1. Save this simple example server as index.py:

   
   from flask import Flask, render_template, request
   import base64

app = Flask(name)

@app.route('/', methods=['GET']) def index(): return '', 200

@app.route('/test', methods=['GET']) def test(): return 'Hello world', 200

@app.route('/test/a', methods=['GET']) def a(): return 'Hello world', 200

@app.route('/test/image.png', methods=['GET']) def image(): return base64.decodebytes(b'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=='), 200

2. Run `python3 index.py`
3. Run `echo -e "a\n" | feroxbuster -u http://localhost:5000 -w /dev/stdin --force-recursion`

**Expected behavior**
`/test/image.png` is discovered using HTML parsing. Then Feroxbuster also checks if any parent directories exists and enumerates these directories (i.e. `/test` and `/test/a`)

**Traceback / Error Output**
Only the link itself is discovered

---

| | |) |) | / ` / \ _/ | | \ | | | | \ | \ | \, _/ / \ | |_/ | by Ben "epi" Risher 🤓 ver: 2.10.1 ───────────────────────────┬────────────────────── 🎯 Target Url │ http://localhost:5000 🚀 Threads │ 50 📖 Wordlist │ /dev/stdin 👌 Status Codes │ All Status Codes! 💥 Timeout (secs) │ 7 🦡 User-Agent │ feroxbuster/2.10.1 🔎 Extract Links │ true 🏁 HTTP methods │ [GET] 🔃 Recursion Depth │ 4 🤘 Force Recursion │ true ───────────────────────────┴────────────────────── 🏁 Press [ENTER] to use the Scan Management Menu™ ────────────────────────────────────────────────── DBG 0.198 feroxbuster sending ["http://localhost:5000"] to be scanned as initial targets INF 0.198 feroxbuster::event_handlers::scans scan handler received http://localhost:5000 - beginning scan INF 0.198 feroxbuster::scanner::ferox_scanner Starting scan against: http://localhost:5000 404 GET 4l 34w 232c Auto-filtering found 404-like response and created new filter; toggle off with --dont-filter DBG 0.238 feroxbuster::filters::wildcard filtered out http://localhost:5000/a DBG 0.238 feroxbuster::filters::container filtering response due to: WildcardFilter { content_length: Some(232), line_count: Some(4), word_count: Some(34), method: "GET", status_code: 404, dont_filter: false } INF 0.239 feroxbuster::event_handlers::scans Added new directory to recursive scan: http://localhost:5000/ DBG 0.240 feroxbuster::extractor::container Parsed link "/test/image.png" from http://localhost:5000/ DBG 0.241 feroxbuster::filters::wildcard filtered out http://localhost:5000/test/ DBG 0.241 feroxbuster::filters::container filtering response due to: WildcardFilter { content_length: Some(232), line_count: Some(4), word_count: Some(34), method: "GET", status_code: 404, dont_filter: false } DBG 0.242 feroxbuster::extractor::container Extracted File: FeroxResponse { url: http://localhost:5000/test/image.png, method: GET, status: 200 OK, content-length: 86 } 200 GET 3l 5w 86c http://localhost:5000/test/image.png 200 GET 1l 2w 54c http://localhost:5000/ INF 0.949 feroxbuster All scans complete!

**Environment (please complete the following information):**
 - feroxbuster version: 2.10.1
 - OS: Ubuntu 22.04.1

**Other information**
If the test route is defined as `/test/` (with trailing slash), Feroxbuster does function as expected

[epi052]

hey there, thanks for submitting this!

What you're seeing is expected behavior (see logic below). forced recursion should be able to get the results you're after (the recursion logic is laid out in plain language on that page as well)

    pub fn is_directory(&self) -> bool {
        if self.status().is_redirection() {
            ...
        } else if self.status().is_success() || matches!(self.status(), &StatusCode::FORBIDDEN) {
            // status code is 2xx or 403, need to check if it ends in /

            if self.url().as_str().ends_with('/') {
                log::debug!("{} is directory suitable for recursion", self.url());
                return true;
            }
        }

[wilco375]

Hi @epi052, thanks for your response. As you can see from the command provided above, I am already using --force-recursion. Is there anything else I should add?

[epi052]

Ooooh, missed that, my bad. Lemme take a look, iirc that should be recursing then.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[epi052]

i started looking into this, but haven't arrived at a suitable solution yet. pinning for now