[FEATURE REQUEST] http/2 support

[RavySena]

Describe the bug Sometimes on some sites the program freezes right at the beginning of the scan, maybe it's the request for robots.txt that freezes?

To Reproduce Steps to reproduce the behavior:

feroxbuster -r -A -n -k -t 300 --rate-limit 300 -L 300 -u https://domain.com -w {wordlists}/disclosure.txt -H "Cookie:{cookie}" --no-state --json -o {caminho}/{dominio}/ferox/{nome_arq}.json -vv

Traceback / Error Output If applicable, add error output to help explain your problem.

INF      1.949 feroxbuster::event_handlers::scans scan handler received https://domain.com - beginning scan
INF      1.949 feroxbuster::scanner::ferox_scanner Starting scan against: https://domain.com
WRN      8.962 feroxbuster::utils Error while making request: error sending request for url (https://domain.com/robots.txt): operation timed out
WRN      8.964 feroxbuster::event_handlers::scans error sending request for url (https://domain.com/robots.txt): operation timed out

Environment:

• feroxbuster version: 2.10.0
• Linux kali 6.1.0-kali9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1kali1 (2023-05-12) x86_64 GNU/Linux

[epi052]

howdy! can you elaborate on freezing? it never proceeds/finishes? It proceeds after the robots.txt timeout? something else?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[devx00]

This is happening to me as well on v2.10.0. It says operation timed out for the target url but I can access that url just fine from any other application.

Weirdly, if I proxy through burp it works though. I thought it might be something to do with http/2 but I couldn't confirm that.

[epi052]

hey @devx00 could you provide the command you used? is it sporadic, or consistent? if you have a consistently bugged target that you can share, that'd be super helpful

[devx00]

hey @devx00 could you provide the command you used? is it sporadic, or consistent? if you have a consistently bugged target that you can share, that'd be super helpful

Sure let me gather some more data for you. It was happening consistently once it started but prior to that I was able to scan the target.

It could have been some option I started using. I will try and find the specific argument if I can.

Just updating my computer right now in case it was something at the OS level but I doubt it is since everything else could access it fine. I'll try again in a few minutes.

[epi052]

no hurry, thanks for helping run this down!

[devx00]

Ok so I just tried again and I actually can't get it to work at all for this target.

The minimal command that fails for me is

feroxbuster -u https://www.rei.com/ -a 'Firefox/118.0'

The -a 'Firefox/118.0' is necessary because it seems to block certain User-Agents. If you try it with curl you will see.

curl https://www.rei.com/ # fails
curl https://www.rei.com/ -H 'User-Agent: Firefox/118.0' # succeeds

[epi052]

Does rei have safe harbor?

[devx00]

Does rei have safe harbor?

Ya Im scanning for a BBP. https://hackerone.com/rei_bbp/policy_scopes?type=team

[epi052]

perfect, ill look into it

[devx00]

I may see the problem but I'm not positive.

It looks like it doesnt use the supplied headers when performing the connectivity check here https://github.com/epi052/feroxbuster/blob/e3ec3aee3a0ea82cb1ca80c774bd9cd31ac5d6b0/src/heuristics.rs#L106-L111 which would cause it to fail in this case. Am I understanding that correctly? I dont see any context being passed in that would cause the headers to be set.

[devx00]

A little more context for why I was looking into the connectivity check

  TRC      1.806 feroxbuster::banner::container exit: check_for_updates -> UpToDate
  TRC      1.807 feroxbuster::heuristics enter: connectivity_test(["https://www.rei.com/"])
  TRC      1.807 feroxbuster::url enter: format(, None)
  TRC      1.807 feroxbuster::utils enter: parse_url_with_raw_path(https://www.rei.com/)
  TRC      1.807 feroxbuster::url exit: format_url -> https://www.rei.com/
  TRC      1.807 feroxbuster::utils enter: make_request(Configuration::Client, https://www.rei.com/, Default, UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x14500a200, ta
  il_position: 1 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 3, rx_fields: "..." } } })
  TRC      8.809 feroxbuster::utils exit: make_request -> error sending request for url (https://www.rei.com/): operation timed out
  WRN      8.810 feroxbuster::utils Error while making request: error sending request for url (https://www.rei.com/): operation timed out
  WRN      8.810 feroxbuster::utils err: error sending request for url (https://www.rei.com/): operation timed out
  TRC      8.810 feroxbuster::scan_manager::scan_container enter: get_base_scan_by_url(https://www.rei.com/)
  TRC      8.810 feroxbuster::scan_manager::scan_container enter: get_base_scan_by_url -> None
  WRN      8.811 feroxbuster::heuristics error sending request for url (https://www.rei.com/): operation timed out

[epi052]

good thought, but i dont think so. if you check burp, it should go something like

• check for updates (github)
• connectivity
• robots

all of them (minus maybe update check) should have the supplied headers

[epi052]

it looks like a HTTP/2 problem. if i use the rustls backend instead of native-tls, i don't get any timeouts. performance goes down significantly tho. and i know i looked into different tls backends at some point with another guy that was submitting a PR. There were reasons we landed on native, but id have to dig up the convo to remember exatly why

[devx00]

Whats weird is if I proxy it through burp. Literally just add --burp to the request it works.

TRC      1.740 feroxbuster::banner::container exit: check_for_updates -> UpToDate
TRC      1.741 feroxbuster::heuristics enter: connectivity_test(["https://www.rei.com/"])
TRC      1.741 feroxbuster::url enter: format(, None)
TRC      1.741 feroxbuster::utils enter: parse_url_with_raw_path(https://www.rei.com/)
TRC      1.741 feroxbuster::url exit: format_url -> https://www.rei.com/
TRC      1.741 feroxbuster::utils enter: make_request(Configuration::Client, https://www.rei.com/, Default, UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x14d808200, ta
il_position: 1 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 3, rx_fields: "..." } } })
TRC      2.762 feroxbuster::utils exit: make_request -> Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("www.rei.com")), po
rt: None, path: "/", query: None, fragment: None }, status: 200, headers: {"content-type": "text/html;charset=UTF-8", "content-language": "zh-CN", "request-uuid": "ade40062d304", "x-frame
-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "x-permitted-cross-domain-policies": "none", "x-akamai-transformed": "9 - 0 pmb=mTOE,3", "cache-control": "max-age=300", "exp
ires": "Tue, 03 Oct 2023 11:56:24 GMT", "date": "Tue, 03 Oct 2023 11:51:24 GMT", "set-cookie": "akamai_session=23.45.12.4.30032821696333884126; path=/; domain=.rei.com", "set-cookie": "co
okie_policy=CA; expires=Mon, 01-Jan-2024 11:51:24 GMT; path=/", "set-cookie": "EdgeLocation=34.1977,-118.6018; expires=Wed, 04-Oct-2023 11:51:24 GMT; path=/", "set-cookie": "akaas_LaborDa
ySaleOfferCardTest=1696335684~rv=25~id=6156f07d5c67bbb8059e7dc356ee3207~rn=; path=/; Expires=Tue, 03 Oct 2023 12:21:24 GMT; Secure; SameSite=None", "set-cookie": "_abck=9D9D8481AC9DAA69F7
92E5D50AE434D7~-1~YAAQBAwtF+KO19mKAQAA3upg9QpfaZTVbBrRKzgFjktJZDm6qjol2FIxw0Hso5YqmBLiUkuZAU91YbWQKaNQIwN/PwqrE8i/gMyj8H2ITkpOSTjspxDhp+eNi4e0ficDrIyPGo6A9CVfDsd+mfrApn6ojmeqnfJgUcyK7j15K
3qiUVCI8smfNC8FfzwuUCIZmqRa4UPoloibAYbVO3Kbw/e0pw3VZepAFkYeVEdnRi/CeDFDLRImSV5N3aWLrhHkWXUXrBcaWAHfKhTUN7p5TS1MNNoFecM0IyVd6Q+4+8iSHXNtzFdjk5BXiAHbPABm7iUCkmuVfwjaUO+pbe+pwboislrPyfdlU2da
lUSALr995ILTNV4=~-1~-1~1696337458; Domain=.rei.com; Path=/; Expires=Wed, 02 Oct 2024 11:51:24 GMT; Max-Age=31536000; SameSite=None; Secure", "set-cookie": "bm_sz=C18ECACACC5EB9F88C048246E
7968571~YAAQBAwtF+OO19mKAQAA3upg9RWEDhMTtyAF8M8H98f41L6TKca6Oh4eRXxY24ATYqVfXmNWw/KyQuZC4w/JDL96U3i4Lcoi2Tge7b2H8hfBo2TIISE1PM3Ek2vPc7rO8Nx64VrA3DrtoQ6TmYl7Zw3jy6x/w+AaRHk+TSoEUohDIwRbkJC
PVTAmxp5ZOiMHL2Ia0cxDMv60NH/TyJXLPzb9/HLdzghA9DkrDlymkwAGUOWAXjDfi9crDDvjrkPuAYRocdhwhMVwDJ1CaSJIf5H72ragie6C4lehg34lXjE=~4473137~3159864; Domain=.rei.com; Path=/; Expires=Tue, 03 Oct 202
3 15:51:24 GMT; Max-Age=14400; SameSite=None; Secure", "strict-transport-security": "max-age=31536000 ; includeSubDomains", "content-security-policy": "default-src 'self' *.rei.com; frame
-ancestors 'self' https://www.chasepaymentechhostedpay.com *.rei.com *.rei-cloud.com; upgrade-insecure-requests; script-src 'self' 'report-sample' 'unsafe-inline' 'unsafe-eval' blob: http
s://tr.snapchat.com https://*.tealiumiq.com https://*.impactradius-event.com *.klarnacdn.net *.klarnaservices.com *.klarna.com *.klarna.net https://edge.curalate.com https://sc-static.net
https://*.snaplinker.com https://*.xg4ken.com googleads.g.doubleclick.net www.googletagmanager.com www.paypalobjects.com s2.go-mpulse.net js-agent.newrelic.com cdnjs.cloudflare.com cdn.a
vmws.com ssl.avmws.com connect.facebook.net odeaiqfw.micpn.com t.channeladvisor.com triggeredmail.appspot.com www.googleadservices.com www.paypal.com seal.websecurity.norton.com https://*
.googlesyndication.com https://*.channeladvisor.com https://api.smooch.io https://www.google.com https://cdn.speedcurve.com https://www.gstatic.com https://*.demdex.net https://*.googlead
services.com https://*.omtrdc.net https://www.cdn-net.com https://six.cdn-net.com https://prod.accdab.net static.ads-twitter.com https://js-agent.newrelic.com https://view.seekxr.com http
s://www.googletagmanager.com https://www.googletagservices.com https://googletagmanager.com https://www.google-analytics.com https://ssl.google-analytics.com https://google-analytics.com
https://*.ngpvan.com https://cdnjs.cloudflare.com https://secure.gravatar.com https://www.paypalobjects.com https://*.nr-data.net https://www.paypal.com https://translate.googleapis.com h
ttps://translate.google.com https://*.verygoodvault.com https://cdn.gladly.com https://www.youtube.com https://tags.tiqcdn.com https://d3rse9xjbp8270.cloudfront.net https://static.ads-twi
tter.com https://*.ep-mimecast.ads-twitter.com https://analytics.twitter.com https://bat.bing.com https://r.bing.com https://code.jquery.com tags.tiqcdn.com https://*.go-mpulse.net https:
//*.avantlink.com https://cdn.forms.office.net https://activitymap.adobe.com *.yottaa.com *.rei.com *.bazaarvoice.com https://www.myregistry.com *.qualtrics.com https://*.googleapis.com h
ttps://*.twitter.com https://cdn.jsdelivr.net https://cdn.syndication.twimg.com https://player.ooyala.com https://*.theknot.com https://*.richrelevance.com https://mpsnare.iesnare.com pna
pi.invoca.net solutions.invocacdn.com *.avmws.com https://*.microsofttranslator.com https://*.akamaihd.net https://*.speedcurve.com https://cdn.cookielaw.org https://geolocation.onetrust.
com https://js.narvar.com https://player.vimeo.com/ https://api.bluecore.com https://www.chasepaymentechhostedpay-var.com https://www.chasepaymentechhostedpay.com https://analytics.tiktok
.com https://bat.bing.com https://s.pinimg.com; style-src 'self' 'report-sample' 'unsafe-inline' *.bing.com fonts.googleapis.com https://cdn.gladly.com https://*.omtrdc.net translate.goog
leapis.com https://d3rse9xjbp8270.cloudfront.net cdnjs.cloudflare.com secure.gravatar.com www.googletagmanager.com https://*.bazaarvoice.com https://*.snaplinker.com code.jquery.com https
://vjs.zencdn.net https://cdn.jsdelivr.net https://platform.twitter.com https://ton.twimg.com https://player.ooyala.com https://cdn.richrelevance.com https://*.rei.com http://*.rei.com *.
klarnacdn.net https://www.gstatic.com; object-src 'self' https://*.rei.com https://*.online-metrix.net https://www.cdn-net.com https://mpsnare.iesnare.com; frame-src https://secure.everya
ction.com *.airstream.com *.klarna.com *.klarna.net *.klarnacdn.net https://player.simplecast.com https://tr.snapchat.com https://tr6.snapchat.com www.paypalobjects.com tpc.googlesyndicat
ion.com www.paypal.com https://*.doubleclick.net https://www.google.com https://connect.facebook.net *.rei.com *.youtube.com https://www.cdn-net.com https://prod.accdab.net https://*.akam
aihd.net https://view.seekxr.com https://*.facebook.com https://*.demdex.net http://*.demdex.net www.googletagmanager.com www.youtube-nocookie.com *.paypal.com reiadventures.force.com htt
ps://*.verygoodvault.com https://*.avantlink.com https://customervoice.microsoft.com https://*.snaplinker.com https://activitymap.adobe.com *.qualtrics.com https://www.youtube.com https:/
/*.bazaarvoice.com https://*.theknot.com https://*.twitter.com https://r.curalate.com sdx.microsoft.com https://www.chasepaymentechhostedpay.com https://player.ooyala.com https://www.myre
gistry.com https://forms.office.com *.narvar.com https://player.vimeo.com/ https://ct.pinterest.com; child-src blob: www.googletagmanager.com www.youtube.com www.paypalobjects.com *.paypa
l.com; img-src data: blob: *; font-src data: *; connect-src blob: about: https://rei.pxf.io *.klarna.net *.klarnaevt.com *.klarnaservices.com *.klarna.com https://edge.curalate.com https:
//tr.snapchat.com p11.techlab-cdn.com c.go-mpulse.net www.paypal.com *.qualtrics.com www.facebook.com https://api.bluecore.com https://gladly-production.sinter-collect.com https://*.akama
ihd.net https://*.akstat.io pnapi.invoca.net https://*.bing.com wss://*.bing.com https://*.online-metrix.net https://prod.accdab.net https://*.matterport.com https://view.seekxr.com t.co
www.googletagmanager.com www.googletagservices.com www.google-analytics.com https://*.googlesyndication.com fonts.gstatic.com https://translate.yandex.net https://*.ngpvan.com https://*.e
veryaction.com cdnjs.cloudflare.com https://labs.observepoint.com *.gravatar.com https://*.seekxr.com https://*.smooch.io wss://*.smooch.io https://*.gladly.com translate.googleapis.com t
ranslate.google.com https://*.verygoodvault.com https://*.nr-data.net https://*.doubleclick.net https://*.demdex.net https://*.omtrdc.net *.bing.com code.jquery.com *.go-mpulse.net *.akst
at.io https://*.snaplinker.com https://i.embed.ly https://bluecore.app *.paypal.com *.bazaarvoice.com *.google.com *.rei.com https://*.googleapis.com https://player.ooyala.com https://hm.
baidu.com https://*.richrelevance.com https://cdn.cookielaw.org https://geolocation.onetrust.com https://rum.browser-intake-datadoghq.com https://session-replay.browser-intake-datadoghq.c
om https://api.bluecore.app https://siteassets.bluecore.com https://onsitestats.bluecore.com https://www.chasepaymentechhostedpay-var.com https://bat.bing.com https://analytics.tiktok.com
https://ct.pinterest.com; base-uri 'self' https://*.nr-data.net https://recs.richrelevance.com about: *.rei.com; form-action https://tr.snapchat.com https://www.facebook.com https://*.fa
cebook.net https://twitter.com www.cdn-net.com https://home.creditcardila.com https://rei.co1.qualtrics.com https://*.qualtrics.com https://vote.escvote.com http://rei.jobs https://rei.jo
bs https://*.rei.com https://rei.com https://checkout.stripe.com https://*.bazaarvoice.com https://*.twitter.com https://www.reimastercard.com https://www.myregistry.com https://*.google.
com https://forms.office.com https://www.reifund.org; media-src 'self' data: https://cdn.gladly.com https://v2.zopim.com https://static.zdassets.com https://ssl.gstatic.com *.rei.com; wor
ker-src blob: https://*.rei.com; report-uri https://csp-report.browser-intake-datadoghq.com/api/v2/logs?dd-api-key=pubbcf1148cbb6c399749f66f5eed93b535&dd-evp-origin=content-security-polic
y&ddsource=csp-report&ddtags=env%3Aprod", "link": "<https://satchel.rei.com/>; rel=preconnect"} }
TRC      2.770 feroxbuster::heuristics exit: connectivity_test -> ["https://www.rei.com/"]

[epi052]

yea, i suspect burp is transparently handling the http/2 part of the conversation on our behalf

[devx00]

yea, i suspect burp is transparently handling the http/2 part of the conversation on our behalf

Ya that was my initial thought. Any idea why performance decreases when you try to use http/2 natively? I have a hard time believing that Burp's implementation of http in Java could outperform rust.

[epi052]

i don't off hand, i use the reqwest library to handle the actual web stuff. i changed otu the backend to (probably) confirm it's http/2 at fault. There may be a way to get http/2 support without a perf hit. I'll need to dig up that old convo (may be on the feroxfuzz repo as well) and look at reqwest's http/2 capabilities.

[epi052]

actually, nevermind on performance. I was comparing debug build to debug build, which is typically ok as a sniff test. I believe since i was using native-tls in debug, those are already compiled C bindings and reasonably fast. The rustls would have been compiled without optimizations, making it a lot slower

[epi052]

built with --release and it performs as expected

[epi052]

ok, here's the thread i was thinking of.

https://github.com/epi052/feroxbuster/pull/892

rustls probably isn't a good fit for ferox. I'll see what other options are availalbe

[epi052]

i made the following change and everything seems to work. rei.com scans fine, even withotu the firefox header

diff --git a/Cargo.toml b/Cargo.toml
index f1e7a09..e543390 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -35,7 +35,7 @@ tokio = { version = "1.29", features = ["full"] }
 tokio-util = { version = "0.7", features = ["codec"] }
 log = "0.4"
 env_logger = "0.10"
-reqwest = { version = "0.11", features = ["socks", "native-tls"] }
+reqwest = { version = "0.11", features = ["socks", "native-tls-alpn"] }
 # uses feature unification to add 'serde' to reqwest::Url
 url = { version = "2.4", features = ["serde"] }
 serde_regex = "1.1"

[epi052]

also confirmed it doesn't break mTLS

[epi052]

@all-contributors add @RavySena for idea

[allcontributors[bot]]

@epi052

I've put up a pull request to add @RavySena! :tada:

[22Goose]

hi, i'm new here. sorry for bothering.

i just found the same issue when i try to bruteforce the hackthebox machine "nibbles", i use the following command feroxbuster -u http://10.10.10.75/nibbleblog -w directory-list-2.3-medium.txt and it will give the 'operation timed out‘ issue.

i notice the HTTP/2 problem you guys just mentioned, but this page uses HTTP/1.1 btw, i just add the --burp option and it will go well

i dont know where goes wrong thus come here searching for help if u wanna more details just tell me