search

sensepost / gowitness

🔍 gowitness - a golang, web screenshot utility using Chrome Headless
GNU General Public License v3.0
2.87k stars 329 forks source link

HTTP/2 Preflight & Preflight Continuation #128

Open randomactsofsecurity opened 2 years ago

randomactsofsecurity commented 2 years ago

As we discussed in issue https://github.com/sensepost/gowitness/issues/125 I've added support for HTTP/2 requests into the preflight phase. This helps with websites which need HTTP/2 to successfully process (though, most of these websites that need HTTP/2 in the preflight stage probably need the 'accept' headers as discussed in this PR https://github.com/sensepost/gowitness/pull/124 . Not sure if this should be an additional flag or built-in logic... I haven't tested this against a bunch of legacy assets on an internal network which probably like HTTP/1 a lot more than HTTP/2, so maybe this logic is inverted.

The reason I've also added a check for the responsecode != 200 is because sometimes the preflight HTTP/2 would "succeed" but you get a 403 cause its missing the headers. Maybe these headers should/could be hardcoded as well. Lots to think about here.

Additionally, I've reworked the preflight logic and 'purpose'. From my perspective, the goal of gowitness is to capture screenshots, and any other additional capabilities such as response body, titles, TLS certs, fingerprints is just extra cool useful features.

Right now, the preflight logic acts as a way to identify if the host is actually 'up' in order to save time so you don't have to spin up chrome. However.... a lot of the errors occur due to the preflight logic since we're not always setting the correct headers, or emulating the browser when chromedp usually works. So, instead of returning if the preflight fails, I've just removed that block and gowitness will still make a DB record in order to attempt a screenshot capture. Chromedp is actually fairly reliable when it comes to just capturing screenshots, if you rip out the preflight logic entirely you get really good screenshot accuracy.. so the issue is the preflight logic entirely.

So instead of honeywell.com failing in a normal scenario, now it actually does a preflight (preflight fails) but the screenshot is captured.

image

image

Anyways, the way I'm storing a failed preflight but still creating the DB record is kinda dumb and not really 'pretty' so to speak. I'm not sure if there's a better way to stub in that information or not. I think in the future it would be good to have some custom errors (preflight failed, screenshot failed etc), so that even more advanced filtering can be possible but thats a PR for another day.

Chromedp does provide ways to get the response body, title, response headers using RunResponse(), and (as far as I can tell) a single TLS certificate, but I had a tough time actually getting this to work as the way I approached it introduced a lot more complexity into the chrome processor and didn't really save time. Not sure if this would work for all responses either.. In an ideal world, it would be nice if we could get it to work that way, but the tradeoff of building all of that vs not having a % of the response body/TLS certificates is a trade-off I can live with.

leonjza commented 1 year ago

Been pondering this one a long time now, and I think its time to get rid of the preflight logic entirely. For 3.x I think this should be a key feature.