search

sensepost / gowitness

🔍 gowitness - a golang, web screenshot utility using Chrome Headless
GNU General Public License v3.0
2.87k stars 327 forks source link

HTTP/2 Support #125

Open randomactsofsecurity opened 2 years ago

randomactsofsecurity commented 2 years ago

Describe the bug Certain websites require an automatic upgrade to HTTP/2

To Reproduce Using an example website such as www.aa.com attempt to capture this screenshot from an AWS/VPC host / socks proxy to a AWS host (honestly not entirely sure what the root cause of this issue is). (Note, you'll need optional headers on this request as seen in the header PR)

image

However, building the project with the following transport settings w/ HTTP/2 in chrome.go as referenced in this article.

"golang.org/x/net/http2"

 err3 := http2.ConfigureTransport(transport)
    if err3 != nil {
        return
    }

image

Might be useful to add this as an optional flag, or a fallback option if a first context-deadline is reached. It doesn't seem to break access to a lot of major websites... but of course on an internal network w/ old appliances I could see this being a problem. Will try to experiment as I run across assets that break with this config enabled.

Expected behavior The preflight and screenshot request to succeed.

Version Information:

Additional context Might be specific to network traffic originating from cloud resources, as I did not notice this behavior on my home network.

randomactsofsecurity commented 2 years ago

Actually, after doing some more testing, this is a valid issue, but I think adding any more improvements into the preflight (where a majority of the connections fail), is probably not worth the effort.

chromedp provides similar stubs for all the titles, response codes, status etc that gowitness is using to populate the db. I ripped out the preflight checks from the code and chromedp finished in the 'same' amount of time. Now, i haven't tried stubbing back in those same things yet to see if it significantly slows down chromedp. But it may be worth it to get rid of preflight entirely, rather than trying to add all the headers/http2/etc that chromedp is already automatically doing by executing a browser session

leonjza commented 2 years ago

Removing the preflight logic is something I've wondered about a lot as well. If that information is obtainable from chromedp during a request then there really is no reason to have it anymore. The preflight logic came from the days when we still manually shelled out to spawn a chrome headless instance.

randomactsofsecurity commented 2 years ago

It's definitely quite a bit more annoying to pull the info (response body, headers) etc from chromedp which adds more complexity w/ listeners. I wonder if there's a better HTTP library to use that would 'emulate' a browser better.

I'm also not sure where I can pull the TLS information from chromedp either, but its somewhere in the docs I'm sure