http: allow raw header capture

[codyprime]

The golang textproto library does a few things when parsing the HTTP headers:

• consume some whitespace characters (e.g. \r\n)
• canonicalizes the header keys (e.g. "content-type" => "Content-Type")
• moves the headers into a map

This all makes sense when parsing HTTP, but for a scanner some may want to have the exact headers, to match on order, non-canonical keys, etc.

This adds that option, if '--raw-headers' is specified during an HTTP scan. This is accomplished by implementing a tee reader on the pconn interface, that tees before the bufio reader is put in place. The tee copy can be disabled once the headers have been read, so as to not waste memory while consuming the HTTP body.

How to Test

echo "8.8.8.8"  | ./zgrab2 http -p 443 --use-https --raw-headers --max-redirects=0 | \
    jq -r .data.http.result.response.headers_raw | base64 -d

stdout output should look like:

X-Content-Type-Options: nosniff
Access-Control-Allow-Origin: *
Location: https://dns.google/
Date: Tue, 12 Apr 2022 15:53:04 GMT
Content-Type: text/html; charset=UTF-8
Server: HTTP server (unknown)
Content-Length: 216
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

[ghost]

lgtm

[p-l-]

Hi @codyprime, thanks for this, it will be super helpful. Is there a reason why this has been merged to a feature branch and not to master? Could it be ported to master?

[codyprime]

Hi @codyprime, thanks for this, it will be super helpful. Is there a reason why this has been merged to a feature branch and not to master? Could it be ported to master?

Hi! The reason for the feature branch was because A) I was using the feature branch :) and B) That enabled some volume testing.

I've cherry-picked the commit and created a PR on the master branch: https://github.com/zmap/zgrab2/pull/349