Nuclei transparently capitalizes and cannonicalizes headers

denandz commented 1 month ago

Nuclei version:

v3.8.2

Current Behavior:

Nuclei cannonicalizes headers sent to back end servers. Lower case headers become capitalized and cannonicalized. This prevents nuclei from being able to test bugs which require a specific capitalization of a header name.

Expected Behavior:

User specified headers should remain lowercase

Steps To Reproduce:

nuclei --proxy http://127.0.0.1:8081 -H "some-lowercase-header: foo" -u http://127.0.0.1:8000

And what actually gets sent:

GET /login.jsp HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36
Accept-Encoding: gzip
Connection: close
Content-Type: application/x-www-form-urlencoded
Some-Lowercase-Header: foo

Anything else:

This is due to a know issue in Golang's net/http, where headers are transparently modified as they are read/written from the Headers object (https://github.com/golang/go/issues/37834)

jimen0 commented 3 weeks ago

Although it would be nice to have nuclei preserve the header capitalization it must be noted HTTP/1.1 headers are case insensitive. See https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.

Each header field consists of a case-insensitive field name followed
by a colon (":"), optional leading whitespace, the field value, and
optional trailing whitespace.

For HTTP/2 the standard specifies all headers must be converted to lower case. See https://datatracker.ietf.org/doc/html/rfc7540#section-8.1.2.

Just as in HTTP/1.x, header field names are strings of ASCII
characters that are compared in a case-insensitive fashion.  However,
header field names MUST be converted to lowercase prior to their
encoding in HTTP/2.  A request or response containing uppercase
header field names MUST be treated as malformed ([Section 8.1.2.6](https://datatracker.ietf.org/doc/html/rfc7540#section-8.1.2.6)).

Then on https://datatracker.ietf.org/doc/html/rfc9113#section-8.2.

Field names MUST be converted to lowercase when constructing an HTTP/2 message.

I wouldn't expect the Go issue be resolved with a change as it is currently compliant with the RFCs. This might need forking Go's http2. See https://github.com/golang/net/blob/9617c6335bca5e4e80949a5b1dbe43273260e8a3/http2/write.go#L331-L349.

denandz commented 3 weeks ago

Sure, the RFC's are clear on what compliant behaviour looks like.

The issue is that non-RFC compliant implementations exist, and Nuclei is currently unable to scan them due to Golang's net/http forcing header canonicalization.

ehsandeep commented 3 weeks ago

@denandz for non-RFC compliant you can make use of unsafe: true that uses https://github.com/projectdiscovery/rawhttp instead of net/http

Docs: https://docs.projectdiscovery.io/templates/protocols/http/unsafe-http

example template:

id: basic-raw-example
info:
  name: Test RAW Template
  author: pdteam
  severity: info

http:
  - raw:
      - |+
        GET / HTTP/1.1
        Host: {{Hostname}}
        Origin: {{BaseURL}}
        Connection: close
        User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
        Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        some-lowercase-header: foo

    unsafe: true
    matchers:
      - type: word
        words:
          - "Test is test matcher text"

Example nuclei with no header canonicalization:

echo https://example.com | nuclei -t test.yaml -debug-req

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.8

        projectdiscovery.io

[INF] Current nuclei version: v3.2.8 (latest)
[INF] Current nuclei-templates version: v9.8.9 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 1
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [basic-raw-example] Dumped HTTP request for https://example.com/

GET / HTTP/1.1
Host: example.com
Origin: https://example.com
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
some-lowercase-header: foo

[INF] No results found. Better luck next time!

dogancanbakir commented 2 weeks ago

@denandz, Could you please confirm if what @ehsandeep shared works for you?

denandz commented 1 week ago

Closing an issue as abandoned after a week is pretty brutal there @dogancanbakir

@ehsandeep's suggestion of unsafe: true method works for specific templates, sure. Headers added via -H at runtime; however, get cannonicalized.

dogancanbakir commented 1 week ago

I don't want to be mean! I close issues quickly to keep things moving, but I'm always happy to reopen them with new information or an update. Thanks for the update!

projectdiscovery / nuclei