tlsi net/http: HTTP/1.x transport connection broken: malformed HTTP response

[SuperXiaoxiong]

use tlsi parameter will cause transport connection broken: malformed HTTP response

.\nuclei.exe -tlsi -v -u target -t "rabbitmq1.yaml" -duc -debug

[WRN] [c027dc71-857c-4298-affc-231355ebd25d] Could not execute step: [:RUNTIME] got err while executing xxx <- GET xxxx giving up after 2 attempts: Get "xxxx": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\x7f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"
[INF] No results found. Better luck next time!

no tlsi parameter will get the correct result [c027dc71-857c-4298-affc-231355ebd25d:word-1] [http] [low] xxx

if without -debug and -v

[WRN] Found 28 template[s] loaded with deprecated paths, update before v3 for continued support.
[INF] Current nuclei version: v3.2.9 (outdated)
[INF] Current nuclei-templates version: v9.9.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 164
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] No results found. Better luck next time!

Expect :

If no random ja request retry should follow a failed random tlsi request

[RamanaReddy0M]

@SuperXiaoxiong can you provide template?

[SuperXiaoxiong]

id: c027dc71-857c-4298-affc-231355ebd25d

info:
  name: RabbitMQ
  author: superx
  description: RabbitMQ
  severity: info
  tags: panel

http:
  - method: GET
    path:
      - "{{BaseURL}}"

    host-redirects: true
    max-redirects: 2

    matchers:
      - type: word
        words:
          - "<title>RabbitMQ Management"
        part: body

  - method: GET
    path:
      - "{{BaseURL}}/favicon.ico"

    matchers:
      - type: dsl
        name: "rabbitmq"
        dsl:
          - "status_code==200 && (\"1064742722\" == mmh3(base64_py(body)))"

[jimen0]

Seems to work without -tlsi. I can reproduce this with the provided template and the official docker image on the current dev branch. Tested on both Linux and Windows 10 x86_64 Enterprise. Sometimes it just works though.

$ docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management

PS C:\Users\user\Desktop> .\nuclei.exe -tlsi -t .\template.yaml -u https://... -debug
...
[INF] [c027dc71-857c-4298-affc-231355ebd25d] Dumped HTTP request for https://...

GET / HTTP/1.1
Host: ...
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.42
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[WRN] [c027dc71-857c-4298-affc-231355ebd25d] Could not execute step: [:RUNTIME] got err while executing https://1.... <- GET ... giving up after 2 attempts: Get "https://...": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x1e\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@\x00\x01\x00\x00\x10\x00\x00\x04\x00\x10\x00\x00"

Without the tlsi flag it just works. Might be the same as https://github.com/refraction-networking/utls/issues/16.

[ehsandeep]

@SuperXiaoxiong -tlsi is experimental and it's possible server reject the request because of change in tls connection.

[jimen0]

@SuperXiaoxiong As a veeeeery ugly and never production-ready workaround and only if your target HTTP server supports HTTP2 you can address this by rebuilding nuclei with a go mod replace directive on retryablehttp-go and forcing the client (do.go) to use HTTP2:

// Do wraps calling an HTTP method with retries.
func (c *Client) Do(req *Request) (*http.Response, error) {
...
        } else {
            // Attempt the request with standard behavior
            resp, err = c.HTTPClient2.Do(req.Request)
        }

Note this might break a lot of stuff and has only been tested against a docker container behind a reverse proxy that supports HTTP/2. Unfortunately just using the -fh2 nuclei flag isn't enough.

[jimen0]

@ehsandeep this is a bit cleaner but still hacky.

diff --git a/do.go b/do.go
index 8ca57f7..09acaae 100644
--- a/do.go
+++ b/do.go
@@ -53,6 +53,12 @@ func (c *Client) Do(req *Request) (*http.Response, error) {
                // Check if we should continue with retries.
                checkOK, checkErr := c.CheckRetry(req.Context(), resp, err)

+               // if err is about malformed responses then an HTTP/1.1 transport received an HTTP/2 response. Retry with http/2.
+               if err != nil && strings.Contains(err.Error(), "net/http: HTTP/1.x transport connection broken: malformed HTTP response") {
+                       resp, err = c.HTTPClient2.Do(req.Request)
+                       checkOK, checkErr = c.CheckRetry(req.Context(), resp, err)
+               }
+
                // if err is equal to missing minor protocol version retry with http/2
                if err != nil && strings.Contains(err.Error(), "net/http: HTTP/1.x transport connection broken: malformed HTTP version \"HTTP/2\"") {
                        resp, err = c.HTTPClient2.Do(req.Request)

Seems to work for this particular template. If this is something PD wants to battle-test before merging I could open a PR on retryablehttp-go for it.

[SuperXiaoxiong]

Here are the same case

GET / HTTP/1.1
Host: sonarqube.porsche-cloud.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.6.22
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[WRN] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Could not execute request for https://sonarqube.porsche-cloud.com: [:RUNTIME] got err while executing https://sonarqube.porsche-cloud.com <- GET https://sonarqube.porsche-cloud.com giving up after 2 attempts: Get "https://sonarqube.porsche-cloud.com": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@\x00\x04\x00\x10\x00\x00"
[INF] No results found. Better luck next time!

the template

id: 0cc49ffc-8b49-4eca-b015-7bd053743fe8

info:
  name: SonarQube panel detect
  author: x
  description: x
  severity: info
  tags: panel

http:
  - method: GET
    path:
      - "{{BaseURL}}"

    host-redirects: true
    max-redirects: 2

    matchers:
      - type: word
        words:
          - "<title>SonarQube</title>"
        part: body

[tarunKoyalwar]

resolved via retryablehttp-go https://github.com/projectdiscovery/retryablehttp-go/pull/282