multiple requests in single TCP stream not working

[bob-rohan]

Sending multiple requests via a single TCP stream results in Error reading stream [...] malformed HTTP request on all but the first request.

$ grep 'Root=' vxlan0.pcap
2024/02/15 11:01:09 request &{POST /v2 HTTP/1.1 1 1 map[Accept:[*/*] Accept-Encoding:[gzip, deflate] Content-Length:[316753] Ignore:[ignore] User-Agent:[python-requests/2.31.0] X-Amzn-Trace-Id:
[Root=1-65cdee86-07508b9f39257a373ab7671e] X-Api-Key:[4c544d3d-7d819c337f367025efe7eb5f037148e5] X-Forwarded-For:[10.5.86.81, 10.5.86.81] X-Forwarded-Port:[443] X-Forwarded-Proto:[https]] 0x400
006f2c0 <nil> 316753 [] false batch map[] map[] <nil> map[]  /v2 <nil> <nil> <nil> <nil>}
X-Amzn-Trace-Id: Root=1-65cdee94-22a2dd27344aeeda7185cd62
2024/02/15 11:01:09 Error reading stream 10.5.86.113->10.5.85.94 5602->33452 : malformed HTTP request "X-Amzn-Trace-Id: Root=1-65cdee9e-19ba61000a6fc2b41015ba20"
2024/02/15 11:01:09 Error reading stream 10.5.86.113->10.5.85.94 5602->33452 : malformed HTTP request "X-Amzn-Trace-Id: Root=1-65cdeead-552179dc76d399fc7646b404"
2024/02/15 11:01:09 request &{POST /v2 HTTP/1.1 1 1 map[Accept:[*/*] Accept-Encoding:[gzip, deflate] Content-Length:[316561] Ignore:[ignore] User-Agent:[python-requests/2.31.0] X-Amzn-Trace-Id:
[Root=1-65cdee81-644456fb51bd429d5f9aef95] X-Api-Key:[4c544d3d-7d819c337f367025efe7eb5f037148e5] X-Forwarded-For:[10.5.86.81, 10.5.86.81] X-Forwarded-Port:[443] X-Forwarded-Proto:[https]] 0x400
006f300 <nil> 316561 [] false batch map[] map[] <nil> map[]  /v2 <nil> <nil> <nil> <nil>}
X-Amzn-Trace-Id: Root=1-65cdee8b-723295d9571b133571089814
2024/02/15 11:01:09 Error reading stream 10.5.86.113->10.5.85.94 5612->33452 : malformed HTTP request "X-Amzn-Trace-Id: Root=1-65cdee8f-7854382b049d116e12ba563a"
2024/02/15 11:01:09 Error reading stream 10.5.86.113->10.5.85.94 5612->33452 : malformed HTTP request "X-Amzn-Trace-Id: Root=1-65cdee99-270d654b4050bf1b10e73954"

[bob-rohan]

Is there additional logic required for handling the request delimeter for HTTP POST body?

Some thoughts on HTTP POST request body being quasi delimited by content-length http header for HTTP 1.1 https://superuser.com/a/1430823

Is that covered by this? https://github.com/aws-samples/http-requests-mirroring/blob/60211429ae23fd436c1013bfd9c421bf74de62a8/main.go#L70C1-L70C36

Code in this aws sample very close to the code from the golang sample on similar subject, but can't see anything extra in latest to suggest this scenario would be handled. https://github.com/google/gopacket/blob/32ee38206866f44a74a6033ec26aeeb474506804/examples/httpassembly/main.go#L60

Happy to do some additional gruntwork if someone with the specialist knowledge can guide

[bob-rohan]

Some more exploration on this today.

Looks to be bufio size 4096 bytes. I had wondered if ReadAll() was either

• only pulling the first 4096 bytes
• crudly pulling all requests as part of one big body

I replaced with io.ReadFull using content-length to delimit the amount of buffer to read

                        body := make([]byte, req.ContentLength)
                        _, err := io.ReadFull(buf, body)
                        if err != nil {
                          log.Println("err ", err)
                        }

This resulted in unexpected incomplete request from the buffer suggesting further issues to debug.

[...]
id:1329870_emPOST /v2 HTTP/1.1
X-Forwarded-For: 10.5.87.102, 10.5.87.102
X-Forwarded-Proto: https
X-Forwarded-Port: 443
[...]

[bob-rohan]

Further to above, found packet size exceeds MTU with do not fragement set, infer this means remainder of packet length is dropped.

Testing this across three packets found content is dropped 8886 bytes into the frame. I've re-tested with permissive security group rules to avoid blocking any ICMP PMTU negotiation but there was no change in behaviour.

This issue is related to but potentially distinct from above issue breaking up multiple POST requests within a single TCP stream.

[bob-rohan]

OK I can grab the full content of packets larger than the configured MTU by increasing abitrarily beyond hardcoded 8951 https://github.com/aws-samples/http-requests-mirroring/blob/60211429ae23fd436c1013bfd9c421bf74de62a8/main.go#L212

[chenrui333]

@bob-rohan 👋 I am saying some similar issue and raised a PR suggesting bump the snapLen to 65535

[bob-rohan]

Hey @chenrui333 , the core AWS service which this small code base builds upon, drops packets by design.

https://repost.aws/knowledge-center/vpc-traffic-mirroring

It was flawed to attempt using it to replay and reconcilie production traffic, even if the replay handler works 100% it's invalidated by dropped packets from the underlying AWS VPC traffic mirroring service.

[chenrui333]

I got you, that sounds bad. :(

what approach did you take after all.

[bob-rohan]

Nginx traffic mirror

[chenrui333]

thanks!