search

google / gopacket

Provides packet processing capabilities for Go
BSD 3-Clause "New" or "Revised" License
6.3k stars 1.13k forks source link

Handling chunked HTTP responses #112

Open vijayendrabvs opened 9 years ago

vijayendrabvs commented 9 years ago

Hi,

It looks like the http assembly in gopkt is reading in an EOF at the end of the first chunk of a chunked HTTP response, and the TCP assembly underneath never passes on the rest of the chunks to the http assembler. Is there a way to be able to read the rest of the chunked HTTP response and assemble the complete HTTP response?

Thanks, Regards, Vijay

sentryo commented 9 years ago

Do you have an example PCAP capture file that shows the problem? Thanks.

vijayendrabvs commented 9 years ago

Hi @sentryo - I will set one up using a reproducible env in a day ping back. Thanks for the prompt response!

vijayendrabvs commented 9 years ago

@sentryo : I ran a modified version of the http assembly program that comes with gopacket, with a pcap file as you suggested, and interestingly, it works with it. However, the same program does not work in the same way when listening on an interface. This should not be the case, should it? I am attaching both the sample pcap file (which I generated using tcpdump -ni tcp -w chunked.pcap ) and the gopkt program to capture and assemble the http streams, for your reference.

Here's how I run the program on the chunked.pcap file -

go run ./main.go -r ./chunked.pcap -f tcp

To replicate this, you could run a web server that returns chunked HTTP responses, issue a curl to it, while running main.go on it.

PS - Github doesn't let me upload the files with their default names, so I've named them both .png files. Please rename main.png to main.go and chunked.png to chunked.pcap.

chunked main

sentryo commented 9 years ago

Using an offline capture instead of a live capture is usually pretty equivalent, unless you have very heavy traffic... I don't see anything obviously wrong with the HTTP assembly code that could cause this.

Could you try with the new reassemblydump program from pull request #116 ? It's a new package we've been working on which implements a more advanced network reassembly.

Thanks,

vijayendrabvs commented 9 years ago

Hi @sentryo Apologies for the delay in me getting back to you on this. Thanks for the pointer to the PR, I will try out that patch and provide my observations at the earliest.

carwyn commented 7 years ago

I've been trying the reassembly branch that relates to #116 it's made a huge difference to chunked encodings and EoF handling. I would be very interested in testing the latest version of this work mentioned in #116

shixiaofeia commented 4 months ago

使用 gopacket-http, 即可获得拼装好的http.Request 和 http.Response