dosubot[bot]
commented
7 months ago Hey there, @tanismh! Fancy seeing you here again. Hope you're ready to dive back into the world of troubleshooting with me. 🚀
Based on the information you've provided and the context from the GoReplay repository, it seems like you're encountering an issue with the response body getting truncated despite increasing the snaplen value. This issue is not directly related to the size of the MTU (Maximum Transmission Unit), but rather could be due to limitations or specific behaviors in the packet capturing library or the underlying network stack implementation.
The snaplen parameter in packet capturing tools determines the maximum amount of data from each packet that will be captured and stored. However, this does not necessarily mean that larger application-level messages (like HTTP responses) will be captured in their entirety without being truncated. This is because large messages may be fragmented across multiple TCP packets, and the capture process needs to correctly reassemble these packets.
In the context of GoReplay, there's no direct mention of handling or reassembling fragmented packets at the application layer in the capture package documentation. The focus is on capturing traffic at the IP level and implementing the TCP layer, including flow control but not explicitly mentioning handling of large, fragmented application messages.
To address the issue of truncated response bodies, you might need to look into whether the packet capturing library supports reassembling fragmented TCP segments and if it's correctly handling large messages that span multiple packets. Additionally, it's worth reviewing the configuration and usage of the library to ensure it's set up to capture and reassemble large messages effectively.
Regarding the similar issue you mentioned, can't record POST body data when body is too big（VERSION 1）, the suggested solution was to use the --input-raw-override-snaplen flag to override the capture snaplen to be 64k. However, this solution may not work in all environments, particularly some virtualized ones like OpenStack. In such cases, further investigation into the specific virtualized environment's networking configuration may be necessary.
I hope this information helps. If you have further questions or need more clarification, please let me know.
buger
tanismh
When my response body gets a little bit longer, it gets truncated. I then used the '--input-raw-override-snaplen' parameter to set snaplen to 64KB, and the situation was better, although the response body tolerance was truncated, more content was captured than before setting. So I forced snaplen to change to larger values 256KB, 512KB, etc., but I got the same amount of content as when I set it to 64KB, no better.
At the same time, I print the size of msg.Data in CopyMulty function and find that the '-- input-raw-overrider-snaplen' parameter is not set. The length of msg.Data is 1634 (len(msg.Data)*int(unsafe.Sizeof(byte('a'))), or 4065 if set. I wonder if it has something to do with the size of MTU
The startup command I used is as follows:
./gor --output-http-response-buffer=10485760 --output-kafka-host=localhost:9092 --output-kafka-topic=goreplay --output-kafka-json-format --input-raw-buffer-size=10485760 --input-raw-expire 10s --copy-buffer-size 10485760 --input-raw-buffer-timeout=5s --input-raw-override-snaplen