Listening to outgoing http requests

[mwaaas]

Hi Was asking if it's possible to use goreplay to listen to outgoing requests so that can use it as a mock server in production

Here is my user case

• I get request
• save it in the db
• queue the request
• return response
• pick messages from worker
• processing the message involves sending request to another service

We Goreplay I can get all those requests and replay them, but can I replay the response send by worker.

[danihodovic]

@buger I take it that there are no plans of implementing tracking outgoing HTTP requests?

[kaistierl]

I had a look today into the topic of filtering outgoing requests. My use-case is to use goreplay on a reverse proxy server to replay requests sent to one of the backend servers to another "invisible" backend server. This seems not to be possible with the current implementation.

Some simple tests showed that it might be enough to just alter the bpf filters in the raw_socket_listener a bit so that they catch the outgoing traffic instead of the incoming. I did a prototype with a hardcoded filter which seems to work fine so far. I'll try to find the time within the next few weeks to build a full implementation out of the "dirty hack" I did today, making capturing of outgoing traffic a configurable option of goreplay.

@buger is this something you would accept a PR for or would you consider such a feature of "out of scope"? Do you have anything in mind that could cause problems with my approach of altering the bpf filters?

[kaistierl]

...while looking at the command line options even more closely i just stumbled over an option that might make any change obsolete:

-input-raw-bpf-filter string BPF filter to write custom expressions. Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80 '

In my case this is working to capture outgoing traffic that goes to destination 10.0.116.99:8402 and replay it to 192.168.234.47:8402 ./gor --input-raw :8402 --input-raw-bpf-filter 'dst host 10.0.116.99' --output-http http://192.168.234.47:8402

[imann24]

Hi @kaistierl, could I trouble you to share more about how you were able to capture outgoing traffic? I'm specifically interested to hear if anyone has solved this for outgoing requests sent from within a Docker container.

Edit: I may have figured out how to do this for a limited case. From what I can tell, the --input-raw-bpf-filter 'dst host ...' supports IP addresses, not hostnames (I'm not too familiar with BPF filters).

I would still be curious if anyone has found a way to capture the response from the destination? In my testing, I can only capture the outbound request sent to it.

[kaistierl]

Hey @imann24! As far as I understand, it's not possible to capture specific answers to outgoing requests as well. If you want this to work, you need some way to follow the TCP stream in both ways. But the BPF filter acts on pure IP layer and doesn't hold any context of a TCP session so it's not possible for it to determine, what packets are answer packets - let alone what requests they belong to.

At least this is what I think - not an expert here as well ;-)

[imann24]

Thanks very much for the help! This helped my understanding a lot 🙏