Observing requests/responses and passing them on?

[nylen]

Normally if node-mitm observes a request, then it is not passed on to the server (as you would expect).

I'd like to observe the request, send it to the server, and observe the response as it comes back, but I'm having a hard time figuring out what to put in the 'request' event handler to make this work. Is this possible?

[moll]

Hey, James! Hmm, doing so is not built-in, that's for sure. So, I don't know of a quick way to do that to suggest you either. You would have to mock or intercept that internal socket object (which you can see in lib/internal_socket.js that Node.js's Net uses for sending out requests.

Have you looked into some existing "proxy" modules? Are there any? Perhaps they do that heavy lifting for you. It does seem like a common problem, at least proxying. Though those proxies might not work with SSL.

[nylen]

Would be nice to be able to do this with mitm, but for now I've released a request-debug module which observes requests made with mikeal/request and their responses.

[moll]

That's cool. I'll put a link to it in the README. ;-)

Mind enlightening me how you use that request observing?

[nylen]

When I'm writing a scraper or something, it is pretty common to have a request fail because the browser does it one way and Node.js does it another way. This makes it a lot easier to figure out which headers are different.

Eventually I'd like to come up with a clever way to use request-debug to save request data, and then use mitm to replay the requests for testing.

[moll]

Okay, debugging differences between requests sounds reasonable.

I wouldn't suggest, however, for you to create tests with entire requests. They're very unmaintainable. You're far better off creating test cases with the minimal reproducible parameters by hand. I briefly mentioned that here, too: https://github.com/moll/node-mitm/issues/1#issuecomment-44026236.

[nylen]

Actually something like Betamax mentioned in #1 is what I really want.

I'm not sure I understand the problem with that approach - my first thought would be to start with the server response and pare it down, removing unnecessary stuff like date and caching headers. Can you give an example of how this would become hard to maintain?

[moll]

Given that to get decent coverage you'd have to ensure every branch, conditional, boundary and statement gets executed, and preferably, do so in separate test cases. Lugging around huge responses for otherwise tiny tests is cumbersome, especially for the code reviewer. There's no reason either to include irrelevant response properties whose handling gets exercised in another test. I'd say white-list approach (select only the properties absolutely necessary) is simpler than trying to pare things down.

If the endpoint's API is unclear or unspecified, then that's a wholly different problem and probably better solved by manual exploratory coding.