Chaining and Continuing request or response modification

[Justin99b]

Im trying to use multiple "addRequestRule" and modifying the Request in the first one and use the modified request in the next one

Is something like this possible?

Here is a more "visual" representation of what i want to do

Get XYZ -> UserAgentMatcher -> UserAgentHandler1 (replaces the user agent) -> UserAgentMatcher (dosent trigger anymore because the user agent was modified) -> -> OtherMatcher -> Other Handler (has the modified header set by the previous handlers)

[pimterry]

Right now, no, that's not possible. Each request is tested against the registered matchers once, and once it's found a match it runs a single handler which handles the request.

I have been looking into supporting a multi-step handlers, although I hadn't planned to add multi-step matching as well.

Multi-step handlers would look like something like this:

If request is a GET to /xyz:

• Replace the user-agent header with X
• Wait 1 second
• Proxy the request to the server upstream

The code could look something like:

server.forGet('/xyz')
  .setHeader({ 'user-agent': 'X' })
  .delay(1000)
  .thenPassThrough();

Of course the steps could be scriptable (like the existing callback handler) so that you could define a step en route like .run((req) => { if (req.headers['user-agent'] === ... }).

Something similar is already supported in MockRTC: https://github.com/httptoolkit/mockrtc/#example. This shares some internal components with Mockttp, but focuses on mocking WebRTC traffic instead.

Would something like that work for your use case?

[Justin99b]

The problem is while this could be usefull for a simple setHeader i would actually need to be able to replace certain strings within the user agent aswell. Same goes for the body.

I might share my workaround when im done. The only solution i see is just to use a "catch all" handler which checks everything. The handler would extend PassThroughHandler and override the handler if the request should not passthough. Its pretty ugly but its the only way i know right now

Would be helpfull if i would be able to use the new ThingMatcher().matches" inside a beforeRequest. Problem is the typ difference between ongoingRequest and completedReqeust

[pimterry]

The problem is while this could be usefull for a simple setHeader i would actually need to be able to replace certain strings within the user agent aswell. Same goes for the body.

Yes - but I'm discussing the possible steps structure, rather than the specific step handlers available. It could just as well be .replaceHeader('user-agent', /abc/, 'def') apply a regex replace instead. The structure is the important question to design this.

In a hypothetical API built with one matcher + many step-handler commands like this (any commands you like) do you think what you want to do could work?

I might share my workaround when im done.

Please do! Yes, of course none of this is available now, so workarounds will be required.

For traffic that will be proxied, using beforeRequest and/or beforeResponse are the best current options for custom logic like this. Within those callbacks you can use any logic you like modify the message content and then send it upstream (or directly send a response without proxying, or even just kill the connection if you prefer).

Would be helpfull if i would be able to use the new ThingMatcher().matches" inside a beforeRequest

You might be able to fudge things to reuse matcher code, but to be honest the matchers aren't very complicated, so I would just duplicate it in your callback. That will also be more flexible. The source is here: https://github.com/httptoolkit/mockttp/blob/main/src/rules/matchers.ts. Most of them are just one line.

Probably doesn't matter for your case, but just FYI: the differences between OngoingRequest & CompletedRequest are important, most notably CompletedRequest is only available with the full body data, while with OngoingRequest the body has not necessarily been received yet (and might never be received). Not waiting for that data is important for matching performance.

[Justin99b]

In a hypothetical API built with one matcher + many step-handler commands like this (any commands you like) do you think what you want to do could work?

I think i dont quite get what you mean but a "nextStep" inside the addRequestRule would do the job Otherweise yes one matcher and all logic inside the Handler is what im doing right now and trying to make this work.

the differences between OngoingRequest & CompletedRequest are important, most notably CompletedRequest is only available with the full body data, while with OngoingRequest the body has not necessarily been received yet (and might never be received).

Why does OngoingRequest not have the body? We are talking about the reqeust that is about to be send to the upstream (or whatever i want to do with it beforehand). And the http/s request is jkust being send to the proxy so shouldnt both have the full request on them? I dont understand how you would wait for the body when its already in the http request that is being send in the proxy or am i missunderstanding something?

Really usefull information tho!

[Justin99b]

export class AllHandler extends PassThroughHandler {
  // @ts-ignore
  public readonly beforeRequest(request: CompletedRequest) {
    return new ReplaceUserAgentHandler().beforeRequest(request);
  }

  // @ts-ignore
  async handle(clientReq: OngoingRequest, clientRes: OngoingResponse, options: RequestHandlerOptions) {
    if (await new RequestHasSpecificUserAgentMatcher().matches(clientReq)) {
      await super.handle(clientReq, clientRes, options);
    }
    // Timeout if no matches
    await new TimeoutHandler().handle();
  }

  explain(): string {
    return `..."`;
  }
}

export class ReplaceUserAgentHandler extends PassThroughHandler {
  // @ts-ignore
  public beforeRequest(request: CompletedRequest) {
    let agent = request.headers['user-agent'];

    agent = agent.replace('Something', 'Other');
    agent = agent.replace('yay', 'woob');

    request.headers['user-agent'] = agent;
    // @ts-ignore
    return request;
  }

  explain(): string {
    return `Replace the user agent`;
  }
}

I have a lot of @ts-ignore's because the types are doing weird thing. I think its my lint config but i dont know

This way i am able to go though multiple steps while maintaining your usual approach to handling request and using matchers.

[pimterry]

Those workarounds make sense! Thanks, that's an useful example to keep in mind in future.

Why does OngoingRequest not have the body? We are talking about the reqeust that is about to be send to the upstream (or whatever i want to do with it beforehand). And the http/s request is jkust being send to the proxy so shouldnt both have the full request on them? I dont understand how you would wait for the body when its already in the http request that is being send in the proxy or am i missunderstanding something?

If it's a small message, the body & the request metadata (headers/url/method) will all arrive at once, so you're right that we could have all the data immediately and this would be fine.

Many messages aren't small though. If the body is gigabytes. We'll receive the request details & headers a long time before we receive the full body.

For streaming requests, there might not even be a fixed body size - a client can open a connection, make a request, and then keep it open to slowly send body data occasionally (e.g. send a temperature measurement every second) almost indefinitely (this is much more common in the server->client direction, but it's possible the other way too).

To handle all of this correctly, you have to stream requests - that means as soon as the initial part of the request is received, matching & handling starts, before the body arrives. For proxied requests, we start connecting to the upstream server and forwarding data upstream before the body has necessarily been sent by the client at all (this is very important for big requests - in extreme cases we might not have enough RAM to buffer the whole body otherwise!)

None of this really matters for your case, but it's interesting to understand how this works under the hood :smile:

[Justin99b]

Many messages aren't small though. If the body is gigabytes. We'll receive the request details & headers a long time before we receive the full body.

Oh i didnt know that! Is that part of the http specification? Never heard of it so happy i learned something new ^^

I always love learning new things and im happy i could provide a nice example for you :)

Maybe you can add it to a Example in the docs or something? maybe other people can make use of this