Network module is missing a mechanism to alter incoming response body

[jimevans]

As currently written, the network module provides network.provideResponse, which allows the user to provide a fully-formed response, including a body. It also provides network.continueResponse to alter some portions of an in-process response that has been intercepted (headers, cookies, etc.). However, there is no current way to examine and manipulate a portion of the body of an intercepted request. This was a known limitation of the initial pass at specifying network interception as a feature, because of the potential large amount of data contained in a response body, and questions about how to best handle that (as a stream, in chunks, etc.). This issue is intended to create a mechanism to fill that gap.

[css-meeting-bot]

The Browser Testing and Tools Working Group just discussed Add support for examining/manipulating intercepted network response bodies.

The full IRC log of that discussion

<AutomatedTester> topic: Add support for examining/manipulating intercepted network response bodies
<AutomatedTester> github: https://github.com/w3c/webdriver-bidi/issues/541
<orkon> ScribeNick: orkon
<orkon> jgraham: with the network request interception, we can change req body and headers but we don't allow intercept the actual network response body and edit it in place
<orkon> jgraham: the reason is that the body can be very large
<orkon> jgraham: sending large bodies in a json base 64 message is problematic
<orkon> jgraham: this is a feature request that you could rewrite the body and we should support it
<shs96c> q+
<orkon> jgraham: so supporting it depends on the general facility on doing streamed IO of large payloads
<orkon> jgraham: basically, it works in CDP: instead of returning text in one go, you return a handle that allows you to pull more data in chunks
<orkon> jgraham: I think it is the most obvious design. For network req interception you also need a way to send the data from the client to the browser (the reverse)
<orkon> jgraham: you might need a write steam in addition to the read stream
<orkon> jgraham: and a way to fall back to the existing stream
<orkon> jgraham: for network request interception it will be an opt in per request
<orkon> jgraham: so you intercept the requests and tell that you will need a body later
<orkon> jgraham: do we want to be declared upfront or a decision during the lifecycle?
<orkon> q+
<orkon> ack next
<shs96c> https://github.com/googleapis/googleapis/blob/6598bda8b438cd39440f71bbe88915587ec79c05/google/bytestream/bytestream.proto
<orkon> shs96c: just two things: we are going to be sending binary data back and forth. JSON is not ideal format for it. At some point we might consider an alternative format for the spec, perhaps, protobuf or some other encoding supporting binary data. Second: we want to read and write memory remotely, ByteStreams might be a good fit.
<orkon> shs96c: perhaps we don't need to create new IO mechanism
<orkon> ack next
<shs96c> Two things. Firstly, I would prefer to decide on a request by request basis. It's more consistent with how puppeteer works, and it allows us to be more flexible (eg. reacting to other data, or modifying data for every N requests)
<shs96c> orkon: Secondly, uploading the binary data after the request, we'll need to see what support we have in CDP. It only supports a base64 encoded body. Not sure if there's a precedent for file upload
<shs96c> s/Two things/orkon: Two things/
<orkon> ScribeNick: orkon
<orkon> q?
<sadym> q+
<orkon> q?
<orkon> ack next
<orkon> q?
<orkon> @sadym: is the use case modifying the response body?
<orkon> Jim Evans: the point of the issue is that I am as a user that I want to modify the entire body or its parts that gets returned in the response
<orkon> Jim Evans: not only replacing the complete body or manipulating the parts of the body
<orkon> sadym (IRC): we can currently fulfil the static body
<orkon> jgraham: we already have it
<orkon> shs96c: only if the response fits into memory, currently no streaming
<shs96c> q+
<orkon> ack next
<orkon> shs96c: the first thing the list of URLs is optional, the second thing is if providing the complete response will be handled in the consistent way, e.g., to indicate if it is a final part of the body. Or we always send the IOHandle
<orkon> q+
<orkon> shs96c: so we need a way to get the body from the response
<orkon> shs96c: but when we do network provideResponse we replace the body with the IO handle
<orkon> shs96c: I guess the IO handle for read will return ???
<orkon> shs96c: it will return network.Bytes
<orkon> and if it is the end it will return eof
<orkon> and write would take a network.Bytes value
<orkon> shs96c: so we perhaps have a read handle and a write handle
<orkon> jgraham: it makes sense to have separate handles
<orkon> jgraham: next steps is to define the details
<orkon> q-

[milahu]

duplicate of #66

sending large bodies in a json base 64 message is problematic

so... just dont do it : ) instead, let a http proxy handle all the unencrypted traffic

this would also allow to use chromium as a full http client

supporting it depends on the general facility on doing streamed IO

IO.write is not implemented in CDP, see also Fetch.fulfillRequest and (very) long body https://github.com/kaliiiiiiiiii/Selenium-Driverless/issues/123#issuecomment-1912534536