Streaming upload with `fetch`

[jimmywarting]

What version of Bun is running?

1.0.2+37edd5a6e389265738e89265bcbdf2999cb81a49

What platform is your computer?

Darwin 22.4.0 arm64 arm

What steps can reproduce the bug?

Bun.serve({
  port: 3434,
  fetch(req) {
    console.log('received headers from client', Object.fromEntries(req.headers))
    console.assert(req.body !== null, 'expect body to have something')
    return new Response(req.body)
  }
})

const response2 = await fetch('http://localhost:3434', {
  method: 'POST',
  body: new Response('abc').body,
  duplex: 'half'
})

console.log(Object.fromEntries(response2.headers))
console.log(await response2.text())

What is the expected behavior?

There where some weird stuff going on here... i tried to make a echoing server that spitted out the response back to the client.

I was surprised to see that i got a back a content-length. that should not happen if a) use request.body as a ReadableStream b) and it should not happen if i return a response with a ReadableStream.

it should be chunked transfer.

if i instead changed it to new Response('abc').body.pipeThrough(new TransformStream())

Bun.serve({
  port: 3434,
  fetch(req) {
    console.log('received headers from client', Object.fromEntries(req.headers))
    console.assert(req.body !== null, 'expect body to have something')
    return new Response(req.body)
  }
})

const response2 = await fetch('http://localhost:3434', {
  method: 'POST',
  body: new Response('abc').body.pipeThrough(new TransformStream()),
  duplex: 'half'
})

console.log(Object.fromEntries(response2.headers))
console.log(await response2.text())

then the content-length is zero?! and the response is empty.

What do you see instead?

I see content-length being added even tough it's a ReadableStream upload.

I guess somehow Bun is able to figure out how large a ReadableStream is? but piping it via a TransformStream will make it unable to determinate what the size is, so it sends a content-length: 0 when it shouldn't.

likewise, if i do:

const response2 = await fetch('http://localhost:3434', {
  method: 'POST',
  body: new ReadableStream({
    start(ctrl) {
      ctrl.enqueue(new Uint8Array([97,98,99]))
      ctrl.close()
    }
  }),
  duplex: 'half'
})

then i get this logs:

received headers from client {
  accept: "*/*",
  "accept-encoding": "gzip, deflate",
  connection: "keep-alive",
  "content-length": "0",
  host: "localhost:3434",
  "user-agent": "Bun/1.0.2"
}
⚠️ expect body to have something
{
  "content-length": "0",
  date: "Sun, 24 Sep 2023 20:11:58 GMT"
}

In Summary

if body is a ReadableStream, don't add content-length

[jimmywarting]

This also do not work:

const rs = new ReadableStream({
  start(ctrl) {
    ctrl.enqueue(new TextEncoder().encode('hello'))
    ctrl.enqueue(new TextEncoder().encode('world'))
    ctrl.close()
  }
})

fetch('https://httpbin.org/post', {
  method: 'POST',
  duplex: 'half',
  body: rs
}).then(r => r.json()).then(console.log)

expected json data to incl "helloworld"

[aifrim]

Bun: 1.0.26+c75e768a6 Platform: Darwin 23.3.0 arm64 arm

I have also tried to fetch(url, { body: stream }). My use case is a bit different. I have 3 servers. 1 & 2 are the ones doing all the work and the 3rd one is just there for me to gather some metrics.

Basically the insights server will tee the request.body (if present) and call both servers on the desired endpoints with the given method and headers.

Everything goes through correctly but the body. And that shows as well since the two servers receive the content-length set to 0.

I have even tried to mock the body before sending each request and still remained unsuccessful by either tee-ing a mocked stream or by creating two separate streams.

From what I can tell, if you give fetch a body that is a Readable<UInt8Array> stream then it will send and empty body.

[riywo]

FYI: Probably the related code is here: https://github.com/oven-sh/bun/blob/4b0eb4716447f46763552045bfcd77c15c64ed91/src/http.zig#L2157

[gro-ove]

Tried all sorts of ways to track down upload progress without using something like curl or a secondary service, but sadly it doesn’t seem possible with Bun for now. I’m guessing a fix for this issue could make the most straightforward solution possible though