FetchObserver (for a single fetch)

[annevk]

In https://github.com/whatwg/fetch/issues/447#issuecomment-281731850 @jakearchibald sketched some APIs based on @stuartpb's work which @bakulf then implemented:

• https://developer.mozilla.org/en-US/docs/Web/API/FetchObserver
• https://dxr.mozilla.org/mozilla-central/source/dom/webidl/FetchObserver.webidl
• https://dxr.mozilla.org/mozilla-central/source/dom/webidl/Request.webidl#59

That issue got closed after we landed a subset of that API in Fetch (you can finally abort a fetch). Let's discuss the remainder here (though not priorities, that's #436).

cc @wanderview

[glococo]

Hi, I found no way of observing the progress on Upload fetch()

var garbage = new Float64Array(new ArrayBuffer(104856)) for (let i=0; i<garbage.length; i++) garbage[i] = Math.random() var gArray = [] for (let i=0; i<100; i++) gArray.push(garbage); var gBlob = new Blob(gArray) //10MB

var controller = new FetchController(); // FF55 or AbortController() in Firefox58 var signal = controller.signal; fetch( "/?action=upload",{ signal, observe(observer) { observer.onresponseprogress=e=>console.log(e);observer.onrequestprogress=e=>console.log(e);observer.onstatechange=n=>console.log(observer.state) }, method:"POST", body:gBlob } )

The only working observer is "state" ( Firefox 55/58 Ubuntu)

[jakearchibald]

The OP reflects the current status. My priorities right now are fetch abort + service worker & background fetch.

[jakearchibald]

Some thoughts on observing H2 push promises https://gist.github.com/slightlyoff/18dc42ae00768c23fbc4c5097400adfb#gistcomment-2227534.

[Cap32]

https://developer.mozilla.org/en-US/docs/Web/API/FetchObserver

This link is 404

[wanderview]

Yeah, I think we removed it from MDN when it became apparent the spec was not immediately moving forward.

[rektide]

Both #51 and #65 are related.

I would love to see this feature added. It would enable some very interesting new reactive models that mesh well with existing HTTP-centric web-systems. Using websocket & SSE is un-ideal in that messaging done there isn't inherently resource-oriented, unlike Push support.

[RikkiGibson]

FetchObserver would enable us to give users a good experience with file uploads without falling back to XMLHttpRequest. Would really like to see it happen.

[annevk]

@RikkiGibson I noticed you're working for Microsoft (not that this is a requirement by all means, but having an employer that supports standards development definitely helps). I don't know how much you're into standards and work with the Edge team, but if you have some time you could help make this happen. We basically need to formalize the proposal and work on tests. There's ample implementer interest already.

[oliverjanik]

Any update on this? Why is such a basic requirement such as upload progress such problem? Why does it take years of debate?

Give me a callback function if you have to. I'll wrap it in RXJS observable myself.

[jakearchibald]

@oliverjanik I think the comment before your's sums up the current situation. If you have time to formalise the proposal and write tests, great!

[jakearchibald]

@rolivo The proposal should probably be formalised first, but if you're interested in contributing tests, https://github.com/web-platform-tests/wpt has the info. https://github.com/web-platform-tests/wpt/tree/master/fetch/api/abort - these are the tests for aborting fetch, in case they're a useful reference.

[oliverjanik]

@jakearchibald Who's working on formalising the proposal and how is the work conducted?

Can I lend a hand?

[jakearchibald]

@oliverjanik the work is conducted right here. No one's working on it right now. It's something I'll get to eventually unless someone (you?) picks it up before me.

[siddharth-meesho]

A lot of us are using fetch for our entire projects & we don't want to implement XHR for just getting progress information while uploading. It's important feature as soon as anybody starts uploading large files. we need to show progress. Is there any update on this? @jimmywarting @bitinn

[jimmywarting]

it's already possible to somewhat get a download progress indicator yourself without any new spec changes. but it's not as fancy/ergonomic as if it would be directly implemented to fetch itself

1. get the response (you have now uploaded everything)
2. read the Content-Length response header to figure out how large the response is
3. get the response.body and pipeThrough a transform stream that count bytes that you receive. now you know how much you have downloaded
4. (optionally) you might want to pipe it to an other dedicated stream handler like the TextDecoderStream to get a strings back or a custom blob builder or anything, here is an example of just piping it to a new reusable Response object so that you can use it as you normally would have

let downloaded = 0
let res = await fetch('./')
console.log('done uploading')
const total = res.headers.get('content-length') // might be null also

// An identity stream (no transformation done) 
const ts = new TransformStream({
    transform (chunk, ctrl) {
        downloaded += chunk.byteLength
        console.log(downloaded) // out of `total`
        ctrl.enqueue(chunk)
    }
})

// **edit** another way of solving it would be to use `response.clone()` maybe instead of constructing a new Response

// create a new Response object /w a new ReadableStream from the transformer
res = new Response(res.body.pipeThrough(ts), res)

// use response as you would normally use
res.text().then(console.log)

as for the upload progress it ain't possible yet as Request don't support ReadableStreams yet afaik

if you would like to get a wider browser support then you have to use getReadable and do more manually reads as pipes are not well supported in browsers other then Blink (?)

Edit Note: This only works for uncompressed responses, gzip, brotli and deflate content-length isn't the same as the uncompressed size that you can read. So you would kind of have to check what content-encoding it's sending also

[hp4k1h5]

I have achieved similar results with almost no perf loss even on 50MB+ responses you can use this like await response.json() ie await prog(response) prog is sometimes faster than .json()

async function prog(response) {
  store.state.search.viewProg = 0
  const reader = response.body.getReader()
  const contentLengthHeader = response.headers.get('Content-Length')
  const resourceSize = parseInt(contentLengthHeader, 10)
  let res = []
  async function read(reader, totalChunkSize = 0) {
    const { value = {}, done } = await reader.read()
    if (done) {
      store.state.search.viewProg = 0
      let r = Buffer.from(res)
      return JSON.parse(r.toString())
    }

    // push new bytes to res
    const length = value.byteLength
    for (let i = 0; i < length; i++) {
      res.push(value[i])
    }

    const percentComplete = Math.round(
      ((totalChunkSize + length) / resourceSize) * 100,
    )
    store.state.search.viewProg = percentComplete

    return await read(reader, totalChunkSize + length)
  }
  return await read(reader)
}

this is in Vue on the server side i am sending

  writeJSON: function(res, json) {
    if (res.finished) return
    const str = JSON.stringify(json)
    res.writeHead(200, {
      // 'Content-Type': 'application/json',
      'Content-Type': 'application/octet-stream',
      'Content-Length': Buffer.byteLength(str),
    })
    res.write(Buffer.from(str))
    res.end()
  },

[wmertens]

@HP4k1h5 nice! Perhaps you could improve it by using a streaming JSON parser like http://oboejs.com/ - then you don't have to store the response in memory before parsing

🤔 although https://github.com/jimhigson/oboe.js/issues/206 complains it's slow - and since response.json() returns a Promise, I wonder if the browser is not already doing streaming parsing - but then you miss the progress reporting.

[hp4k1h5]

i looked at the fetch polyfill when doing mine, and i borrowed some of the idea. i believe this is where theyre transforming the equivalent response.

im using these calls to make large requests and show those (ie, pdfs) in the dom, therefore, i have no real need to pipe the data to a subsequent download, although the data would be instantly available for such.

ill definitely check out the oboe library. i dont believe i can parse the binary of a docx or pdf before the entire document is returned. but it might still be useful for parsing one or the other first if a doc has both.

[mitar]

it's already possible to somewhat get a download progress indicator yourself without any new spec changes. but it's not as fancy/ergonomic as if it would be directly implemented to fetch itself

I do no think this i true if server-side uses content encoding. See #986 for more information about another proposal how to support it for download. But yes, having FetchObserver could address both upload and download.

[jimmywarting]

@mitar you are absolutely right, i have updated my comment, and mention that it only works for uncompressed responses.

[rektide]

I'm super sad that https://github.com/whatwg/fetch/issues/65 & this has been around for half a decade & made seemingly no progress.

I would really like push to be useful on the web. Can we please, pretty please, make a way for the page to be aware that push'es have happened, somehow? It's so sad that we have seemingly no progress on such a basic, logical, powerful, capability of HTTP that is still unusable for websites. Please. Pretty please. I am so sad that we seemingly are no where closer to making the not-even-new-any-more HTTP specification useful on the web. Please. Please. Can we please do something.

[asutherland]

As a heads up, we're probably going to remove Firefox's (never shipped, always disabled via preference) WIP (as implemented in parts 4 and 5 of this bug) related to FetchObserver in an upcoming refactor.

[jimmywarting]

Think this deserves a little pump, priority, notice... have waited far too long for this now. I needed this today for a new project... Thought about using old xhr for progress, but it dose not support streaming :(

sad that firefox pulled the plug on fetchObserver

[piranna]

I'm not so much interested on the uploaded bytes, that I can count them myself before sending the data, that by the acknowledgedBytes, the actual bytes that the server has received, and that can be get from the TCP ACK packages but there's no high level APIs for that except if you do it yourself at application level.

[PodaruDragos]

I am a bit confused about the comments here. What's the blocker for implementing a progress indicator ? A lot of time has passed since this issue was created.

[jimmywarting]

I guess with the new addition of duplex stream (sending & receiving data at the same time) also has to be taken into consideration now for how to design it

[annevk]

https://github.com/whatwg/fetch/issues/607#issuecomment-382625102 is still applicable, but implementer interest might be reduced? As far as I know nobody is actively working on this, but if someone is interested I'd be happy to help them. Best to reach out on https://whatwg.org/chat and not comment here as it ends up pinging a lot of people.

[wanderview]

FWIW, one reason I have seen to implement FetchObserver is that using streams to observe progress of wasm bundle downloads ends up breaking bytecode caching optimizations.

[jakearchibald]

It's also unreliable for uploads right? Since you're tracking the time the data is taken from the stream, which isn't the same as the data being transferred over the network.

[piranna]

It's also unreliable for uploads right? Since you're tracking the time the data is taken from the stream, which isn't the same as the data being transferred over the network.

One thing is when data is sent over the wire, and another want is the acknowledge of the received bytes at the other end. They are different counters, and the useful and needed one is the second one, former one can be already computed at application level just by counting how much bytes we are writting on the stream, but we currently don't have a way to know how much bytes has been acknowledged on the receiver side.

[jimmywarting]

Hmm, just did some backend server stuff where i receive a request event (in particular true for deno, service worker and cloudflare workers) where things are not initiated by a fetch call And i would like to get the progress out of them too So a fetchObserver init option where you pass it down via 2nd argument to fetch() wouldn't work...

but this could:

globalThis.addEventListener('fetch', evt => {
  const request = evt.request
  const response = new Response(...)

  response.onprogress = 
  request.onprogress = console.log

  evt.respondWith(response)
})

[trusktr]

I think this is the last piece for fetch to have a good enough DX. Wanting to measure progress has always been tricky, so having any sort of simple API would be great.

Why not an EventTarget-based solution somewhere in the fetch API? Is there a good reason to have a new event pattern when we already have one?

[dead-claudia]

What's the current state of this, given that Firefox unshipped their idea 3 years ago? Is it in need of a new proposal?

Personally, I feel just a simple progress: {upload(current, total), download(current, total)} on fetch would be enough for my uses.

[annevk]

https://github.com/whatwg/fetch/issues/607#issuecomment-1298738899 is very much still applicable.