refactor web streams to async iterator/generators

[Sec-ant]

Some related links

- [The async iterator and async iterable protocols](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Iteration_protocols#the_async_iterator_and_async_iterable_protocols) - https://streams.spec.whatwg.org/#rs-asynciterator - [MattiasBuelens/web-streams-polyfill](https://github.com/MattiasBuelens/web-streams-polyfill) - [dy/readable-stream-async-iterator.js](https://gist.github.com/dy/8ca154c96e2b2a823c6501d29972b8a8) - [streamxorg/streamx](https://github.com/streamxorg/streamx) - https://github.com/webtorrent/create-torrent/issues/177#issuecomment-1323750843 - https://github.com/webtorrent/webtorrent/issues/1971 - https://github.com/webtorrent/create-torrent/pull/185 - [Some Async Iterators Implemented By @ThaUnknown](https://github.com/ThaUnknown?tab=repositories&q=iterator&type=&language=&sort=) - [Sec-ant/readable-stream](https://github.com/Sec-ant/readable-stream) - [Typescript Module Augmentation](https://www.typescriptlang.org/docs/handbook/declaration-merging.html#module-augmentation) - [rauschma/iterable-to-readable-stream.mjs](https://gist.github.com/rauschma/ef61906c15de3f5a2b73c2d480477359)

Basically, there're two functions that need implementing:

• Consume a readable stream as an async iterator.

  This function is called when input files are consumed as readable streams and we expect to use async generators to manipulate chunks (regulating them to equally sized chunks, applying hash algorithm on them) of the file streams.

  This feature is already merged in the web streams spec, but not yet implemented by mainstream browsers:

  • spec: https://streams.spec.whatwg.org/#rs-asynciterator
  • related issue: https://github.com/whatwg/streams/issues/778
  • pr: https://github.com/whatwg/streams/pull/980
  • tests: https://github.com/MattiasBuelens/wpt/blob/master/streams/readable-streams/async-iterator.any.js

  So, a polyfill is in need for now. There're already some good implementations:

  • MattiasBuelens/web-streams-polyfill Written by the spec contributor Mattias Buelens, this repo is considered canonical but it polyfills all readable stream related APIs which seems not appropriate to be used only as a polyfill of async iteration.

  • ThaUnknown/fast-readable-async-iterator Written by Cas who pointed out the performance issue in using web streams in my repo and implemented async iterations for webtorrent/create-torrent. His implementation uses one-value ahead cache, for performance reasons, I think, and properly cancels the stream when the iteration is returned or thrown, which is good. But I think there're still some problems:

  • The readable stream won't be released when the iteration is normally exited (i.e., loop exits without calls of return or throw)

  • It does not support stream.values({ preventCancel: true })

  • A lack of Typescript types

  There're some other existing snippets that have similar or more significant problems, including this one by ji...@warting.se or this one by dy.

  I intend to implement a spec-compliant polyfill in typescript. Performance is my second consideration as I think browser support will eventually supersede this polyfill soon enough (hopefully). This polyfill I'm working on is Sec-ant/readable-stream. It now passes all the tests that are copied from wpt.

• Construct a readable stream from an async iterator or iterator.

  This function is called when we want to collect the results and produce them as a ReadableStream, which can be later consumed as a Response or be piped to a FileSystemWritableFileStream.

  ReadableStream.from is the proposed API yet not merged in the spec:

  • related issue: https://github.com/whatwg/streams/issues/1018
  • pr: https://github.com/whatwg/streams/pull/1083
  • reference implementation: https://github.com/MattiasBuelens/streams/blob/rs-from/reference-implementation/lib/abstract-ops/readable-streams.js#L1884
  • tests: https://github.com/MattiasBuelens/wpt/blob/rs-from/streams/readable-streams/from.any.js

  Reference implementations:

  • https://github.com/MattiasBuelens/streams/blob/2badb5f15a20531a150af8bebe92daf215b9e645/reference-implementation/lib/abstract-ops/readable-streams.js#L1884
  • https://deno.land/std@0.168.0/streams/readable_stream_from_iterable.ts?source#L39
  • https://github.com/ThaUnknown/webtorrent/blob/b9942016ebfe8224ed0685171f57fd9734554e99/lib/file.js#L147

  I implemented a fromIterable function in Sec-ant/readable-stream to fill in the absence of ReadableStream.from. It now passes all the tests that are copied from wpt.

Other works left including refactoring transformers to async generators are already done in another branch of this repo. But they still need testing as I also refactored many other places.

[ThaUnknown]

rauschma/iterable-to-readable-stream.mjs is bad as it never closes the iterator if the stream is cancelled see for a safer implementation: https://github.com/webtorrent/webtorrent/pull/2414/files#diff-387904f6b07263914bc1b0cb291a27832cc6963e01b5d44b75460771fc25ee8fR153 this doesn't implement lazy loading iterators via start, but you can pretty much cop-paste that

dy/readable-stream-async-iterator.js, see https://github.com/ThaUnknown/fast-readable-async-iterator/blob/main/index.js which also caches 1 value ahead for streams without hwm, both are good

[Sec-ant]

@ThaUnknown Again, thanks for your tips! Yeah, I simply gathered all the related discussions/implementations as a reminder. I went through some of the links and realized some of them might not be what I need but nevertheless steps guiding me to the right direction😄. I really appreciate that many of your repos have offered me much help! I'm currently using a simple implementation written by myself: Sec-ant/readable-stream.