search

ipld / js-car

Content Addressable aRchive format reader and writer for JavaScript
Other
46 stars 7 forks source link

Add support for WebStreams #120

Open 2color opened 1 year ago

2color commented 1 year ago

Background

I'm exploring verified (trustless) retrieval of CAR files (See https://github.com/olizilla/ipfs-get/issues/15) in browsers.

I discovered that there's no way to instantiate a CarReader from a ReadableStream returned from the Fetch API in the browser.

This is because Readable Web Streams are not iterable by default.

Adding support for Web Streams

For us to be able to promote verified retrieval in browsers, it would be nice to have full CAR support in browsers. For this, we need a way to instantiate a CAR Reader in the browser from a Readable Web Streams.

My current work around is to use @achingbrain's browser-readablestream-to-it library

RangerMauve commented 1 year ago

In my experience, one can convert ReadableStreams into async iterators with either polyfills for a built in [Symbol.asyncIterator]() method, or using the example in the mdn docs which can convert to one in code.

Is this something we can add docs in the README for, or is it absolutely necessary to be able to accept ReadableStreams as arguments?

RangerMauve commented 1 year ago

The spec discussion for making them async iterable can be found here: https://github.com/whatwg/streams/issues/778#issuecomment-461341033

I think the current best practice is to include a polyfill in your code for it since it's slowly making it's way into implementations.

2color commented 1 year ago

or using the example in the mdn docs which can convert to one in code.

That converts it the other way (iterator -> stream)

Is this something we can add docs in the README for, or is it absolutely necessary to be able to accept ReadableStreams as arguments?

That's not for me to decide, but I'd add a note about this, at the very least. Accepting ReadableStreams would be a nice DX improvement, given the primacy of web APIs and their universality.

BigLep commented 1 year ago

2023-02-21 maintainer conversation: IPLD maintenance team is a skeleton crew right now. Is this something you would want to pick up @2color ? @rvagg is happy to review or answer questions.

2color commented 1 year ago

That's understandable.

I don't have the capacity right now. But I'll revisit this if time allows.

RangerMauve commented 1 year ago

Might experiment with this in WebRecorder since we're working on some custom chunking to CAR use cases.