Streams3 adaptor / transform (s)

[Fishrock123]

Heya @mcollina & @mafintosh - I think this is the next required step here, since we'd need this kinda thing to be able to do anything in node core anyways.

I've created a repo for this at https://github.com/Fishrock123/bob-streams3 and invited you both as collaborators. It contains some basic bits but no functioning code.

There are a good bit of docs and examples lying around here & in the linked-to module repos, please holler if you need any of my help.

[Fishrock123]

Ok so I dunno why I didn't think about this before but pull-stream has modules that do this that I should be able to adapt. I am not sure they do it 100% correctly, but if it works it works.

e.g. https://github.com/pull-stream/pull-stream-to-stream, https://www.npmjs.com/package/pull-to-stream

Edit: thanks to @raynos for this very obvious suggestion. 🤦‍♂

[Fishrock123]

Some semi-working progress here: https://github.com/Fishrock123/bob/tree/streams3-adaptor

Seems to work(?): fs-source -> duplex --pipe-> duplex -> fs-sink

Doesn't work: fs-source -> duplex --pipe-> zlib-gzip-transform --pipe-> duplex -> fs-sink

cc @mcollina

[mcollina]

Have you got a repro for that? What can I run to test?

I would do a slightly different design on the Duplex. I would create an internal bob sink/source, and wrap that in the duplex. I would not expose bob specific methods, such as pull or next. I would also create two more pairs on bindSink/bindSource and have an internal sink/source to match for the counterpart. Something like:

source -> (sink -> duplex) -> zlib -> (duplex -> source) -> sink

I would also avoid listening for "finish" and "close", but rather rely on:

• _destroy
• autoDestroy option

Cheers

Matteo

Il giorno mer 12 giu 2019 alle ore 02:16 Jeremiah Senkpiel < notifications@github.com> ha scritto:

Some semi-working progress here: https://github.com/Fishrock123/bob/tree/streams3-adaptor

Seems to work(?): fs-source -> duplex --pipe-> duplex -> fs-sink

Doesn't work: fs-source -> duplex --pipe-> zlib-gzip-transform --pipe-> duplex -> fs-sink

cc @mcollina https://github.com/mcollina

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Fishrock123/bob/issues/18?email_source=notifications&email_token=AAAMXYZKL3JKQJPGPXEIBUDP2A553A5CNFSM4F7UOTAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXO33OI#issuecomment-501071289, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAMXY4BSGKVZJGXAQXYWBLP2A553ANCNFSM4F7UOTAA .

[Fishrock123]

source -> (sink -> duplex) -> zlib -> (duplex -> source) -> sink

Wouldn't constructing that be really awkward?

Ideally I'd like:

Stream(sourceOrStream).pipe(streams3_transform).pipe(Stream(sinkOrStream))

[Fishrock123]

OK I think it works. Like. Actually. WORKS.

https://github.com/Fishrock123/bob/blob/streams3-adaptor/helpers/bob-duplex.js

Test cases: (manually verify the output file, `readmeorreadme.gz(there is an ungzip helper intests/`). (may require Node 13)_

node tests/bob-streams3-from-source.js readme.md

node tests/bob-streams3-to-sink.js readme.md

node tests/bob-streams3-files.js readme.md

node tests/bob-streams3-zlib.js readme.md

Use NODE_DEBUG=bob to see what is happening.

Replace readme.md with your choice of file.

[Fishrock123]

Ok next bit is to write reasonable automated tests, haha 😬 .