Add support for WhatWG streams

[Pajn]

Now when we have nice standardized streams it would be nice if this library was usable with them. API wise node-csv should probably provide a transform stream which the readable stream from a fetch call could be piped to.

[wdavidw]

Each of the 4 packages supported in this library could wrap the existing Node.js stream API. What you're describing seems very close to what we are already doing. To be honest, this WhatWG initiative makes me feel depressed at the moment. I hope my feelings will evolve but I cannot see it differently than an additional JavaScript madness. Are you willing to contribute?

[petermikitsh]

@Pajn If you're looking to use WHATWG Streams with libraries expecting Node v2 Streams today, there are projects to help bridge the gap. For example, if you're working with File objects, filereader-stream will do this. Here's an example.

A more direct solution would be node-web-streams, but I had some trouble using this module with IE11.

[wdavidw]

Out of curiosity, what is the level of endorsement by the community with WHATWG Streams, is it becoming mainstream?

[petermikitsh]

@wdavidw By no means is this a complete answer, but I think there's mainstream interest. Days ago, a Chromium maintainer stated they're exploring getting ReadableStream integration into the fetch API (the WHATWG spec permits a request body to be a WHATWG ReadableStream). Maybe the others will follow suit. There polyfill is in active development. Things are happening.

[wdavidw]

Correct/confirm my understanding: it is still a bit early to jump into the wagon; if polyfill are emerging, does it means future version of JavaScript will bundle the API like Node.js does or will it remains just a specification?

[petermikitsh]

Probably a little early. The specification was last updated July 29th. WHATWG Streams are least partially implemented in all evergreen browsers-- Chrome 76 already has some of the classes defined in the specification, e.g., ReadableStream, WritableStream, TransformStream, etc. The intention, I believe, is for all browsers to ship Streams natively. The polyfill would be valuable for older browsers which will never get support (e.g., IE11).

[brsanthu]

fyi that ReadableStream is supported in all evergreen browsers. https://caniuse.com/#search=ReadableStream

[dantman]

Support for this would be great if it means I can get rid of the stream package when simply trying to use csv-stringify in the browser without even using streams anywhere in code.

[wdavidw]

Not for tomorrow but I have been considering it lately.

[stormbard]

I think I've hit an issue caused by this. If I take a File from the browser upload and try to pipe directly to the parser as in the stream example I get a 'interface doesn't implement WritableStream' error.

[wdavidw]

There is an "unofficial" and undocumented module implementing WhatWG streams. It is not very tested. Last time I checked, about 2 years ago, it was about 50% slower than the Node.js stream implementation when running with Node.js. Both share the same code, just two different wrappers.

[arukiidou]

Duplex.toWeb Certainly available, but a bit redundant

  const csvParser = parse({ columns: true, });
  const csvFilter = transform<lGov, lGov | null>((data)=> {
    if (data.city.includes("札幌市")) {
      return data
    }
    return null
  });
  const csvStringifier = stringify({header: true, columns: ["pref", "city", "phrase"], quoted: false})
  const filteredStream = utf8EncodedStream[1]
    .pipeThrough(Duplex.toWeb(csvParser))
    .pipeThrough(Duplex.toWeb(csvFilter))
    .pipeThrough(Duplex.toWeb(csvStringifier))