async scan and custom transforms

[jeromew]

Hello,

I tried to use the scan function with an async iterator just to realize that the iterator has to be synchronous.

do you see another way of doing an async scan ? if not, should scan be modified to accept the fact that the iterator be more complex (async, promise) or would it be interesting to have it as a new primitive ?

[greelgorke]

you could implement your own scan. just copy the code of scan and make the call to the reducer function async.

[jeromew]

yes no problem tha is what I did (use my own scan)

do you think there could be room for an async scan in core ? or a way to have the 2 mechanisms (sync & async) work with scan ?

By the way, when developing my own scan, I used this trick to add a function to highland :

var proto = Object.getPrototypeOf(_());
proto.customScan = function(z, f) { ... }

is that a correct way to add a userland, non-native function to highland ?

[greelgorke]

i'm not sure about async/sync. it would be nice to have async versions of map, reduce etc. but with async-versions of the methods or make them handle both under the hood, i don't know which is better. (tbh. i didn't think about it yet)

however, i wouldn't expose your custom scan like you did. why do you want to attach it to the prototype? and why do you override the core scan?

[jeromew]

The overriding of the core 'scan' was an error in my example ; I changed it to customScan.

is there another way than attaching to the prototype ? I must say I found this method but there is maybe a simpler solution (?) I thought it was necessary to add it on the prototype of the Stream object.

What would be the way to create an npm module delivering custom highland transform or create plugins to highlandjs ?

[greelgorke]

there is a discussion about modulariation of the lib, but it's kind of on hold. The idea here is to set up highland as a plugin system to make it easier to extend.

[jeromew]

ok thanks I found the discussion on https://github.com/caolan/highland/issues/3 and will look into it.

I close this issue for now as I guess that 'async scan' is not a very high priority in the core transforms and to want to pollute the issue list.

[juanpaco]

I know this is old, but I was trying to figure out how to do an async reduce. I solved it by doing a consume that would collect the things I wanted to reduce in an async manner, and then calling reduce on the resulting stream.

That's more or less what an async reduce would have been under the hood, but it didn't require patching highland itself.

[apaleslimghost]

Haskell has foldM, scanM is a logical extension of that. Given our current naming scheme (map/flatMap, filter/flatFilter) we could have flatReduce and flatScan? I think these would be useful additions.