jlongster / transducers.js

A small library for generalized transformation of data (inspired by Clojure's transducers)
BSD 2-Clause "Simplified" License
1.72k stars 54 forks source link

Fork/join and general parallelization? #36

Open chadobado opened 9 years ago

chadobado commented 9 years ago

Curious how forking and joining might be accomplished with this library?

..similar to this.

Something like this looks really verbose (and isn't really parallel), so assuming there is a better way?

var processedBags = []

var transformFragile = compose(
    filter(x => x.bagType == 'fragile'),
    map(x => merge(x,{luggageTag: 'Handle with care'}))
);

var transformHeavy = compose(
    filter(x => x.bagType == 'heavy'),
    map(x => merge(x,{luggageTag: 'Watch your back!!'}))
);

var transformNormal = filter(x => x.bagType == 'normal')

processedBags = into(processedBags, transformFragile, bags)
processedBags = into(processedBags, transformHeavy, bags)
processedBags = into(processedBags, transformNormal, bags)

Also attempted other variations w/a switch statement inside a mapper. Also doesn't feel right.

Thanks in advance.

jaawerth commented 9 years ago

You'd probably want to do it with iterators. Fork spits out an object of key-iterator pairs - you can then keep working on them as needed and merge them back together. The bags use-case probably isn't a good one for forking (there's a better way to do it), but here's what forking would look like at least how I sometimes do it:

// Reusable fork and merge
var fork = filtersObj => itemList => seq(filtersObj, map( ([key, filter]) => [key, toIter(itemList, filter)] ));
var merge = forkedObj => toArray(forkedObj, mapcat(kp => kp[1]));

var transformFragile = compose(
    filter(x => x.bagType === 'fragile'),
    map(x => Object.assign({}, x, {luggageTag: 'Handle with care'}))
);
var transformHeavy = compose(
    filter(x => x.bagType === 'heavy'),
    map(x => Object.assign({}, x, {luggageTag: 'Watch your back!!'}))
);
var transformNormal = filter(x => x.bagType === 'normal')

var transform = compose(merge, fork({transformFragile, transformHeavy, transformNormal}));
var processedBags = transform(bags);