sindresorhus / p-map

Map over promises concurrently
MIT License
1.27k stars 58 forks source link

Add `pMapIterable` #63

Closed Richienb closed 9 months ago

Richienb commented 1 year ago

Main differences from pMap():

If you're ok with this, I'll add docs and tests

Fixes https://github.com/sindresorhus/promise-fun/issues/21

sindresorhus commented 1 year ago

Bump :)

sindresorhus commented 1 year ago

What's left to do here?

Richienb commented 1 year ago

What's left to do here?

docs

sindresorhus commented 1 year ago

Why do we need the separate backpressure option?

Richienb commented 1 year ago

Why do we need the separate backpressure option?

Values returned by the async iterator can only be consumed one at a time. However, we can run the mapper function concurrency times in parallel. This means there will end up being a backlog of values that are waiting to be consumed. We limit the size of the backlog using backpressure such that we don't allow further calls to the mapper function if doing so would cause it to exceed the backpressure.

Richienb commented 1 year ago

Might be a good idea to try re-implement pMap using pMapIterable to simplify the code and see where its holes are.

sindresorhus commented 1 year ago

Values returned by the async iterator can only be consumed one at a time. However, we can run the mapper function concurrency times in parallel. This means there will end up being a backlog of values that are waiting to be consumed. We limit the size of the backlog using backpressure such that we don't allow further calls to the mapper function if doing so would cause it to exceed the backpressure.

👍

Richienb commented 1 year ago

What should happen when the mapper function returns an error? pMap simply throws the error because it only returns the values when everything finishes. However, pMapIterable streams returned values. Should it just prevent future values down the iterable from running and let the ones before it finish?

sindresorhus commented 1 year ago

An error should generally make things stop as soon as possible, so I don't think it should even let the current ones finish.

Richienb commented 1 year ago

Ok I guess I'll throw out values that arrived out of order after an error.

Richienb commented 9 months ago

@sindresorhus I did stuff

sindresorhus commented 9 months ago

Nice! Big improvement.

sindresorhus commented 9 months ago

Nice work 🙏