Allow iterable to be an async-iterable / async-generator

[thomasjahoda]

I currently see no way of asynchronously and incrementally retrieving the items to process concurrently. For example, if we want to concurrently process a lot of items from a database table, one would have to load at least the ids of ALL elements into memory first to be able to process them concurrently via asyncPool. This is infeasible for lots of data.

If the iterable could also be an async-iterable (e.g. an async-generator), this scenario could be implemented easily and efficiently. Also, this could be implemented in a non-breaking way because checking whether the parameter is an AsyncIterable/AsyncIterator can be done beforehand. See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_async_iterator_and_async_iterable_protocols and https://stackoverflow.com/questions/70337056/verify-iterator-versus-asynciterator-type for this.

Even for little amounts of data, this would make code nicer as not all data has to be fetched and aggregated beforehand.

Would you accept such an extension into the library?

Example code:

import asyncPool from 'tiny-async-pool';

async function main() {
  for await (const result of asyncPool(20, getItems(), (id) => this.processItem(id))) {
    console.log(result);
  }
}

async function processItem(id: string) {
  // heavy computation
  await sleep(1000);
  return "yay";
}

async function *getItems() {
  for (let i = 0; i < 100; i++) {
    await sleep(100);
    yield i;
  }
}

function sleep(durationInMs: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, durationInMs));
}

main();

[Wheredidigo]

This capability is EXACTLY what I was looking for when I came to this library. Would love to know if there are any plans to add this capability, or if anyone else has found another library capable of doing this.

[rxaviers]

Excellent idea and I am open for it. Sorry for the long delay in my response.

[trsvax]

I changed

for (const item of iterable) {

to

for await ( const item of iterable ) {