sindresorhus / p-map

Map over promises concurrently
MIT License
1.27k stars 58 forks source link

stopOnError doesn't work as expected with high concurrency values #42

Closed Yehonal closed 3 years ago

Yehonal commented 3 years ago

I've an array of millions of elements to save in a database and I want to run those queries in parallel with a concurrency of 1000. I also would like to stop the entire execution if one of those queries fails. However, when is set concurrency: 1000 together with stopOnError: true, the process stops only when all the 1000 promises are resolved (even though the first one fails)

I cannot share my code but it might be tested using the sample from the readme in this way:

import pMap from 'p-map';
import got from 'got';

const sites = [
    'invalidurl',
    ...new Array(10000).map(()=>'https://github.com'),
];

const mapper = async site => {
    const {requestUrl} = await got.head(site);
    return requestUrl;
};

const result = await pMap(sites, mapper, {concurrency: 1000, stopOnError: true});

console.log(result);

Invalid url should throw an error and stop the execution of the other in pending urls