DJWassink / Promise-parallel-throttle

It's kinda like Promise.all(), but throttled!
MIT License
81 stars 3 forks source link

Some problems #18

Closed monolithed closed 7 years ago

monolithed commented 7 years ago

The first attempt

let queue = [];

for (let index = 0; index < count; index++) {
    queue.push(async () => {
        let response = await request('https://...');

        if (response.status !== 200) {
              throw new Error(`The current status code "${response.status}" is not expected`);
                }

        return response;
    });
}

let [ response ] = await PromiseThrottle.all(queue, 10, true);
 Deliveryd
>> The current status code "500" is not expected
>> Could not send data
    1) Multiple requests (count)
>> The current status code "500" is not expected
>> The current status code "500" is not expected
>> The current status code "500" is not expected
>> The current status code "500" is not expected
>> The current status code "500" is not expected
>> The current status code "500" is not expected
....

  0 passing (4s)
  1 failing

  1) Deliveryd Multiple requests (count):
     Error: Promise rejected with no or falsy reason
      at process._tickCallback (internal/process/next_tick.js:109:7)

The second attempt

let [ response ] = await PromiseThrottle.all(queue, 10, true, ({ amountRejected, taskResults }) => {
    if (amountRejected > 0) {
         throw new Error(`The current status code "${response.status}" is not expected`);
    }

    status.taskResults.some((task: any) => {
        if (!task || task.status !== 200) {
                throw new Error(`The current status code "${task.status}" is not expected`);
        }
    })
});
ç(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 948): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 949): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 950): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 951): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 952): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 953): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 954): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 955): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 956): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 957): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 958): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 959): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 960): 2
(node:30279) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 961): 2
^Csh: line 1: 30278 Terminated: 15          mocha --timeout 50000 tests
npm ERR! Test failed.  See above for more details.
....

My temporary solution is:

let queue = [];
let errors = {};

for (let index = 0; index < count; index++) {
    queue.push(async () => {
        let response = await request('https://...');

        if (response.status !== 200) {
                      errors = response;
                }

        return response;
    });
}

let [ response ] = await PromiseThrottle.all(queue, 10, true);

if (errors.status !== 200) {
     throw new Error(`The current status code "${errors.status}" is not expected`);
}

The problems:

  1. { amountRejected, taskResults, ... } (due to progressCallback: Function) are not defined in d.ts
  2. The 3rd param failFast seems not to be worked as well
  3. it's unknown how to throw an exception to prevent the task execution
  4. taskResults[x] may be empty:

     { status: 200,
       htmlencoded: false,
       email: 'test.box_wuzpx688@list.ru',
       body: [Object] },
     , 
     ,
     ,
     ,
     ,  // How is it possible?
     ,
     ,
     ,
     ,
     ,
     { status: 200,
       htmlencoded: false,
       email: 'test.box_wuzpx688@list.ru',
       body: [Object] },
     ,
     ,
     ,
     ,
DJWassink commented 7 years ago

It's not completely clear what the issues is but let me clarify a bit what we are seeing here.

The all and sync methods return a array containing either the result of a task or the thrown error. If you would like the filter out the errors from the results please use the raw method which returns a Result object containing the indexes of the resolved/rejected tasks. (it should be easy to map these arrays with the result array to only get your rejects / resolved tasks).

Now about the failFast not seeming to work is because the initial queue gets run with 10 promises. Since it's kinda impossible to abort promises which already started, the failFast only kicks in on the 11th promise.

Please let me know if I missed something in your question!

monolithed commented 7 years ago

Got it, thanks!