search

sindresorhus / p-queue

Promise queue with concurrency control
MIT License
3.39k stars 182 forks source link

Possibility to tap into already executed data #207

Open stychu opened 2 months ago

stychu commented 2 months ago

Hi I want to hit my external API with bunch of requests. API allows for 30 TPS. I need to execute 500 requests which each request returns 100 objects that I need to save to the database. Is it possible to tap into the queue while the queue is still running and access already resolved data? It will be a bit heavy process on memory where I need to wait till the whole queue resolve and only then I can access my in memory result of 50 000 objects to be saved into database.

If I have pqueue as 

const queue = new PQueue({
    intervalCap: 30,
    interval: 1000,
    carryoverConcurrencyCount: true,
  });

const result = await queue.addAll([500requestPromises])

I would like to be able to tap into the data and every batch save to database so approx. 17 writes to database. Is it possible ??