Perhaps a limiter.on('finish', ...) event?

[MikeFair]

I'd like to use a pattern like the following:

  const limit = require('simple-rate-limiter');
  var f = limit(proccessWork).to(1).per(100);

  function batchWork(n) {
    q = getNextWorkBatch(n);
    for (var idx in q) { processWork(q[idx]); }
  }
  batchWork(100);
  f.on('finish', function() { batchWork(100) });

Any suggestions?

(Maybe an on 'begin' event if the queue was empty and the function got called?)

[xavi-]

You should be able to add all the work batches at once. The library will make sure that only one is sent every 100ms. You shouldn't have to add them 100 at a time.

[MikeFair]

I am sending them 100 at a time to control for memory consumption, and to manage the workflow.

There's a few thousand objects, each with hundreds of messages to process, the messages are stored in a big database that can be queried, and the object/message event is getting posted to various URLs (I'm implementing webhooks on the objects).

One thing I'd like to support is for Objects to request a "rollback" to an earlier point and "replay" the messages in order again.

This means invalidating the entire queue, a concept the limiter doesn't support, so I am not using the rate limited function as straightforwardly as to simply match a queued message to a function call because I'm making delivering the messages in order matter.

Instead I use a function on the object to peek at the head item in a queue array, send it, then shift it off if it was successful (got 200 OK response), it calls setImmediate on itself either way (processing whatever message is now at the head of the array). When the array is empty it simply returns.

When I wrote the post, I was using a Promise based mechanism, where if the processing failed, it returned a new Promise instead of resolving/rejecting (injecting itself before the next Promise in the queue chain to retry the message). I was thinking at the time that I could use the rate limiter like a queue to tell me everything had finished, and when to fetch more data.

If you can think of a better way to use the library to ensure message delivery, I'm all ears. Currently I use the .to and .per functions to adjust the backoff period when things start failing, which is a really nice feature.

Thanks!

On Mon, May 14, 2018 at 3:23 PM, Xavi notifications@github.com wrote:

You should be able to add all the work batches at once. The library will make sure that only one is sent every 100ms. You shouldn't have to add them 100 at a time.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/xavi-/node-simple-rate-limiter/issues/11#issuecomment-388982085, or mute the thread https://github.com/notifications/unsubscribe-auth/ACMqLYxTv5vavv0dZfPNJGTSTzaSVhIsks5tygPagaJpZM4T-P3H .