Long polling with queue.get?

[shivanshu3]

Hi,

Is it possible to do long polling using this library? If I understand correctly, queue.get simply passes undefined to the callback if the queue is empty right? Is there a way for it to simply not call the callback until the queue becomes non empty?

[chilts]

It doesn't do long polling currently, since I don't think MongoDB does long polling - and since we're dependent on that, this would be impossible.

I guess the best way to do this would be just 'poll' (rather than 'long poll') and the library could poll for you. e.g. every 5 seconds (opts.interval), for a maximum of 10 times (opts.maxtimes) or something like that.

I'd leave .get(opts, callback) as it was and perhaps add a .poll(opts, callback). Then .poll() could easily just call .get() internally and could also take the same opts.visibility option too which it also just passes on. How does that sound?

I'm not promising I'm going to implement it, though I think it's a good idea. Feel free to send a PR if you're keen. :)

[shivanshu3]

I think you could implement long polling by storing the callback internally inside the mongodb-queue instance when the .get method is called. When .add is called later on, then the callback gets called (if it exists).

I haven't really looked at your code yet. I might try implementing it at some point.

[chilts]

This won't work (in general) since you could be using the MongoDB queue collection from any number of services. ie. your web service might be adding messages to the queue, and a background worker might be taking them off, therefore your suggestion here won't really work. (It won't really harm it either, but it won't do what you're expecting it to.)

Even if it was added like this, there should definitely be an upper limit amount of time to wait before returning undefined. You wouldn't want to wait forever.

[shivanshu3]

Oh I see what you mean. I'm working on an email service and I was planning on using this queue for sending emails and retrying failed attempts. All of this happens on a server within a single Node process.

So this might be a very specific use case I guess? I think this feature would be useful if you're only planning on using this queue from a single service.

[chilts]

I think .poll() would be useful in general, however .longpoll() or the implementation you mentioned in two comment above (three comments ago) wouldn't be.

Anyway, feel free to close if you like. I may have a crack at .poll() sometime, or I may not. :)

[coderofsalvation]

I'm trying to use this polling function:

          var q = mongoDbQueue(db, 'queue')
          var handler = (err, id) => err ? console.error(err) : null                                            
          var poll                                                                                        
          poll = (cb) => {
              q.get( (err, msg) => {  
                  if( err || !msg ) return setTimeout( () => poll(cb), 500 )
                  msg.payload = JSON.parse(msg.payload)
                  cb(err, msg, function(){
                      q.ack(msg.ack, handler)                                                                                   
                      q.clean(handler) // cleanup 
                      poll(cb)         // get next message                                                
                  })
              })  
          }                             
          q.poll = poll

          q.poll( function(err,msg,next){
              // process message
              next()
          })

It's probably not perfect, but we'll see