search

imranmomin / Hangfire.Couchbase

Couchbase storage provider for Hangfire
https://www.hangfire.io/
MIT License
9 stars 5 forks source link

Chatty - lots of queries #40

Open mgroves opened 4 years ago

mgroves commented 4 years ago

I just noticed that hangfire is making a lot of queries to Couchbase: it seems very chatty, even when not looking at the dashboard. Is this an issue with hangfire, or the couchbase.extension? Is there something I can do in config to cut down on the amount of queries being sent?

Just for instance, I'm seeing these same two queries over and over (like every second-ish):

SELECT RAW Extent1 FROM myhangfirebucket as Extent1 WHERE ((Extent1.type = 3) AND (Extent1.name = 'default')) ORDER BY Extent1.created_on ASC

and

SELECT true as result FROM myhangfirebucket as Extent1 WHERE ((((Extent1.type = 9) AND (Extent1.name = 'locks:job:dequeue')) AND (Extent1.expire_on IS NOT NULL)) AND (Extent1 .expire_on > 1579624268)) LIMIT 1

If these are necessary, could it better be served by using the key/value API instead? I.e. create 'marker' documents and do a key lookup on them instead of a N1QL query? I'm happy to dive into the code to give it a shot, but I'm wondering if you've already considered and eliminated this as a possibility.

imranmomin commented 4 years ago

It is how hangfire works to dequeue the jobs. Based on the queue interval defined hangfire will check the queue to see if any job are left to be processed

I would be interested in learning the key/value API

mgroves commented 4 years ago

The key/value API is already being used in this project, but I'm wondering if there's a way to use it more. So, looking at JobQueue.cs, the Enqueue function creates a new document representing a job in the queue. But Dequeue runs a N1QL query (via Linq2Couchbase). This works, but I'm wondering if there's a different way to do this.

Perhaps in Enqueue, hangfire could just modify another document that represents the queue, then Dequeue would only need to get the one document with key lookup instead of running the query.

The other thing I noticed is the Thread.Sleep in Dequeue. It says the QueuePollInterval is 2 minutes, but based on the console output, it seems to be running more often than that.