search

apinstein / jqjobs

A job queue engine for PHP.
32 stars 5 forks source link

Increase enqueuing performance #12

Open apinstein opened 11 years ago

apinstein commented 11 years ago

This looks interesting:

https://github.com/ryandotsmith/queue_classic as does https://github.com/chanks/que/blob/master/lib/que/sql.rb

apinstein commented 11 years ago

I did recently switch the postgres locking mechanism from full table locks to row-level locking and that has increased throughput significantly.

I haven't done a lot of benchmarking as the main goal of that refactor was to reduce the incidence of deadlocks on heroku since heroku enforces connection pooling and I was seeing spurious undetected deadlocks due to a worker being killed in a transaction, thus leaving an open transaction with a table lock in the connection pool and completely blocking the system until the entire app was restarted, which would clear the db connection pool.

apinstein commented 11 years ago

I'd also say that I looked at removing the lock used in enqueue() but it didn't seem to have a material performance impact at the tested concurrency (~15).

apinstein commented 10 years ago

the irc#postgresql people suggested advisory locks, but I wasn't able to make them work any faster than my select ... for update (see branch locking-fixes-advisory-instead-of-exclusive)

would be interesting to compare to https://github.com/chanks/que/blob/master/lib/que/sql.rb as they did the same thing but report better results