search

gearman / gearmand

http://gearman.org/
Other
740 stars 137 forks source link

gearmand with mysql queue type - high CPU load #156

Open remizyaka opened 6 years ago

remizyaka commented 6 years ago

Hi, Installed latest version of gearmand(1.1.18) on ubuntu 16. Have 12 different workers but queue is empty most of the time. But for some reason gearman consuming to much CPU resources. 30-100% all the time. /usr/sbin/gearmand --listen=127.0.0.1 --pid-file=/run/gearman/server.pid --log-file=/var/log/gearman-job-server/gearman.log --queue-type=MySQL --mysql-host=localhost --mysql-port=3306 --mysql-user=username --mysql-password=password --mysql-db=dbname --mysql-table=gearman_queue --http-port 90210 --hashtable-buckets=1733003 What can cause such a big CPU load on server without active jobsa and 0 queue? How can I optimise it?

remizyaka commented 5 years ago

any thoughts on this?

esabol commented 5 years ago

Sorry you haven't received a response. I don't use any of the persistence plugins personally. I think the consensus by those who have used them is that you shouldn't. They're not recommended because they doesn't scale well, which is probably what you are seeing. The recommended design pattern is to write workers that store your jobs in the database and re-submit your jobs from the store.

References:

I hope this helps!

remizyaka commented 5 years ago

@esabol cant see in docs that it is not recommended approach http://gearman.org/manual/job_server/#persistent_queues

Also I do have persistence besides gearman but I also whant gearman jobs in-memory queue will have some back off in case of failure. Also I don't think that it's about scaling as gearman starts eating CPU even on low jobs rate when the queue is empty most of the time

esabol commented 5 years ago

@esabol cant see in docs that it is not recommended approach http://gearman.org/manual/job_server/#persistent_queues

Yeah, it’s not in the docs. It’s more of a “best practices” thing or “expert opinion.”

Also I do have persistence besides gearman but I also whant gearman jobs in-memory queue will have some back off in case of failure.

If you read the links I provided, that’s exactly what that design pattern does.

p-alik commented 5 years ago

other reports regarding mysql queue