Gearmand performance worse than Python gear in our production, is some configuration missing / not correct ?

[pythonerdog]

In our production, Zuul as a gear client, and Jenkins gearman plugin as a gear worker (each jenkins node executor registered as a gear worker).

1. gearmand version: 1.1.21 gearmand -t 0 --job-retries 1 --keepalive --keepalive-idle 600 --keepalive-count 9 --keepalive-interval 75 --verbose DEBUG -p 4730
2. Python gear: 0.16.0 (https://pypi.org/project/gear/) GearServer(4730, host="0.0.0.0", statsd_prefix='zuul.geard', keepalive=True, tcp_keepidle=100, tcp_keepintvl=30, tcp_keepcnt=5)

Running on the same production env, kubernets With gearmand, 13000 tasks can be consumed per hour With Python gear, 24000 tasks can be consumed per hour

Is any comment for this ? Thanks very much

PS: next, we continue try enable gearmand multi-thread with parameter "-t" to further verification.

[esabol]

Well, I imagine -t 0 is the problem. It's not the default, and I wouldn't recommend that for anyone unless they are encountering some weirdness. You've hamstrung gearmand with that setting alone.

[esabol]

You're also not comparing the same keepalive settings. I doubt it matters much, but you should compare the two implementations with the same settings.

[esabol]

-verbose DEBUG is also doing an excessive amount of logging for a production environment. Either get rid of that option entirely or at least change it to --verbose INFO.

Kubernetes? Are you using the Docker image from https://hub.docker.com/r/artefactual/gearmand/ ?

[pythonerdog]

Thanks esabol Will try the test with same parameters. the keepalive may be a suspect and also the "-t 0" And I also notice the other 2 parameter "-b" and "-f" -b [ --backlog ] arg (=32) Number of backlog connections for listen. -f [ --file-descriptors ] arg Number of file descriptors to allow for the process (total connections will be slightly less). Default is max allowed for user.

What do you think about these 2 parameter with default value, and what's that potential impact ?

The docker image is built by ourself RUN wget https://github.com/gearman/gearmand/releases/download/1.1.21/gearmand-1.1.21.tar.gz && tar -zxvf gearmand-1.1.21.tar.gz && cd gearmand-1.1.21 && ./configure --with-boost-libdir=/usr/lib/x86_64-linux-gnu/ --enable-ssl && make && make install && gearmand --help && rm -rf gearmand-1.1.21 && rm -rf gearmand-1.1.21.tar.gz And gearmand process managed by supervisor

Thanks again for your quick support

[SpamapS]

-b will only matter if you have a lot of churn in workers/clients.

from man listen:

   The backlog argument defines the maximum length to which the queue of pending connections for sockfd may grow.  If  a connection
 request  arrives when the queue is full, the client may receive an error with an indication of ECONNREFUSED or, if the underlying
  protocol supports retransmission, the request may be ignored so that a later reattempt at connection succeeds.

For -f, that's likely not important unless you're seeing socket/file errors. Open file limits are mostly meant to stop runaway processes from eating up kernel resources. The only open files gearmand is going to use is sockets or a handful for things like logs or local sqlite files if you're using a background queue plugin. The user-level ulimit will be the highest they can go so this would only be to reduce it anyway.

[esabol]

-b will only matter if you have a lot of churn in workers/clients.

And it seems to me like that could be the case if one is doing performance testing with a trivial worker. So a higher value might be better in this arbitrary scenario?