Demo project fails when using recent versions of this plugin

[edwardotis]

I've tried , 1.2.0, 1.1.9, 1.1.7 and they all hang on enqueing a task. I downloaded the demo with 1.0.6, and it ran fine: https://github.com/Grails-Plugin-Consortium/grails-jesque-demo

Then I tried the demo with 1.1.9, and the demo project also hangs.

I've just confirmed that rolling back to 1.0.6 frees up the enqueing bug on my grails 3.1 project.

[ctoestreich]

Tagging @peh

[edwardotis]

With my setup, I found that "org.grails.plugins:jesque:1.1.4" was the most recent release that works. 1.1.5, 1.1.6, 1.1.7, 1.1.9, 1.2.0 all hung trying to enqueue a task.

I'm using compile 'net.greghaines:jesque:2.1.1' grails 3.1.14 java 1.8.0_111 redis 3.0.7 ubuntu 14.04 lts

[peh]

Hey @edwardotis,

this is by no means a bug in jesque but a question of configuration. As all the versions > 1.0.9 are using proper connection pooling for the redis connection you will have to adjust your applications redis configuration. By default the redis pool is using a maximum of 10 connections. As every worker is constantly using a connection to redis, your application itself might not get a ressource from the pool. So the trick is pretty easy: https://github.com/Grails-Plugin-Consortium/grails-jesque-demo/pull/1/commits/4dc386936b8c83a77d8a022bee2d5ad23204486f#diff-8551624c356f6a57d8412ff7f4072689R8

just set maxActive to something appropiate (this number highly depends on how many workers your are using and how many connections your application might need)

[edwardotis]

Hi @peh,

Thanks for the updated version.

Many popular hosted redis providers only allow 20 to 30 connections in their first tier, so users have to be able to tightly control the number of connections and workers.

https://elements.heroku.com/addons/heroku-redis - 20 max https://elements.heroku.com/addons/rediscloud - 30 max

How many connections are required per worker? I tried 10 workers with a poolsize of 15, and the demo hangs with no exceptions in the logs. I tried 3 workers with poolsize of 15, and the demo hangs, but with exceptions in the logs about hitting maxClients and grails.plugins.jesque.WorkerImpl - Could not return resource to jedis pool. Then the whole thing crashes.

In addition, it looks like the number of workers I put in application.yml actually produces 2x workers in the logs.

INFO grails.plugins.jesque.JesqueService - Starting worker processing queueus: [DemoJesqueJobQueue, JobWithNoQueueJobQueue]

`grails: redis: port: 6379 host: localhost poolConfig: testWhileIdle: true maxTotal: 15 jesque: enabled: true pruneWorkersOnStartup: true createWorkersOnStartup: true schedulerThreadActive: true delayedJobThreadActive: true startPaused: false workers: DemoJesqueJobPool: workers: 3 queueNames:

• DemoJesqueJobQueue
• JobWithNoQueueJobQueue jobTypes:
• org.grails.jesque.demo.DemoJesqueJob
• org.grails.jesque.demo.DemoTwoJesqueJob
• org.grails.jesque.demo.JobWithNoQueueJob`

[edwardotis]

I don't understand why this issue was closed, as it highlighted several fatal bugs around connection pooling and max clients that make >1.1.4 unsuitable for a production environment where a redis server has a finite number of connections available. Jesque >1.1.4 stops processing tasks if it hits the connection limit, and often produces no logging output of the fatal error.

After much testing, the most recent stable version I could find is still 1.1.4 Even when workers in this version try to use more connections than the redis server permits, all the jobs are still quickly dequeued and successfully processed. That's the behavior I need from jesque in production. The same is not the case for any version >1.1.4 including the most current release 1.2.0

[ctoestreich]

It was closed with the merge, not explicitly

[ctoestreich]

My best guess between 1.1.4 and 1.1.5 would be around https://github.com/Grails-Plugin-Consortium/grails-jesque/commit/78dc4aefdcd71956d73b41148f2649ac7158abcc#diff-f41de94a69a6258c3a3e8ea1638b72cb

We hit the same connection size limit several months ago. I don't have access to that code any longer so I will have to see if I can recall what we did. I will also be able to do some debugging next week.

[peh]

Yeah something is weird with returning the resource to the pool. I will debug that stuff over the coming days

[peh]

Okay. i changed the "returning" to directly closing the jedis client. like this i could run a jesque-admin with 2 different jobs and 5 workers with only 10 maxActive redis connections. So like this you should be save to use hosted redis solutions