search

contribsys / faktory

Language-agnostic persistent background job server
https://contribsys.com/faktory/
Other
5.73k stars 227 forks source link

WebUI unresponsive after a while in production #364

Closed amehrdad88 closed 3 years ago

amehrdad88 commented 3 years ago

I have been experiencing the exact same issues as this closed issue: #154

Basically, after a few minutes of restarting the docker container, the WebUI just slows down (all pages excel Home work) but is very slow. All pages other than Home load in 2-3 seconds, Home loads in 10+ but the graphs do not load and the network call to the server fails. Jobs still continue running, no issues there.

Also noticed RAM is creeping up (started at 70mb, going up to 300mb now and keeps going up). Thanks

mperham commented 3 years ago

I need more information. Show me the log output. Show me the contents of the debug page.

amehrdad88 commented 3 years ago

I had to restart the docker container as the debug page was not loading at all but here is what I have for a freshly restarted container, with the following command : sudo docker run -d --restart unless-stopped -v faktory-data:/var/lib/faktorydb -e "FAKTORY_PASSWORD=XXXXXX" -p 0.0.0.0:7419:7419 -p 0.0.0.0:7420:7420 contribsys/faktory:latest /faktory -b :7419 -w :7420 -e production

Overall Redis Redis-2 Redis-3 Disk

The docker logs show this:

dockerLogs

Thanks

mperham commented 3 years ago

You’ve overloaded Faktory. It has a hard limit of 2000 worker connections to Redis.

On Wed, Jun 9, 2021 at 07:57 amehrdad88 @.***> wrote:

I had to restart the docker container as the debug page was not loading at all but here is what I have for a freshly restarted container, with the following command : sudo docker run -d --restart unless-stopped -v faktory-data:/var/lib/faktorydb -e "FAKTORY_PASSWORD=XXXXXX" -p 0.0.0.0:7419:7419 -p 0.0.0.0:7420:7420 contribsys/faktory:latest /faktory -b :7419 -w :7420 -e production

[image: Overall] https://user-images.githubusercontent.com/17452273/121378483-db3c1000-c908-11eb-8c21-a85260b32d79.png

[image: Redis] https://user-images.githubusercontent.com/17452273/121378503-e131f100-c908-11eb-9583-4c7893dd5f53.png

[image: Redis-2] https://user-images.githubusercontent.com/17452273/121378516-e42ce180-c908-11eb-9410-4a54ddf0d2e6.png

[image: Redis-3] https://user-images.githubusercontent.com/17452273/121378540-e727d200-c908-11eb-9588-e824309e48ba.png

[image: Disk] https://user-images.githubusercontent.com/17452273/121378613-f9a20b80-c908-11eb-8382-1069e6f47f46.png

Thanks

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/contribsys/faktory/issues/364#issuecomment-857768265, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAWX2LLXMVEFXZGKE74S3TR56MDANCNFSM46MDZLAA .

mperham commented 3 years ago

Sorry, 1000.

https://github.com/contribsys/faktory/blob/master/cli/cli.go#L174

amehrdad88 commented 3 years ago

Thanks. I just have a few processes currently on Faktory (I use faktory_node_workers) and have 7 workers connected. Is there any other way to overload Faktory? I send about 200 jobs per hour (across 6 workers) and the actual processing of it seems to work well

processes
mperham commented 3 years ago

Sounds like the node worker package might not be pooling connections but rather creating a new connection every time and not closing old connections? Best practice is for each worker process to use a small pool of open connections to talk to Faktory.

On Wed, Jun 9, 2021 at 08:10 amehrdad88 @.***> wrote:

Thanks. I just have a few processes currently on Faktory (I use faktory_node_workers) and have 7 workers connected. Is there any other way to overload Faktory? I send about 200 jobs per hour (across 6 workers) and the actual processing of it seems to work well

[image: processes] https://user-images.githubusercontent.com/17452273/121380851-d710f200-c90a-11eb-8de1-f8feecbb50ab.png

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/contribsys/faktory/issues/364#issuecomment-857785992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAWX6LRF3M5SQIQFJMRS3TR575RANCNFSM46MDZLAA .

amehrdad88 commented 3 years ago

Thank you, will look into that.

jbielick commented 3 years ago

Default concurrency in FWN is 20, which would be a pool of (max) 22 connections per process. That would be well under the 1000 connection limit.

@amehrdad88 could you share your concurrency settings and how you’re starting faktory node worker processes if you open an issue on faktory_worker_node?