Task Queue Stuck

[thiner]

I was doing the load test. I set the queue-size to 4, and then fired 5 concurrent requests to the API server. All requests are failed. And the server console keeps outputing

[Task Queue] Already waiting for 290.31711424199966S, seq=2
[Task Queue] Already waiting for 290.31784643300125S, seq=3
[Task Queue] Already waiting for 290.3178547669995S, seq=4
^C[Task Queue] Already waiting for 300.32805200299845S, seq=2
[Task Queue] Already waiting for 300.3288617170001S, seq=3
[Task Queue] Already waiting for 300.32879362600215S, seq=4
[Task Queue] Already waiting for 310.33908944799987S, seq=2
[Task Queue] Already waiting for 310.3399451869991S, seq=3
[Task Queue] Already waiting for 310.33989274000123S, seq=4
...

I can't even stop it by Ctrl+C

[konieshadow]

Are you using test tools, like Jmeter or scripts?

[thiner]

I am using hey for load testing. Test script as below:

hey -n 50 -c 4 -t 60 -m POST \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-D data-1pic.json \
http://host:port/v1/generation/text-to-image\?accept=image%2Fpng

[thiner]

and I found that, if I kill the the process by kill -9 pid, the models/checkpoints folder will be renamed to models/checkpoints_corrupted.

[konieshadow]

Maybe an error from previous version of Fooocus-API. Please try with latest version. I just fixed a bug which will make worker queue stucked when load model failed or something like that.

[thiner]

Tested it on the latest version, problem solved. Thanks!