search

yiisoft / yii2-queue

Yii2 Queue Extension. Supports DB, Redis, RabbitMQ, Beanstalk and Gearman
BSD 3-Clause "New" or "Revised" License
1.07k stars 295 forks source link

Write down that redis driver can't handle parallel workers #451

Open wapmorgan opened 2 years ago

wapmorgan commented 2 years ago

I ask to Write it down that library can't handle parallel workers when redis driver used.

What steps will reproduce the problem?

Run few workers with redis driver

What's expected?

Few workers working in parallel

What do you get instead?

Few workers working as 1 worker - spreading payloads in one-thread-work

Additional info

Q A
Yii version latest
PHP version doesn't matter
Operating system RedHat
samdark commented 2 years ago

That is expected from queue but we may, indeed, state it more clearly.

rob006 commented 2 years ago

@samdark I'm pretty sure that Redis driver handles multiple concurrent workers (or at least it did in the past).

samdark commented 2 years ago

@rob006 https://github.com/yiisoft/yii2-queue/blob/master/src/drivers/redis/Queue.php#L134

rob006 commented 2 years ago

@samdark this is about moving delayed and interrupted jobs to waiting queue - one worker can handle this, while the rest will skip it and process actual jobs in queue.

samdark commented 2 years ago

Right. That needs to be verified. I've heard the same info about single worker being active while all others waiting for it to finish.

wapmorgan commented 2 years ago

That is expected from queue but we may, indeed, state it more clearly.

Sorry, but rabbit-driven queue do it in parallel. Inconsistent

After all, In docs there is example with supervisor, where few workers used. Few workers for single-thread work?) doubt

wapmorgan commented 2 years ago

It would be good to clarify which drivers can handle few consumers

rob006 commented 2 years ago

It would be good to clarify which drivers can handle few consumers

AFAIK all (except sync) drivers should support this. How do your run these multiple workers?

wapmorgan commented 2 years ago

It would be good to clarify which drivers can handle few consumers

AFAIK all (except sync) drivers should support this. How do your run these multiple workers?

Few docker containers with entrypoint: php yii queue/listen 3

silverslice commented 2 years ago

I tried to test redis queue in docker, everything works correctly, the workers process jobs in parallel. @wapmorgan, please try my example.

The problem in redis driver is that the message can be lost between pop new message from "waiting" list and place it to the "reserved" set if worker crashes unexpectedly. To be more precise, the message will remain in the "messages" hash, but will not be sent.