search

dusterio / laravel-aws-worker

Run Laravel (or Lumen) tasks and queue listeners inside of AWS Elastic Beanstalk workers
MIT License
311 stars 59 forks source link

Multiple Worker Server Environment #16

Closed mubbashar closed 11 months ago

mubbashar commented 7 years ago

Hi, Thanks for this awesome package. Can you tell me how this handle scheduler command when there are multiple web server are running. Is it only run on one worker server Or the the scheduler will run on every worker server and this will cause duplicate message in the queue ?

thanks!

dusterio commented 7 years ago

@mubbashar every command will only be executed once - by one of the worker instances, the one that pops the job from the queue first (even though technically with AWS the queue comes to you as a POST request). There will be no duplicates

mubbashar commented 7 years ago

thanks for clarification. i was afraid if worker run schedule command then there will be duplicates because every worker will have their own cron.yaml file and every one call /worker/schedule every minute

reganjohnson commented 7 years ago

@dusterio We are considering this package. Could you clarify your point regarding the SCHEDULER and multiple workers?

We have multiple jobs setup that get executed at the same time (ie ->hourly()). Does the scheduler parse the commands that need to be run and push them to the queue as individual items or does the worker receiving the request process all jobs that evaluate true for the given cron statement (which could be many jobs).

dusterio commented 7 years ago

@reganjohnson one scheduled command (eg. hourly()) will only be passed and processed by one worker - whichever worker pulls from the queue first

poisa commented 6 years ago

For future reference: cron.yaml is only created in the leader instance in you aws workers. So if you have 10 instances, only ONE will run the crons. Technically one instance will create an SQS message which goes into the queue and is processed by any other instance that picks it up. If your queue is super busy, then this cron message might not be processed in time.

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks

arianacosta commented 4 years ago

From that @poisa 's link regarding jobs defined in cron.yml:

Elastic Beanstalk uses leader election to determine which instance in your worker environment queues the periodic task. Each instance attempts to become leader by writing to an Amazon DynamoDB table. The first instance that succeeds is the leader, and must continue to write to the table to maintain leader status. If the leader goes out of service, another instance quickly takes its place.

saleoid commented 2 years ago

How I can process jobs in multiple queues with this. Any suggestions would be great?