Support for periodic task

[CatchZeng]

How can I implement a periodic task like python celery?

'taskA_schedule' : {
    'task':'tasks.taskA',
    'schedule':20,
    'args':(5,6)
}

I found a delayed task, but I did n’t find a periodic task.

[kingzbauer]

@CatchZeng you mean a periodic task as such https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#entries? I am not sure machinery offers that out of the box. I found a work around using https://pkg.go.dev/github.com/robfig/cron?tab=doc which would work like celery beat to trigger a task to a machinery worker at specific intervals

You could probably create a wrapper package that takes the above syntax , probably from a config file and dispatches the specified task, when triggered by cron, to a machinery worker.

I could help with that. On my todo list.

[CatchZeng]

@kingzbauer thank you very much! Yes, what I want is a periodic task. I also thought about doing it with cron, but if multiple workers handling periodic tasks are deployed in a distributed system, how is the machinery handled? So, I raised this issue and wanted to know if machinery has a better way to deal with periodic tasks.

[kingzbauer]

As long as you have a single cron process deployed, that takes care of scheduling the periodic tasks with machinery service struct, something like asyncResult, err := server.SendTask(signature), having multiple workers shouldn't be a problem since they all feed off from the same Queue, so it's sort of a "round-robin" distribution of tasks.

The workers know nothing of any periodic tasks, they only care for what they are receiving from the job queue. Only make sure that only one cron process is running that takes care of the scheduling. I believe this is how celery beat does it.

You can run a simple simulation locally, where you start multiple workers and a simple beat like process sending tasks to the job queue in short bursts of time.

[kingzbauer]

@CatchZeng here is a link to a simple demo app I put up using both libraries https://github.com/kingzbauer/periodic-scheduler-demo-go You can log the output for the workers. No task is replicated across the workers.

[HitmanRanbo]

see my pr: https://github.com/RichardKnop/machinery/pull/534

[charleswhchan]

Also consider https://github.com/go-co-op/gocron (formerly https://github.com/jasonlvhit/gocron) for scheduling periodic tasks.