Celery

[rnovec]

Django + Celery

Getting Started

This PR is a configuration for Celery & Django to run crontab or periodic jobs/tasks

Prerequisites 📋

• RabbitMQ broker
• Docker
• Docker Compose

Installing

pip install -r requirements.txt

start single RabbitMQ service with Docker or use CloudAMQP

docker run -d --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management 

Change .env.example to .env

set broker url as enviroment variable: CLOUDAMQP_URL=amqp://guest:guest@127.0.0.1:5672/

Finally, run celery beat and worker

celery worker -A codeandomexico -B --loglevel=INFO

... Or run with Docker Compose

Additionally, we automate our project setup with Docker Compose

Open terminal and run:

docker-compose up

this will build project and start:

• Django project with Guinicorn
• RabbitMQ service
• Celery broker & beat

Tasks

All bussines logic is in api_covid19/tasks.py: For more info visit: Celery periodic tasks / Crontab schedules

from celery.decorators import task
from celery.task.schedules import crontab
from celery.decorators import periodic_task

from .models import *
from .scripts.fetch_data import run

@periodic_task(run_every=(crontab(hour=2)), name="update_info: Scrap https://www.gob.mx/ website every 2 hrs")
def update_info():
    """
    This is a real world task
    """
    print("Updating data...")
    run()

@periodic_task(run_every=(crontab()), name="example_task: This is example task running every minute")
def example_task():
    """
    This is a example task
    """
    # do some stuff
    # use models
    # make requests
    print("Running every minute...")
    run() # just for test. May be deleted later

Results