Use redis and celery to create a task queue

[ankitchiplunkar]

Is your feature request related to a problem? Please describe. Currently, the application uses a loop to push data from the ethereum node to the sql. This is cannot be parallelized, and I would like to integrate some UI so basically see which blocks are missing from the sql.

Describe the solution you'd like I would like to push interesting block_numbers in a Redis instance. Then use Celery to pick up the block_numbers from the redis instance and push them into sql. Interesting links:

1. https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb
2. https://hackernoon.com/asynchronous-tasks-with-celery-redis-in-django-3e00d3735686
3. http://docs.celeryproject.org/en/2.0-archived/tutorials/otherqueues.html

Describe alternatives you've considered Was thinking of also possibly using RabbitMq but have read in documentation that redis might be faster.

[ankitchiplunkar]

More resources

1. https://denibertovic.com/posts/celery-best-practices/
2. https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/