analyseether / ether_sql

A python library to push ethereum blockchain data into an sql database.
http://ether-sql.readthedocs.io
Apache License 2.0
69 stars 22 forks source link

Use redis and celery to create a task queue #22

Closed ankitchiplunkar closed 6 years ago

ankitchiplunkar commented 6 years ago

Is your feature request related to a problem? Please describe. Currently, the application uses a loop to push data from the ethereum node to the sql. This is cannot be parallelized, and I would like to integrate some UI so basically see which blocks are missing from the sql.

Describe the solution you'd like I would like to push interesting block_numbers in a Redis instance. Then use Celery to pick up the block_numbers from the redis instance and push them into sql. Interesting links:

  1. https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb
  2. https://hackernoon.com/asynchronous-tasks-with-celery-redis-in-django-3e00d3735686
  3. http://docs.celeryproject.org/en/2.0-archived/tutorials/otherqueues.html

Describe alternatives you've considered Was thinking of also possibly using RabbitMq but have read in documentation that redis might be faster.

ankitchiplunkar commented 6 years ago

More resources

  1. https://denibertovic.com/posts/celery-best-practices/
  2. https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/