Integrate Celery support for laborious, asynchronous tasks.

TorSpider / TorSpider-Backend

The database backend with which the spiders share their discoveries.

BSD 3-Clause "New" or "Revised" License

1 stars 0 forks source link

Integrate Celery support for laborious, asynchronous tasks. #21

Closed haxys closed 6 years ago

haxys commented 6 years ago

In order to lighten the burden of the backend, let's offload some of the more processing-intensive and repetitive tasks to Celery:

1. Populating the url_queue table.
2. Rebuilding the top-20 lists so they're not constantly being refreshed live.
3. Rebuilding the various page indexes and search tables.
4. Eventually, building a daily snapshot of the node links for the force-directed graph.
5. Processing newly-submitted .onions from the public.

We'll use RabbitMQ as the broker.

haxys commented 6 years ago

We could also use Celery to generate a database dump of interesting data for people to download while search functionality is not yet available.