Use Celery to implement tasks to run collectors in a distributed fashion - Githubissues

request-yo-racks / api

A REST API for the Request-Yo-Racks projects.

https://api.requestyoracks.org

MIT License

0 stars 4 forks source link

Use Celery to implement tasks to run collectors in a distributed fashion #26

Closed rgreinho closed 6 years ago

rgreinho commented 6 years ago

Issue Type

Feature request

Expected Behavior

Query all the collectors in parallel
Merge the results at the end of the process (use Celery Canvas)

Current Behavior

The collectors are queried sequentially.

rgreinho commented 6 years ago

GitMate.io thinks possibly related issues are https://github.com/request-yo-racks/api/issues/5 (Implement a Facebook collector), https://github.com/request-yo-racks/api/issues/6 (Implement a Full contact collector), https://github.com/request-yo-racks/api/issues/22 (Implement a proper interface for the collectors), https://github.com/request-yo-racks/api/issues/24 (Use the Yelp collector in addition to the Google one), and https://github.com/request-yo-racks/api/issues/23 (Research spike: Services we could use as collectors).