inspirehep / hepcrawl

Scrapy project for feeds into INSPIRE-HEP
http://inspirehep.net
Other
17 stars 30 forks source link

setup: bump celery to version ~4.0 #241

Closed jacquerie closed 6 years ago

jacquerie commented 6 years ago

Description:

It should be noted that hepcrawl has been tested against version 4 of Celery since that version was released due to how the constraint is expressed, so nothing is expected to break here.

Checklist:

david-caro commented 6 years ago

Though I don't understand why hepcrawl needs celery in the first place. The whole hepcrawl/inspire-crawler/workflow interaction is very confusing.

So it needs celery to be able to trigger a task, the flow is: hepcrawl -> celery task -> inspire-crawler -> workflow

inspire-crawler is the part on inspire-next that receives the crawl results and generates workflows from them. Celery is just the protocol used between hepcrawl and inspire-crawler.