harvard-lil / perma

Indelible links
408 stars 72 forks source link

Use a dedicated queue for warc -> wacz conversion. #3496

Closed rebeccacremona closed 3 months ago

rebeccacremona commented 3 months ago

We are experimenting with converting Perma's WARCs to WACZ using a Celery task:

This PR adds config so that these celery tasks will be routed to a dedicated queue, for maximum flexibility: we can use a dedicated worker minion with js-wacz installed on it if we want, or not, we can configure concurrency exactly as desired, we don't have to worry about these jobs intermingling with capture jobs (unless we want them to), etc.