MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

Consider migrating BG Tasks to Celery #184

Closed ghukill closed 5 years ago

ghukill commented 6 years ago

Not sure how, or if, this is possible with Django Background Tasks. Perhaps restarting the supervisor listening process is a possibility? or would that trigger retries?

ghukill commented 6 years ago

This has proved to be pretty important functionality.

Unfortunately, it might suggest looking at a more robust bg task framework like Celery.

ghukill commented 5 years ago

Scaffolding begins in celery branch, looking hopeful it will nicely address scaling, extensibility, and blocking nature of django bg tasks.

ghukill commented 5 years ago

Consider callback for closing task, updating timestamp, etc.?

ghukill commented 5 years ago

Done. Fairly major change, but much of the work was swapping out Django background tasks for celery, while scaffolding for CombineBackgroundTask was already in place. Likely issues and bugs will arise, but will open seperately. Closing here.