dailybruin / kerckhoff

Flatpage/static site manager for the Daily Bruin (proxy for Google Drive)
https://kerckhoff.dailybruin.com
4 stars 1 forks source link

Add celery or other task scheduler to reduce peak cpu usage #87

Closed RandomY-2 closed 1 year ago

RandomY-2 commented 1 year ago

Currently, if a lot of packaging requests are sent to Kerkhoff, the peak CPU usage exceeds Kubernetes resource limit as in last night, which causes worker timeout:

Screen Shot 2022-11-29 at 1 15 15 PM

We can add a task scheduler so large packaging such as for prime can be done asynchronously and can improve our availability.

Also #82 is needed. Currently, if a fetch fails, we cannot re-fetch it easily.