search

q-m / scrapyd-k8s

Scrapyd on container infrastructure
MIT License
12 stars 8 forks source link

Limit maximum number of jobs #10

Open wvengen opened 10 months ago

wvengen commented 10 months ago

Scrapyd has configuration options jobs_to_keep and finished_to_keep.

It would be good to have both. Containers (Docker) / jobs (Kubernetes) can be removed when it is time according to these options.

For Kubernetes, one could alternatively use a max lifetime (which is possible to set on jobs). But let's start with the two above.

JeremyVriens commented 3 weeks ago

This issue is blocked by https://github.com/q-m/scrapyd-k8s/issues/11 as we need to sort by end_time before cutting off the amount of finished jobs with finished_to_keep (you want to keep the x most recent ones)