search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.86k stars 3.77k forks source link

jobs: Add exponential backoff to scheduled jobs. #73255

Open miretskiy opened 2 years ago

miretskiy commented 2 years ago

jobs added exponential backoff to "regular" jobs. HOwever, schedules do not yet have this functionality. This is bad because 1 misbehaving schedule (e.g. schedule that OOMs the node) may fail while the scheduled job is in the planning state -- before the actual job is created. This will result in "schedule of death".

Jira issue: CRDB-11498

shermanCRL commented 2 years ago

Putting this back in triage, just as a reminder -- do we this this will help with observed issues?

amruss commented 2 years ago

Backlogging this for now as a nice to have since we have limited scheduled jobs atm, unless someone wants to pick it up. We will revisit for 22.2 release cycle.

amruss commented 2 years ago

Additional note: we may want to add an env variable as an escape hatch

github-actions[bot] commented 1 year ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!