search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

Clean termination when AddSSTable has a deadline #79443

Open gh-casper opened 2 years ago

gh-casper commented 2 years ago

We require a deadline on AddSSTable to prevent delayed replays after a Revert/ClearRange request has been executed. This would result in unexpected data ending up in the keyspace.

The server would refuse to process any AddSST requests past the deadline.

OnFailOrCancel() and revert logic would wait until the deadline has passed before executing their cleanup. To this effect, we would require the resume to be aware of the deadline. This information will probably be persisted in the job by the oracle processor that determines this deadline. Ideally, the job could have been this oracle but we do not have a way of pushing a row into the flow once it has been started.

The oracle processor mentioned above would be the root processor of the flow. It would be responsible for “heartbeating” the downstream processors by sending a new deadline every x time intervals via a mirror router.

The downstream stream ingestion processor would pull rows from the oracle checking for a deadline update. If the processor receives a row, it will update the deadline on the batcher being used to send AddSSTable requests.

Related to: https://github.com/cockroachdb/cockroach/issues/64790 Epic https://cockroachlabs.atlassian.net/browse/CRDB-10341

Jira issue: CRDB-14868

blathers-crl[bot] commented 2 years ago

cc @cockroachdb/cdc

blathers-crl[bot] commented 2 years ago

cc @cockroachdb/bulk-io

mari-crl commented 2 years ago

:information_source: Hello! I am a human and not at all a robot! Look at my very human username! :robot: :notes: :thinking: Although I tried very hard to figure out what to do with this issue, more powerful human brains will need to help me. (specifically: Both Github and Jira issues were modified after exalate problems) :confounded: :arrows_counterclockwise: Please visit this issue's mirror at CRDB-14868 and try to sync the two sides up manually. :star2: :white_check_mark: When you're finished, comment saying as much asn a member of Developer Infrastructure will be along to finish linking. :link: :no_entry_sign: Note that until this is done, this issue is not and will not be synced to Jira with Exalate. :no_entry_sign: :sweat_smile: Feeling lost? Don't worry about it! A member of @cockroachdb/exalate-22-cleanup-team will be along shortly to help! :+1: :construction_worker: Developer Infrastructure members: when ready, open Exalate from the right-hand menu of the mirror issue in Jira, then choose Connect and enter this issue's URN: cockroachdb/cockroach-79443. Either way, delete this comment when you're done. :key: :pray: Thank you for your compliance, my fellow humans! :robot: :wave:

mwang1026 commented 2 years ago

sync done