search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.59k stars 584 forks source link

Read replicas may see no data after truncating and then restarting #11936

Open andrwng opened 1 year ago

andrwng commented 1 year ago

Version & Environment

Redpanda version: (use rpk version): dev

What went wrong?

The current implementation for read replicas' max collectible offset sets it to the max value. We are currently relying on retention overrides to ensure that a read replica's Raft log doesn't get truncated. The current policy is that we don't truncate local data for 1 hour, which may not work if the sync interval is greater than an hour.

What should have happened instead?

Max collectible offset should be set to the latest replace_manifest batch.

How to reproduce the issue?

  1. Set the read replica sync interval to higher than 1hr
  2. Wait for local truncation to truncate the local log of the read replica
  3. Restart the read replica

JIRA Link: CORE-1368

jcsp commented 1 year ago

Max collectible offset should be set to the latest replace_manifest batch.

Yep, that sounds like the right change.

I guess this is kind of a specialization of https://github.com/redpanda-data/redpanda/issues/7601 -- that's the general case that if we have no manifest we shouldn't serve reads, this is the particular case where we can avoid getting into the no-manifest case by doing more prudent log trimming.

github-actions[bot] commented 8 months ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

github-actions[bot] commented 8 months ago

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.