search

tikv / raft-engine

A persistent storage engine for Multi-Raft log
Apache License 2.0
565 stars 88 forks source link

Raft Group tombstone is rewritten in the wrong order #142

Closed tabokie closed 2 years ago

tabokie commented 3 years ago

Bug Report

Description

When a Raft Group is deleted (via Clean command), a tombstone is written to log file, and added to MemTableAccessor. During rewrite, log file holding that tombstone is purged, but tombstone in memtable still can be retrieved and rewritten to Rewrite queue.

However, consider a Append file containing these operations [append1 (..), clean, append2 (..)]. During rewrite, the tombstone is actually rewritten in the wrong order [(..), append2 (..), clean]. This will cause the newly recreated Raft Group to disappear after restart.

TiKV guarantees to not recreate a region with the same ID, so this bug is of low severity.

tabokie commented 2 years ago

It turns out TiKV can recreate region with the same region_id after reschedule.

Triggering condition:

  1. Region reschedule (scheduled out and back again)
  2. No recent (~1m after the reschedule) writes to the region
  3. TiKV restarts

Workaround: Remove rewrite tombstones and repair rewrite queue.

I'm fixing this in #180