[DocDB] Remote bootstrapping tablet should notify master when post-split compactions are done

[SrivastavaAnubhav]

Jira Link: DB-1860

Description

A remote bootstrapping tablet does not send data about whether a post-split compaction has been run to the master. This can prevent new splits in the tablet split manager, since it assumes the tablet split is still ongoing.

Test to reproduce this behavior (add a log in the tablet split manager when we check if all replicas are compacted, and see that we are waiting on the restarted tserver to set may_have_orphaned_post_split_data to false).

TEST_F_EX(
    TabletSplitExternalMiniClusterITest, OngoingRemoteBootstrapReportsCompactionFinished,
    TabletSplitRbsTest) {
  // If a new tablet is created and split with one node completely uninvolved, then when that node
  // rejoins it will have to do a remote bootstrap.

  auto server_to_restart = cluster_->tablet_server(0);
  ASSERT_OK(cluster_->SetFlag(server_to_restart, "rocksdb_disable_compactions", "true"));
  ASSERT_OK(cluster_->SetFlagOnMasters("enable_automatic_tablet_splitting", "true"));
  ASSERT_OK(cluster_->SetFlagOnMasters("tablet_split_low_phase_shard_count_per_node", "1000"));
  ASSERT_OK(cluster_->SetFlagOnMasters("tablet_split_low_phase_size_threshold_bytes", "0"));

  constexpr int kNumRowsPerBatch = 2000;
  int start_key = 1;
  const auto kWaitForTabletsRunningTimeout = 20s * kTimeMultiplier;

  auto ts_map = ASSERT_RESULT(itest::CreateTabletServerMap(
      cluster_->GetLeaderMasterProxy<master::MasterClusterProxy>(), &cluster_->proxy_cache()));

  CreateSingleTablet();
  const auto source_tablet_id = CHECK_RESULT(GetOnlyTestTabletId());
  LOG(INFO) << "Source tablet ID: " << source_tablet_id;

  ASSERT_OK(WriteRows(kNumRowsPerBatch, start_key));
  start_key += kNumRowsPerBatch;
  for (size_t i = 0; i < cluster_->num_tablet_servers(); i++) {
    auto* ts = cluster_->tablet_server(i);
    ASSERT_OK(WaitForAllIntentsApplied(ts_map[ts->uuid()].get(), 15s * kTimeMultiplier));
    ASSERT_OK(cluster_->FlushTabletsOnSingleTServer(ts, {source_tablet_id}, false));
  }
  for (size_t i = 0; i < cluster_->num_tablet_servers(); i++) {
    ASSERT_OK(cluster_->WaitForTabletsRunning(cluster_->tablet_server(i), kWaitForTabletsRunningTimeout));
    ASSERT_OK(WaitForTabletsExcept(2, i, source_tablet_id));
  }
  auto tablet_to_flush = ASSERT_RESULT(cluster_->GetTablets(cluster_->tablet_server(0))).front().tablet_id();

  server_to_restart->Shutdown();
  ASSERT_OK(cluster_->WaitForTSToCrash(server_to_restart));

  for (int i = 0; i < 100; ++i) {
    LOG(INFO) << "Writing rows from start key " << start_key;
    ASSERT_OK(WriteRows(kNumRowsPerBatch, start_key));
    start_key += kNumRowsPerBatch;
  }

  ASSERT_OK(server_to_restart->Restart(
      ExternalMiniClusterOptions::kDefaultStartCqlProxy,
      {std::make_pair("TEST_tablet_bootstrap_delay_ms", "100000")}));

  ASSERT_OK(cluster_->FlushTabletsOnSingleTServer(server_to_restart, {tablet_to_flush}, /* is_compaction = */ true));

  SleepFor(30s);
}

[rthallamko3]

Good starter task.

[luoteresa]

Hi,

My name is Teresa Luo and I'm a student at UT Austin taking a graduate distributed systems course. I'd like to work on this issue with @Akl35 as a part of a group project.

Will be sure to use Slack as correspondence if any questions arise.