search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.65k stars 1.04k forks source link

[DocDB][Tablet splitting] Performance drop with large tablet splitting #19969

Open Arjun-yb opened 7 months ago

Arjun-yb commented 7 months ago

Jira Link: DB-8945

Description

Observed throughput drop and write rejections with large tablet splitting

Steps: Backup was taken in 2.16(backup size 1.4 TB) Created Universe in master(2.21) without automatic tablet splitting(automatic tablet splitting is OFF) Then started a workload(CassandraBatchTimeseries) After some time enable automatic tablet splitting After a day noticed throughput drop and write rejections

More details (slack discussions) are added into respective JIRA ticket

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

rthallamko3 commented 3 months ago

@Arjun-yb , If you keep tablet splitting off, would you be able to run CassandraBatchTimeseries for over a day on that hardware? If so, then we have to take a closer look at this. cc @arybochkin

rthallamko3 commented 2 months ago

@Arjun-yb , Let me know if you got a chance to look at this. Wondering if the priority of the issue can be lowered.

Arjun-yb commented 1 month ago

@rthallamko3 I re-tried the scenario with and without tablet splitting and repro-ed it with tablet splitting and didn't see issue without tablet splitting(workload ran fine), added details in the JIRA