tonbo-io / tonbo

A portable embedded database using Arrow.
https://tonbo.io
Apache License 2.0
764 stars 42 forks source link

level slice are not ordered #152

Closed crwen closed 1 month ago

crwen commented 1 month ago

Bug Report

Both minor and major compaction are applied to level slice in apply_edits, and new sstables are append to level slice. So we can not ensure the ordered of level slice.

But in the scope search relative methods, they are assume the slice is order.

Here is a simple test code

test code ```rust #[tokio::test] async fn test_flush_major_level_sort() { let temp_dir = TempDir::new().unwrap(); let mut option = DbOption::from(temp_dir.path()); option.immutable_chunk_num = 1; option.immutable_chunk_max_num = 0; option.major_threshold_with_sst_size = 2; option.level_sst_magnification = 1; option.max_sst_file_size = 2 * 1024 * 1024; option.major_default_oldest_table_num = 1; option.trigger_type = TriggerType::Length(5); let db: DB = DB::new(option, TokioExecutor::new()).await.unwrap(); for i in 5..9 { let item = Test { vstring: i.to_string(), vu32: i, vbool: Some(true), }; db.insert(item).await.unwrap(); } db.flush().await.unwrap(); for i in 0..4 { let item = Test { vstring: i.to_string(), vu32: i, vbool: Some(true), }; db.insert(item).await.unwrap(); } db.flush().await.unwrap(); db.insert(Test { vstring: "6".to_owned(), vu32: 22, vbool: Some(false), }) .await .unwrap(); db.insert(Test { vstring: "8".to_owned(), vu32: 77, vbool: Some(false), }) .await .unwrap(); let version = db.version_set.current().await; { dbg!(version); } db.flush().await.unwrap(); let version = db.version_set.current().await; { dbg!(version); } db.insert(Test { vstring: "1".to_owned(), vu32: 22, vbool: Some(false), }) .await .unwrap(); db.insert(Test { vstring: "5".to_owned(), vu32: 77, vbool: Some(false), }) .await .unwrap(); db.flush().await.unwrap(); let version = db.version_set.current().await; dbg!(version); db.insert(Test { vstring: "2".to_owned(), vu32: 22, vbool: Some(false), }) .await .unwrap(); db.insert(Test { vstring: "7".to_owned(), vu32: 77, vbool: Some(false), }) .await .unwrap(); db.flush().await.unwrap(); let version = db.version_set.current().await; dbg!(version); } ```

Here is the status of level slice before and after last compaction

log before compaction ``` Version { level_slice: [ [ Scope { min: "6", max: "8", }, Scope { min: "1", max: "5", }, ], [ Scope { min: "5", max: "8", }, Scope { min: "0", max: "3", }, ], [], [], [], [], [], ], } ``` after compaction ``` Version { level_slice: [ [ Scope { min: "6", max: "8", }, Scope { min: "2", max: "7", }, ], [ Scope { min: "5", max: "8", }, Scope { min: "0", max: "5", }, ], [ Scope { min: "0", max: "3", }, ], [], [], [], [], ], } ``` `[5-8]` should also involved into compaction, but it does not