ever0de
commented
3 weeks ago Most of the counts arise from skipping entries where the InternalKey.Trailer type is set to DEL.
Since the data represents an orderbook where each order has a globally unique key, it seems this is causing the observed behavior.
I suspect that the elision-only compaction may not have been functioning properly.
Could this issue be related to the ReadSamplingMultiplier option? When it was set to -1, the situation gradually resolved over time.
Based on the current observations, In my opinion
- Read-triggered compaction frequently occurs for prefixes where data is not often written, though occasional SET/DELETE operations happen.
- Elision-only compaction does not occur as frequently, allowing tombstones to persist down to L6.
- As a result, the Iterator encounters many tombstones, especially in L6. However, despite frequent encounters in
(*Iterator).sampleRead(), read-triggered compaction does not occur, possibly because thenumOverlappingLevelsdoes not reach 2. https://github.com/cockroachdb/pebble/blob/9f3904a705d60b9832febb6c6494183d92c8f556/iterator.go#L819-L880
jbowens
Environment
Version: v1.1.2 Total Data Size: 5GB Resources:
memory: 10GiB
Options
[Version] pebble_version=0.1 [Options] bytes_per_sync=524288 cache_size=1073741824 cleaner=delete compaction_debt_concurrency=1073741824 comparer=leveldb.BytewiseComparator disable_wal=false flush_delay_delete_range=10s flush_delay_range_key=0s flush_split_bytes=2097152 format_major_version=16 l0_compaction_concurrency=10 l0_compaction_file_threshold=500 l0_compaction_threshold=2 l0_stop_writes_threshold=1000 lbase_max_bytes=67108864 max_concurrent_compactions=3 max_manifest_file_size=134217728 max_open_files=16384 mem_table_size=67108864 mem_table_stop_writes_threshold=4 min_deletion_rate=134217728 merger=pebble.concatenate read_compaction_rate=16000 read_sampling_multiplier=16 strict_wal_tail=true table_cache_shards=12 table_property_collectors=[] validate_on_ingest=false wal_dir= wal_bytes_per_sync=0 max_writer_concurrency=0 force_writer_parallelism=false secondary_cache_size_bytes=0 create_on_shared=0 [Level "0"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=rocksdb.BuiltinBloomFilter filter_type=table index_block_size=262144 target_file_size=2097152 [Level "1"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=rocksdb.BuiltinBloomFilter filter_type=table index_block_size=262144 target_file_size=4194304 [Level "2"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=rocksdb.BuiltinBloomFilter filter_type=table index_block_size=262144 target_file_size=8388608 [Level "3"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=rocksdb.BuiltinBloomFilter filter_type=table index_block_size=262144 target_file_size=16777216 [Level "4"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=rocksdb.BuiltinBloomFilter filter_type=table index_block_size=262144 target_file_size=33554432 [Level "5"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=rocksdb.BuiltinBloomFilter filter_type=table index_block_size=262144 target_file_size=67108864 [Level "6"] block_restart_interval=16 block_size=32768 block_size_threshold=90 compression=Snappy filter_policy=none filter_type=table index_block_size=262144 target_file_size=134217728Write Pattern: A batch of data sized between 600KB to 800KB is generated approximately every 0.6 seconds.
Batch.{Set/Delete}Currently, the total number of keys in the database is around 36 million. We are experiencing high latency when using an iterator to search for 1,120 specific records, with a total latency of approximately 22.37ms and a seek time of 1.25ms.
We have observed that the
PointCountin the Iterator Stats occasionally spikes to very high levels, correlating with the increased latency. Despite the LSM tree metrics not indicating an inverted LSM state, there seems to be frequent occurrences of(*mergingIter).findNextEntry().pebble metrics:
result:
iter stats:
Could you provide advice on optimizing options to improve overall iterator performance or insights into any potential misconfigurations in our current settings?
Jira issue: PEBBLE-250