SunguckLee
commented
6 years ago I have found the reason why RangeDeleter of MongoDB(with RocksDB) is so slow.
MongoDB use "indexScan" instance for RangeDeleter(of MongoDB not RocksDB) processing. MongoDB RangeDeleter is not re-use indexScan for all documents of chunk, but re-create indexScan object for each document of chunk. This is also documented in MongoDB JIRA(https://jira.mongodb.org/browse/SERVER-33237).
But in this code(https://github.com/mongodb/mongo/blob/r3.4.14/src/mongo/db/dbhelpers.cpp#L366), worse thing is the first exec->getNext() call is really slow with RocksDB storage engine. Not for Wiredtiger storage engine. I have profiled the elapsed time of the very first exec->getNext() call on busy server.
>> The first exec->getNext() call
- Wiredtiger : 50~100 us (micro second)
- RocksDB : 12000 ~ 600000 us
>> Following exec->getNext() call is almost same speed for both storage engine.Because of current implementation of RangeDeleter, getNext() func is called only once for each indexScan object. Meaning RangeDeleter use very frist(very slow) exec->getNext(). So RangeDeleter with RocksDB is so slow but not slow for Wiredtiger storage engine.
Why first call of indexScan->getNext() is so slow with RocksDB ?
Regards,
igorcanadi
q96456
eharry
sunlike-Lipo
After chunk migration, MongoDB call RangeDeleter(Not RocksDB, it's mongodb background thread name) to remove obsolete documents moved to another shards.
With MongoRocks, RangeDeleter is too slow. Sometimes it takes over 2 hours to remove only 1 chunk's documents. About 20000 ~ 80000 documents in a chunk. Some shards, RangeDeleter of 1 chunk takes 7~8 hour.
So now, chunk balancing is blocked because stacked task in RangeDeleter queue is full.
And I also cleanup these obsolete documents(after moved to another shard) with "cleanupOrphaned" command. It also takes a lot of time.
There's only 189984 documents in this collection. And removed only 38328 documents with "cleanupOrphaned" command, But it took 14 minutes. What is wrong with my MongoRocks setup ?
[DBOptions] manual_wal_flush=false allow_ingest_behind=false avoid_flush_during_shutdown=false avoid_flush_during_recovery=false info_log_level=INFO_LEVEL access_hint_on_compaction_start=NORMAL write_thread_max_yield_usec=100 write_thread_slow_yield_usec=3 wal_recovery_mode=kPointInTimeRecovery allow_concurrent_memtable_write=true enable_pipelined_write=false fail_if_options_file_error=false stats_dump_period_sec=600 wal_bytes_per_sync=0 max_total_wal_size=0 delayed_write_rate=1073741824 two_write_queues=false bytes_per_sync=1048576 writable_file_max_buffer_size=1048576 wal_dir=/data/mongodb/db/journal log_file_time_to_roll=0 keep_log_file_num=1000 WAL_ttl_seconds=0 db_write_buffer_size=0 table_cache_numshardbits=6 max_open_files=-1 max_file_opening_threads=16 WAL_size_limit_MB=0 max_background_flushes=2 db_log_dir= max_background_compactions=10 max_subcompactions=1 max_background_jobs=12 random_access_max_buffer_size=1048576 delete_obsolete_files_period_micros=21600000000 skip_stats_update_on_db_open=false skip_log_error_on_recovery=false dump_malloc_stats=false paranoid_checks=true is_fd_close_on_exec=true max_manifest_file_size=18446744073709551615 error_if_exists=false use_adaptive_mutex=false enable_thread_tracking=true create_missing_column_families=false create_if_missing=true manifest_preallocation_size=4194304 base_background_compactions=-1 use_fsync=false allow_2pc=false recycle_log_file_num=0 use_direct_io_for_flush_and_compaction=false compaction_readahead_size=0 use_direct_reads=false allow_mmap_writes=false preserve_deletes=false enable_write_thread_adaptive_yield=true max_log_file_size=0 allow_fallocate=true allow_mmap_reads=false new_table_reader_for_compaction_inputs=false advise_random_on_open=true
[CFOptions "default"] compaction_style=kCompactionStyleLevel compaction_filter_factory=PrefixDeletingCompactionFilterFactory memtable_factory=SkipListFactory memtable_insert_with_hint_prefix_extractor=nullptr merge_operator=nullptr prefix_extractor=nullptr comparator=leveldb.BytewiseComparator bottommost_compression=kDisableCompressionOption compression=kSnappyCompression max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1 max_sequential_skip_in_iterations=8 memtable_prefix_bloom_size_ratio=0.000000 bloom_locality=0 memtable_huge_page_size=0 inplace_update_num_locks=10000 arena_block_size=16777216 target_file_size_multiplier=1 num_levels=7 min_write_buffer_number_to_merge=1 max_write_buffer_number_to_maintain=0 write_buffer_size=134217728 level0_stop_writes_trigger=36 level0_slowdown_writes_trigger=10 level0_file_num_compaction_trigger=3 max_bytes_for_level_multiplier=10.000000 hard_pending_compaction_bytes_limit=274877906944 max_compaction_bytes=3355443200 force_consistency_checks=false optimize_filters_for_hits=true compaction_options_fifo={allow_compaction=false;max_table_files_size=1073741824;ttl=0;} max_bytes_for_level_base=536870912 level_compaction_dynamic_level_bytes=true target_file_size_base=134217728 max_write_buffer_number=4 soft_pending_compaction_bytes_limit=68719476736 inplace_update_support=false compression_per_level=kNoCompression:kNoCompression:kLZ4Compression paranoid_file_checks=false table_factory=BlockBasedTable disable_auto_compactions=false compaction_options_universal={allow_trivial_move=false;size_ratio=1;min_merge_width=2;max_size_amplification_percent=200;max_merge_width=4294967295;compression_size_percent=-1;stop_style=kCompactionStopStyleTotalSize;} compaction_filter=nullptr compaction_pri=kByCompensatedSize max_successive_merges=0 report_bg_io_stats=false
[TableOptions/BlockBasedTable "default"] read_amp_bytes_per_bit=8589934592 format_version=2 whole_key_filtering=true filter_policy=rocksdb.BuiltinBloomFilter verify_compression=false block_size_deviation=10 block_size=16384 partition_filters=false checksum=kCRC32c hash_index_allow_collision=true index_block_restart_interval=1 block_restart_interval=16 no_block_cache=false pin_l0_filter_and_index_blocks_in_cache=false cache_index_and_filter_blocks_with_high_priority=false metadata_block_size=4096 cache_index_and_filter_blocks=true index_type=kBinarySearch flush_block_policy_factory=FlushBlockBySizePolicyFactory