search

rust-rocksdb / rust-rocksdb

rust wrapper for rocksdb
Apache License 2.0
1.83k stars 721 forks source link

Options, ReadOptions, WriteOptions missing many settings #260

Open iSynaptic opened 5 years ago

iSynaptic commented 5 years ago

The Options, ReadOptions, and WriteOptions structures do not exposes setters for many of the underlying settings. The underlying options object exposes over 150 settings, many of which are necessary for tuning for different scenarios. Also, many of the settings apply to a ColumnFamily and some to the DB. Some work should be done to expose more of these settings and place them in an appropriately scoped *Options struct.

Available FFI functions for reference:

Header Files for reference:

Example LOG output showing many of the options (if we upgrade to a newer RocksDB library, there may be even more options):

RocksDB version: 5.14.2
Write Ahead Log file in /home/iSynaptic/tmp:
                        Options.error_if_exists: 0
                      Options.create_if_missing: 1
                        Options.paranoid_checks: 1
                                    Options.env: 0x55c871ce9800
                               Options.info_log: 0x55c87249f040
               Options.max_file_opening_threads: 16
                             Options.statistics: (nil)
                              Options.use_fsync: 0
                      Options.max_log_file_size: 0
                 Options.max_manifest_file_size: 18446744073709551615
                  Options.log_file_time_to_roll: 0
                      Options.keep_log_file_num: 1000
                   Options.recycle_log_file_num: 0
                        Options.allow_fallocate: 1
                       Options.allow_mmap_reads: 0
                      Options.allow_mmap_writes: 0
                       Options.use_direct_reads: 0
                       Options.use_direct_io_for_flush_and_compaction: 0
         Options.create_missing_column_families: 0
                             Options.db_log_dir:
                                Options.wal_dir: /home/iSynaptic/tmp
               Options.table_cache_numshardbits: 6
                     Options.max_subcompactions: 1
                 Options.max_background_flushes: 4
                        Options.WAL_ttl_seconds: 0
                      Options.WAL_size_limit_MB: 0
            Options.manifest_preallocation_size: 4194304
                    Options.is_fd_close_on_exec: 1
                  Options.advise_random_on_open: 1
                   Options.db_write_buffer_size: 0
                   Options.write_buffer_manager: 0x55c87249f2e0
        Options.access_hint_on_compaction_start: 1
 Options.new_table_reader_for_compaction_inputs: 1
          Options.random_access_max_buffer_size: 1048576
                     Options.use_adaptive_mutex: 0
                           Options.rate_limiter: (nil)
    Options.sst_file_manager.rate_bytes_per_sec: 0
                      Options.wal_recovery_mode: 2
                 Options.enable_thread_tracking: 0
                 Options.enable_pipelined_write: 0
        Options.allow_concurrent_memtable_write: 1
     Options.enable_write_thread_adaptive_yield: 1
            Options.write_thread_max_yield_usec: 100
           Options.write_thread_slow_yield_usec: 3
                              Options.row_cache: None
                             Options.wal_filter: None
            Options.avoid_flush_during_recovery: 0
            Options.allow_ingest_behind: 0
            Options.preserve_deletes: 0
            Options.two_write_queues: 0
            Options.manual_wal_flush: 0
            Options.max_background_jobs: 12
            Options.max_background_compactions: 4
            Options.avoid_flush_during_shutdown: 0
          Options.writable_file_max_buffer_size: 1048576
            Options.delayed_write_rate : 16777216
            Options.max_total_wal_size: 0
            Options.delete_obsolete_files_period_micros: 21600000000
                  Options.stats_dump_period_sec: 600
                         Options.max_open_files: -1
                         Options.bytes_per_sync: 0
                     Options.wal_bytes_per_sync: 0
      Options.compaction_readahead_size: 4194304
--------------- Options for column family [default]:
              Options.comparator: leveldb.BytewiseComparator
          Options.merge_operator: None
       Options.compaction_filter: None
       Options.compaction_filter_factory: None
        Options.memtable_factory: SkipListFactory
           Options.table_factory: BlockBasedTable
           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c87249a270)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x55c8724992a8
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
       Options.write_buffer_size: 134217728
 Options.max_write_buffer_number: 16
         Options.compression: LZ4
                 Options.bottommost_compression: Disabled
      Options.prefix_extractor: nullptr
  Options.memtable_insert_with_hint_prefix_extractor: nullptr
            Options.num_levels: 7
       Options.min_write_buffer_number_to_merge: 2
    Options.max_write_buffer_number_to_maintain: 0
           Options.compression_opts.window_bits: -14
                 Options.compression_opts.level: -1
              Options.compression_opts.strategy: 0
        Options.compression_opts.max_dict_bytes: 0
        Options.compression_opts.zstd_max_train_bytes: 0
     Options.level0_file_num_compaction_trigger: 4
         Options.level0_slowdown_writes_trigger: 30
             Options.level0_stop_writes_trigger: 40
                  Options.target_file_size_base: 134217728
            Options.target_file_size_multiplier: 1
               Options.max_bytes_for_level_base: 268435456
Options.level_compaction_dynamic_level_bytes: 0
         Options.max_bytes_for_level_multiplier: 10.000000
Options.max_bytes_for_level_multiplier_addtl[0]: 1
Options.max_bytes_for_level_multiplier_addtl[1]: 1
Options.max_bytes_for_level_multiplier_addtl[2]: 1
Options.max_bytes_for_level_multiplier_addtl[3]: 1
Options.max_bytes_for_level_multiplier_addtl[4]: 1
Options.max_bytes_for_level_multiplier_addtl[5]: 1
Options.max_bytes_for_level_multiplier_addtl[6]: 1
      Options.max_sequential_skip_in_iterations: 8
                   Options.max_compaction_bytes: 3355443200
                       Options.arena_block_size: 16777216
  Options.soft_pending_compaction_bytes_limit: 68719476736
  Options.hard_pending_compaction_bytes_limit: 274877906944
      Options.rate_limit_delay_max_milliseconds: 100
               Options.disable_auto_compactions: 0
                       Options.compaction_style: kCompactionStyleLevel
                         Options.compaction_pri: kByCompensatedSize
Options.compaction_options_universal.size_ratio: 1
Options.compaction_options_universal.min_merge_width: 2
Options.compaction_options_universal.max_merge_width: 4294967295
Options.compaction_options_universal.max_size_amplification_percent: 200
Options.compaction_options_universal.compression_size_percent: -1
Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
Options.compaction_options_fifo.max_table_files_size: 1073741824
Options.compaction_options_fifo.allow_compaction: 0
Options.compaction_options_fifo.ttl: 0
                  Options.table_properties_collectors:
                  Options.inplace_update_support: 0
                Options.inplace_update_num_locks: 10000
              Options.memtable_prefix_bloom_size_ratio: 0.000000
  Options.memtable_huge_page_size: 0
                          Options.bloom_locality: 0
                   Options.max_successive_merges: 0
               Options.optimize_filters_for_hits: 0
               Options.paranoid_file_checks: 0
               Options.force_consistency_checks: 0
               Options.report_bg_io_stats: 0
                              Options.ttl: 0
iSynaptic commented 5 years ago

I believe addressing this issue is critical before we can label this library v1.0. I'm sure there are other similar issues, but this is one of them.

rw commented 4 years ago

I'd like to bump the importance of this issue. In particular, enabling settings equivalent to PrepareForBulkLoad is important to my use case.

methyl commented 4 years ago

@rw is there any setting in PrepareForBulkLoad that is unavailable now, or you rather need a convenience method to do it it in one pass?

Chaoses-Ib commented 1 year ago

Some missing options:

Also, format_version now defaults to 5 instead of 2, but rust-rocksdb still documents it as 2.

We should probably reconsider making FFI public (#108). The lack of functionality in binding is a frustrating thing.