Revert an internal change in 7.8.0 associated with some memory usage churn.
7.8.2 (2022-11-27)
Behavior changes
Make best-efforts recovery verify SST unique ID before Version construction (#10962)
Fix failed memtable flush retry bug that could cause wrongly ordered updates, which would surface to writers as Status::Corruption in case of force_consistency_checks=true (default). It affects use cases that enable both parallel flush (max_background_flushes > 1 or max_background_jobs >= 8) and non-default memtable count (max_write_buffer_number > 2).
Tiered Storage: fixed excessive keys written to penultimate level in non-debug builds.
Bug Fixes
Fixed a regression in scan for async_io. During seek, valid buffers were getting cleared causing a regression.
Fixed a performance regression in iterator where range tombstones after iterate_upper_bound is processed.
7.8.1 (2022-11-02)
Bug Fixes
Fix memory corruption error in scans if async_io is enabled. Memory corruption happened if there is IOError while reading the data leading to empty buffer and other buffer already in progress of async read goes again for reading.
7.8.0 (2022-10-22)
New Features
DeleteRange() now supports user-defined timestamp.
Provide support for async_io with tailing iterators when ReadOptions.tailing is enabled during scans.
Tiered Storage: allow data moving up from the last level to the penultimate level if the input level is penultimate level or above.
Added DB::Properties::kFastBlockCacheEntryStats, which is similar to DB::Properties::kBlockCacheEntryStats, except returns cached (stale) values in more cases to reduce overhead.
FIFO compaction now supports migrating from a multi-level DB via DB::Open(). During the migration phase, FIFO compaction picker will:
picks the sst file with the smallest starting key in the bottom-most non-empty level.
Note that during the migration phase, the file purge order will only be an approximation of "FIFO" as files in lower-level might sometime contain newer keys than files in upper-level.
Added an option ignore_max_compaction_bytes_for_input to ignore max_compaction_bytes limit when adding files to be compacted from input level. This should help reduce write amplification. The option is enabled by default.
Tiered Storage: allow data moving up from the last level even if it's a last level only compaction, as long as the penultimate level is empty.
Add a new option IOOptions.do_not_recurse that can be used by underlying file systems to skip recursing through sub directories and list only files in GetChildren API.
Add option preserve_internal_time_seconds to preserve the time information for the latest data. Which can be used to determine the age of data when preclude_last_level_data_seconds is enabled. The time information is attached with SST in table property rocksdb.seqno.time.map which can be parsed by tool ldb or sst_dump.
Bug Fixes
Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed.
Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716.
Fixed an optimistic transaction validation bug caused by DBImpl::GetLatestSequenceForKey() returning non-latest seq for merge (#10724).
Fixed a bug in iterator refresh which could segfault for DeleteRange users (#10739).
Fixed a bug causing manual flush with flush_opts.wait=false to stall when database has stopped all writes (#10001).
Fixed a bug in iterator refresh that was not freeing up SuperVersion, which could cause excessive resource pinniung (#10770).
Fixed a bug where RocksDB could be doing compaction endlessly when allow_ingest_behind is true and the bottommost level is not filled (#10767).
Fixed a memory safety bug in experimental HyperClockCache (#10768)
Fixed some cases where ldb update_manifest and ldb unsafe_remove_sst_file are not usable because they were requiring the DB files to match the existing manifest state (before updating the manifest to match a desired state).
Performance Improvements
Try to align the compaction output file boundaries to the next level ones, which can reduce more than 10% compaction load for the default level compaction. The feature is enabled by default, to disable, set AdvancedColumnFamilyOptions.level_compaction_dynamic_file_size to false. As a side effect, it can create SSTs larger than the target_file_size (capped at 2x target_file_size) or smaller files.
Improve RoundRobin TTL compaction, which is going to be the same as normal RoundRobin compaction to move the compaction cursor.
Fix a small CPU regression caused by a change that UserComparatorWrapper was made Customizable, because Customizable itself has small CPU overhead for initialization.
Fixed an iterator performance regression for delete range users when scanning through a consecutive sequence of range tombstones (#10877).
Revert an internal change in 7.8.0 associated with some memory usage churn.
7.8.2 (11/27/2022)
Behavior changes
Make best-efforts recovery verify SST unique ID before Version construction (#10962)
Fix failed memtable flush retry bug that could cause wrongly ordered updates, which would surface to writers as Status::Corruption in case of force_consistency_checks=true (default). It affects use cases that enable both parallel flush (max_background_flushes > 1 or max_background_jobs >= 8) and non-default memtable count (max_write_buffer_number > 2).
Tiered Storage: fixed excessive keys written to penultimate level in non-debug builds.
Bug Fixes
Fixed a regression in scan for async_io. During seek, valid buffers were getting cleared causing a regression.
Fixed a performance regression in iterator where range tombstones after iterate_upper_bound is processed.
7.8.1 (11/2/2022)
Bug Fixes
Fix memory corruption error in scans if async_io is enabled. Memory corruption happened if there is IOError while reading the data leading to empty buffer and other buffer already in progress of async read goes again for reading.
7.8.0 (10/22/2022)
New Features
DeleteRange() now supports user-defined timestamp.
Provide support for async_io with tailing iterators when ReadOptions.tailing is enabled during scans.
Tiered Storage: allow data moving up from the last level to the penultimate level if the input level is penultimate level or above.
Added DB::Properties::kFastBlockCacheEntryStats, which is similar to DB::Properties::kBlockCacheEntryStats, except returns cached (stale) values in more cases to reduce overhead.
FIFO compaction now supports migrating from a multi-level DB via DB::Open(). During the migration phase, FIFO compaction picker will:
picks the sst file with the smallest starting key in the bottom-most non-empty level.
Note that during the migration phase, the file purge order will only be an approximation of "FIFO" as files in lower-level might sometime contain newer keys than files in upper-level.
Added an option ignore_max_compaction_bytes_for_input to ignore max_compaction_bytes limit when adding files to be compacted from input level. This should help reduce write amplification. The option is enabled by default.
Tiered Storage: allow data moving up from the last level even if it's a last level only compaction, as long as the penultimate level is empty.
Add a new option IOOptions.do_not_recurse that can be used by underlying file systems to skip recursing through sub directories and list only files in GetChildren API.
Add option preserve_internal_time_seconds to preserve the time information for the latest data. Which can be used to determine the age of data when preclude_last_level_data_seconds is enabled. The time information is attached with SST in table property rocksdb.seqno.time.map which can be parsed by tool ldb or sst_dump.
Bug Fixes
Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed.
Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716.
Fixed an optimistic transaction validation bug caused by DBImpl::GetLatestSequenceForKey() returning non-latest seq for merge (#10724).
Fixed a bug in iterator refresh which could segfault for DeleteRange users (#10739).
Fixed a bug causing manual flush with flush_opts.wait=false to stall when database has stopped all writes (#10001).
Fixed a bug in iterator refresh that was not freeing up SuperVersion, which could cause excessive resource pinniung (#10770).
Fixed a bug where RocksDB could be doing compaction endlessly when allow_ingest_behind is true and the bottommost level is not filled (#10767).
Fixed a memory safety bug in experimental HyperClockCache (#10768)
Fixed some cases where ldb update_manifest and ldb unsafe_remove_sst_file are not usable because they were requiring the DB files to match the existing manifest state (before updating the manifest to match a desired state).
Performance Improvements
Try to align the compaction output file boundaries to the next level ones, which can reduce more than 10% compaction load for the default level compaction. The feature is enabled by default, to disable, set AdvancedColumnFamilyOptions.level_compaction_dynamic_file_size to false. As a side effect, it can create SSTs larger than the target_file_size (capped at 2x target_file_size) or smaller files.
Improve RoundRobin TTL compaction, which is going to be the same as normal RoundRobin compaction to move the compaction cursor.
Fix a small CPU regression caused by a change that UserComparatorWrapper was made Customizable, because Customizable itself has small CPU overhead for initialization.
Fixed an iterator performance regression for delete range users when scanning through a consecutive sequence of range tombstones (#10877).
Behavior Changes
Sanitize min_write_buffer_number_to_merge to 1 if atomic flush is enabled to prevent unexpected data loss when WAL is disabled in a multi-column-family setting (#10773).
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot]
commented
1 year ago
Bumps rocksdbjni from 6.25.3 to 7.8.3.
Release notes
Sourced from rocksdbjni's releases.
... (truncated)
Changelog
Sourced from rocksdbjni's changelog.
... (truncated)
Commits
bf2c335Update HISTORY.md and version to 7.8.37aa573eUpdate HISTORY for revertse0cf5cdRevert "Improve / refactor anonymous mmap capabilities (#10810)"7ad900eRevert "Fix include of windows.h in mmap.h (#10885)"491615cupdate version.h for 7.8.21cf4539batch latest fixes into 7.8.2442b6b8Fix CompactionIterator flag for penultimate level output (#10967)8c06988Fix flush picking non-consecutive memtables (#10921)8d5edb6Prevent iterating over range tombstones beyonditerate_upper_bound(#10966)...32d853fMake best-efforts recovery verify SST unique ID before Version construction (...Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)