search

apache / incubator-pegasus

Apache Pegasus - A horizontally scalable, strongly consistent and high-performance key-value store
https://pegasus.apache.org/
Apache License 2.0
1.99k stars 312 forks source link

Release 1.12.3 #506

Closed neverchanje closed 4 years ago

neverchanje commented 4 years ago

1.12.3-RC1

rDSN

PR (48 TOTAL) TITLE
XiaoMi/rdsn#430 refactor(backup): delay the removal of checkpoint files produced by cold backup
XiaoMi/rdsn#431 refactor: move log-block-writing-related codes from mutation_log to log_block
XiaoMi/rdsn#429 feat(dup): support multiple fail modes for duplication
XiaoMi/rdsn#427 refactor: move log_block class from mutation_log.h to separated file
XiaoMi/rdsn#428 fix(backup): use https to access fds, instead of http
XiaoMi/rdsn#425 fix(dup): multiple fixes on replica duplication
XiaoMi/rdsn#424 feat(dup): add dsn_replica_dup_test to CI testing and fix http outputs
XiaoMi/rdsn#421 feat: add a new app_env to limit scan time
XiaoMi/rdsn#423 feat(dup): rename change_dup_status to modify_dup
XiaoMi/rdsn#422 fix: fix memory leak in tls_transient_memory_t
XiaoMi/rdsn#416 feat: add query_disk_info api for shell command
XiaoMi/rdsn#411 feat(dup): handle non-idempotent writes during duplication
XiaoMi/rdsn#409 feat: add query_disk_info support for replica_server
XiaoMi/rdsn#420 refactor: delete unused log in replica::on_config_sync
XiaoMi/rdsn#399 feat(split): re-implement function split_replica_exec
XiaoMi/rdsn#415 feat(dup): support replica http api for duplication state query
XiaoMi/rdsn#414 feat: forbid large-size-value writes to Pegasus
XiaoMi/rdsn#390 feat(split): child notify parent catch up
XiaoMi/rdsn#401 feat(dup): support HTTP API to inspect duplication state
XiaoMi/rdsn#403 refactor(utils): make logger decoupled from dsn runtime
XiaoMi/rdsn#406 fix: weak constraint of ipv4 and incorrect code logic in hostname_from_ip
XiaoMi/rdsn#407 feat(dup): respond with hint when add_dup failed
XiaoMi/rdsn#394 feat(split): add partition_version
XiaoMi/rdsn#393 feat(dup): add metrics for duplication
XiaoMi/rdsn#398 refactor: make clock decoupled from rdsn runtime
XiaoMi/rdsn#319 feat(split): child replica apply private logs, in-memory mutations and catch up parent
XiaoMi/rdsn#395 fix(utils): fix bug in time_utils
XiaoMi/rdsn#388 refactor(util): make time_utils decoupled from utils.cpp
XiaoMi/rdsn#387 fix(thrift): add start_time_ns for structure mutation_update in thrift file
XiaoMi/rdsn#384 fix(lsan): memory leak in perf_counter_atomic.h
XiaoMi/rdsn#353 feat: add check for app envs
XiaoMi/rdsn#380 refactor(network): refine codes and comments
XiaoMi/rdsn#381 fix(asan): fix heap-use-after-free in replica_split_test
XiaoMi/rdsn#379 refactor: remove useless files (version.h, coredump.win.cpp, ReleaseNote)
XiaoMi/rdsn#377 fix(asan): heap-buffer-overflow in meta_test_base.h
XiaoMi/rdsn#375 fix(asan): heap-use-after-free in simple_lb_cure_test.cpp
XiaoMi/rdsn#374 feat(dup): add is_duplicating API on replication_app_base
XiaoMi/rdsn#371 refactor: remove aspects for dsn runtime
XiaoMi/rdsn#370 feat(dup): implement load_mutation
XiaoMi/rdsn#355 feat(dup): preserve data consistency during replica learn
XiaoMi/rdsn#365 refactor: move load balance related tests under another test suite
XiaoMi/rdsn#361 fix(asan): pointer used after free in autoref_ptr_test
XiaoMi/rdsn#363 fix(asan): global-buffer-overflow in string_conv_test
XiaoMi/rdsn#362 refactor: remove unused function in greedy_load_balancer
XiaoMi/rdsn#360 fix: sanitizer options don't take effect
XiaoMi/rdsn#359 build: add sanitizer support
XiaoMi/rdsn#356 feat(dup): implement is_duplicating
XiaoMi/rdsn#348 refactor: refine replica restart by adding func get_replay_start_decree
XiaoMi/rdsn#320 feat(dup): protect private log from missing when duplication is enabled

Pegasus

PR (25 TOTAL) TITLE
XiaoMi/pegasus#503 refactor: simplify logging in pegasus_server_impl
XiaoMi/pegasus#498 feat: add shell command for query_disk_info
XiaoMi/pegasus#479 feat(hotspot): new algorithm of hotspot detection
XiaoMi/pegasus#489 fix: fix bug in del function of redis_proxy
XiaoMi/pegasus#491 fix: fix the invalid IP address in src/server/test/config.ini
XiaoMi/pegasus#485 feat(shell): mlog_dump support parsing check_and_set
XiaoMi/pegasus#474 feat(split): add partition_version
XiaoMi/pegasus#478 fix(asan): heap-use-after-free in pegasus_write_service.cpp
XiaoMi/pegasus#481 fix: use db when it has been opened
XiaoMi/pegasus#480 fix: add libraries missing for pegasus_client_lib.so
XiaoMi/pegasus#444 feat(collector): add statistics for partition hotspot
XiaoMi/pegasus#472 docs(community): add issue template for feature-request
XiaoMi/pegasus#466 feat(dup): implement duplication related shell commands
XiaoMi/pegasus#470 feat(shell): add dependency argh to parse options and arguments
XiaoMi/pegasus#469 fix(rdsn): update rdsn and adapt to time_utils refactoring
XiaoMi/pegasus#467 refactor: refactor rocksdb options usage
XiaoMi/pegasus#463 refactor: refactor the code about prometheus in pegasus_counter_reporter
XiaoMi/pegasus#462 refactor(config): remove duplicated config items for cluster name
XiaoMi/pegasus#460 refactor: update rdsn and remove DSN_CORE_VERSION
XiaoMi/pegasus#456 feat(dup): implement server handling of duplicate rpc (part 1)
XiaoMi/pegasus#450 feat(rocksdb): Select the option of Direct-IO in Rocksdb
XiaoMi/pegasus#455 fix: fix the bug that threads don't stop when pegasus_io_service is released
XiaoMi/pegasus#399 feat(dup): implement pegasus_mutation_duplicator
XiaoMi/pegasus#453 fix: retry starting zookeeper if it failed on onebox environment
XiaoMi/pegasus#446 build: add sanitizer support

New Perf-Counters

New HTTP APIs

New App Env

Configuration Changes

[replication]
- allow_non_idempotent_write = false
+ max_allowed_write_size = 1048576 # default = 1MB

[pegasus.server]
+ rocksdb_use_direct_reads = true
+ rocksdb_use_direct_io_for_flush_and_compaction = true
+ rocksdb_compaction_readahead_size = 2097152
+ rocksdb_writable_file_max_buffer_size = 1048576
- perf_counter_cluster_name = %{cluster.name}
- perf_counter_enable_falcon = false
- perf_counter_enable_prometheus = false
+ perf_counter_sink = <falcon | prometheus>

[pegasus.collector]
- cluster = %{cluster.name}
+ hotspot_detect_algorithm = <hotspot_algo_qps_variance | hotspot_algo_qps_skew> 

[replication]
+ cluster_name = %{cluster.name}

Blame list:

New limitation of requests

Like aws dynamodb limits, we imposed read/write limitation to protect the service of Pegasus in XiaoMi. All of the limits can be reconfigured:

  max_allowed_write_size = 1048576 # default = 1MB

Please remember to check if any requests to your service exceed the limits.

neverchanje commented 4 years ago

1.12.3-RC2

rDSN

PR (1 RELEASED) TITLE
XiaoMi/rdsn#430 refactor(backup): delay the removal of checkpoint files produced by cold backup

Pegasus

PR (6 RELEASED) TITLE
XiaoMi/pegasus#511 fix: node qps sum inconsistent with app qps sum
XiaoMi/pegasus#512 fix(collector): throughput statistics isn’t right
XiaoMi/pegasus#500 feat: limit long time rocksdb iteration operation
XiaoMi/pegasus#505 feat(collector): add throughput statistic for table
XiaoMi/pegasus#509 fix(asan): global-buffer-overflow in function escape_sds_argv of data_operations.cpp
XiaoMi/pegasus#508 fix(shell): disk_replica printer bug

New perf-counter

On replica-server

On collector

New configuration

[replication]
+cold_backup_checkpoint_reserve_minutes = 10

[pegasus.server]
# cluster level restriction {3000, 30MB, 1000, 30s}
+rocksdb_multi_get_max_iteration_count = 3000
+rocksdb_multi_get_max_iteration_size = 31457280
+rocksdb_max_iteration_count = 1000
+rocksdb_iteration_threshold_time_ms = 30000

Blame list:

neverchanje commented 4 years ago

1.12.3-RC3

rDSN

PR (1 TOTAL) TITLE
XiaoMi/rdsn#435 feat: tcmalloc memory release improvements

Pegasus

PR (1 TOTAL) TITLE
XiaoMi/pegasus#515 fix: fix the bug in restore_test

New perf-counter

New remote command

command help: replica.release-tcmalloc-memory <true|false>