apache / incubator-pegasus

Apache Pegasus - A horizontally scalable, strongly consistent and high-performance key-value store
https://pegasus.apache.org/
Apache License 2.0
1.99k stars 312 forks source link

feat: support to force send non-idempotent write when doing duplication #1908

Open ninsmiracle opened 9 months ago

ninsmiracle commented 9 months ago

What problem does this PR solve?

1907

What is changed and how does it work?

  1. Change the judgment logic in the writing process, and configure it to allow non-idempotent operations in the master cluster.
  2. Modify the logic of the specified mutation committer in prepare_list
  3. Add configuration items and add them to the replica side to modify them through the server-config of admin-cli.
  4. Add the parsing logic after the backup cluster receives the packaged RPC of duplication.
  5. Add the logic for the master cluster to obtain relevant non-idempotent RPC from the plog
  6. Add the metric index of dup non-idempotent write QPS
Tests

when user want to force dup non-idempotent mutations. Type following command in admin-cli:

server-config replica force_send_no_idempotent_when_duplication set true
ninsmiracle commented 9 months ago

This PR's base function is in line with expectations , however I was requested by business department to add more matrix for surveillance system about this feature. I will update this PR in recently days.

ninsmiracle commented 8 months ago

Now , when user config force_send_no_idempotent_when_duplication to true . We can dup non-idempotent write to backup cluster , and we can verify the data consistency by check the number rows of pegasus app. app in master-cluster image

app in backup-cluster image

And because the retries for non-idempotent writes will make influence of the consistency between two clusters. So we should show the metric like following picture: How many non-idempotent write have been retried when doing duplication

show in backup cluster: How many non-idempotent write have been received by backup cluster