search

vitessio / vitess

Vitess is a database clustering system for horizontal scaling of MySQL.
http://vitess.io
Apache License 2.0
18.53k stars 2.09k forks source link

RFC: routing rules to mirror traffic during MoveTables #13772

Closed maxenglander closed 1 week ago

maxenglander commented 1 year ago

Motivation

MoveTables facilitates moving workloads from non-Vitess MySQL into Vitess. Typical issues users may encounter during SwitchTraffic:

Users should mitigate these challenges through best practices such as:

However, these best practices come with their own challenges:

Proposal

To complement these best practices, this RFC proposes a new type of routing rule to mirror some or all traffic to the target keyspace in the course of a MoveTables migration. This feature will allow users to easily test how a new, production-grade Vitess cluster will handle a production workload prior to running SwitchTraffic.

Rule Definitions

Define a new topology record MirrorRules to support mirror rules that look like this:

{
  "rules": [
    {
      "fromKeyspace": "ks1@rdonly",
      "toKeyspace": "ks2",
      "percent": 50.0
    }
  ]
}

VSchema

vindexes.BuildKeyspace will be updated to take new proto definitions into account. vindexes.(*VSchema).FindMirrorTables will provide other packages visibility into mirror rules.

Validate Mirror Rules

Validate constraints on mirror rules usage, e.g.:

+       if sourceMirror.Percent <= 0 || sourceMirror.Percent > 100 {
+               newRule.Error = vterrors.Errorf(
+                       vtrpcpb.Code_INVALID_ARGUMENT,
+                       "mirror rule mirror percentage must be between 0 (exclusive) and 100",
+               )
+               return newRule
+       }

Find Mirror Tables

Other parts of the code base will need to be able to introspect tables which have mirror rules on them.

+// FindMirroredTables finds tables that mirror an authoritative table.
+func (vschema *VSchema) FindMirroredTables(
+        keyspace, 
+        tablename string,
+        tabletType topodatapb.TabletType,
+) (map[*Table]*Mirror, error) {

A table will be considered "mirrored" if there is a mirror rule defined for keyspace, and the target keyspace has a matching table name.

Query Planning

Modify planbuilder to produce engine.Mirror primitives when a SELECT query is planned, and the tables that are touched are all in keyspaces with mirror rules to other keyspaces.

func createInstructionFor(ctx context.Context, query string, stmt sqlparser.Statement, reservedVars *sqlparser.ReservedVars, vschema plancontext.VSchema, enableOnlineDDL, enableDirectDDL bool) (*planResult, error) {
    switch stmt := stmt.(type) {
-   case *sqlparser.Select, *sqlparser.Insert, *sqlparser.Update, *sqlparser.Delete:
+   case *sqlparser.Select:
+       return buildMirroredRoutePlan(ctx, query, stmt, reservedVars, vschema, enableOnlineDDL, enableDirectDDL)
+   case *sqlparser.Insert, *sqlparser.Update, *sqlparser.Delete:
        configuredPlanner, err := getConfiguredPlanner(vschema, stmt, query)

Query Execution

Query planning will produce a Mirror primitive that executes the main and mirrored queries in parallel, returning only the results and the error from the main query.

Some additional goals, not demonstrated in the code sample below:

func (m *Mirror) TryExecute(
       ctx context.Context,
       vcursor VCursor,
       bindVars map[string]*querypb.BindVariable,
       wantfields bool,
) (*sqltypes.Result, error) {
       var wg sync.WaitGroup
       defer wg.Wait()

       for _, target := range m.Targets {
               if !target.Accept() {
                       continue
               }

               wg.Add(1)
               go func(target Primitive, vcursor VCursor) {
                       defer wg.Done()
                       _, _ = target.TryExecute(ctx, vcursor, bindVars, wantfields)
               }(target, vcursor.CloneForMirroring(ctx))
       }

       return m.Primitive.TryExecute(ctx, vcursor, bindVars, wantfields)
}

func (m *PercentMirrorTarget) Accept() bool {
       return m.Percent > (rand.Float32() * 100.0)
}

MoveTables

Introduce a new MoveTables command: MirrorTraffic.

func commandMirrorTraffic(cmd *cobra.Command, args []string) error {
       format, err := GetOutputFormat(cmd)
       if err != nil {
               return err
       }

       cli.FinishedParsing(cmd)

       req := &vtctldatapb.WorkflowMirrorTrafficRequest{
               Keyspace:    BaseOptions.TargetKeyspace,
               Workflow:    BaseOptions.Workflow,
               TabletTypes: MirrorTrafficOptions.TabletTypes,
               Percent:     MirrorTrafficOptions.Percent,
       }

Monitoring

Existing metrics

Mirrored queries will appear in VTTablet-level metrics of the keyspace(s) where they are sent. Users will be able to compare the query rates, error rates, and latencies of the primary and mirrored keyspace(s).

I don't see a compelling reason to add additional metrics, labels, or label values at the VTTablet to indicate which queries were mirrored.

Differences between main and mirrored queries

I think the most useful thing to will be for VTGate to instrument differences between main queries and mirrored queries. For example if mirrored queries take 20% longer than their main counterpart. Similarly, if the main query does not return an error, but the mirrored queries do.

In terms of how this could be reported, it wouldn't be feasible to instrument this in metrics on a per-query-pattern basis. We can log differences, but that could potentially result in a lot of logging for a very busy application.

What I suggest doing is a cache with a user-definable max size, which maps query patterns to stats that can be fetched on demand through :15000 or logged periodically (e.g. every T minutes or every N queries) to stderr or a file.

Also

Other things that could be useful to monitor and/or log:

DML

I think there is use case for mirroring DML. Some applications are write-heavy, and for those applications it would be useful to test the throughput and latency of queries through Vitess. Additionally, mirroring DML may surface subtle issues in V/Schema design, e.g. where a main query to an unsharded source keyspace returns a duplicate key error, but the mirrored query to the target keyspace do not because a unique key is not enforced globally.

However, in the context of MoveTables, it does not make sense to commit DML to the target keyspace, since it would break VReplication (duplicate keys) or corrupt data. One way DML could be mirrored without wreaking havoc would be to wrap every DML in a transaction, and to ensure that every transaction is rolled back and never committed. Even then, there would be the strong likelihood of lock contention which would affect the performance of the mirrored DML.

Given the risk and uncertainty around mirrored DML, I suggest that an initial implementation only mirror Route (= SELECT) operators. A bug in mirrored DML could cause a lot of damage, so I think it is wise to defer any attempt at this until an initial, SELECT-only implementation has stabilized through production trials.

Performance

Performance considerations have been mentioned in various sections above, and are repeated here:

Alternatives

Some alternatives to parts of the proposal above.

Accommodate mirror rules within routing rules

Originally, the RFC proposed that we accommodate mirror rules within routing rules. After discussion, we agreed not to go with this approach, because it restricts the shape that mirror rule definitions can take, and risks breaking or complicating routing rule processing. Here is the content of the original proposal:

 // MirrorRule specifies a routing rule.
 message MirrorRule {
+  enum Action {
+    ACTION_UNSPECIFIED = 0;
+    ACTION_REDIRECT = 1;
+    ACTION_MIRROR = 2;
+  }
+
+  message Mirror {
+    float percent = 1;
+  }
+
   string from_keyspace = 1;
   string to_keyspace = 2;
   float percent = 3;
 }

Pros:

Cons:

Decouple main and mirrored queries

In the proposal above, main and mirror queries execute in parallel, but complete as a group:

func (m *Mirror) TryExecute(
       ctx context.Context, 
       vcursor VCursor,
       bindVars map[string]*querypb.BindVariable,
       wantfields bool,
) (*sqltypes.Result, error) {
       var wg sync.WaitGroup
       defer wg.Wait()

       for _, target := range m.Targets {
               if !target.Accept() {
                       continue
               }

               wg.Add(1)
               go func(target Primitive, vcursor VCursor) {
                       defer wg.Done()
                       _, _ = target.TryExecute(ctx, vcursor, bindVars, wantfields)
               }(target, vcursor.CloneForMirroring(ctx))
       }

       return m.Primitive.TryExecute(ctx, vcursor, bindVars, wantfields)
}

An alternate approach would be to decouple the main and mirrored queries, so that a reply can be sent back to the caller as soon as the main query finishes. Mirrored queries can continue executing afterwards, up to a timeout or global concurrency limit. A similar approach was used to implement replica warming queries.

Pros:

Cons:

Implement mirroring at the operator level

The proposal initially recommended implementing mirroring at the operator level. After receiving feedback from the Vitess team, the proposal was amended to mirror at the plan-level. Here is the content of the original proposal:

Incorporate mirror operators into the operator tree. Prune and consolidate these operators in a way that balances the intent of mirroring with other concerns like performance.

Horizon
└── Mirror
    ├── Route (EqualUnique on user Vindex[shard_index] Values[1] Seen:[m1.id = 1])
    │   └── ApplyJoin (on m1.id = m2.mirror_id columns: )
    │       ├── Table (user.mirror AS m1 WHERE m1.id = 1)
    │       └── Table (user.mirror_fragment AS m2)
    └── PercentMirrorTarget (Percent:50.000000)
        └── Route (Unsharded on main)
            └── ApplyJoin (on m1.id = m2.mirror_id columns: )
                ├── Table (main.mirror AS m1 WHERE m1.id = 1)
                └── Table (main.mirror_fragment AS m2)

Proof-of-concept

https://github.com/vitessio/vitess/pull/14872

Schedule

If this RFC is approved, I think it will make sense to sequence blocks of development in this order:

  1. [x] PR to implement basic mirroring:
    • Add new mirror rules protos.
    • Implement Mirror operator planning and engine primitive.
  2. [x] Separate PR to introduce MoveTables MirrorTraffic.
  3. [ ] Stabilize through production usage.
    • Fix bugs and improve performance.
    • Improve monitoring/logging
  4. [ ] PR to implement DML mirroring with auto-rollback.

Approval

Looking forward to feedback, and ultimately hoping for approval, from the following groups/people.

GuptaManan100 commented 9 months ago

@maxenglander This is a very well-written RFC! I looked at the query-serving related code in the linked PR too, and that looks good to me too!

harshit-gangal commented 9 months ago

@maxenglander The idea is great to have such a feature. I would like to share some pointers.

maxenglander commented 9 months ago

Hey @harshit-gangal thanks for the feedback 🤗

maxenglander commented 8 months ago

In discussions with @rohit-nayak-ps, we agreed that if we go forward with this it should use a new topology record type, and not try to fit this into the routing rules record type. This will give us more flexibility in how we define the record type, and reduce the likelihood that we complicate or break the routing rules implementation. Updated RFC to reflect.

maxenglander commented 8 months ago

Met with members of the Vitess team yesterday to gather feedback on the proposal and the demo implementation.

Some notes:

Next steps: update current demo implementation to use alternative query planning/execution approach, and then loop back for another round of feedback.

maxenglander commented 7 months ago

I will no longer be able to work on this. If someone else wants to take this over feel free to re-open. Thanks everyone for the feedback and discussions on this!