Bug Report: Doing a MoveTables Cancel does not remove tablet controls

Overview of the Issue

When running MoveTables workflows it's common for us to perform various test such as doing a SwitchTraffic to point our source; and to do a ReverseTraffic to point back to the source keyspace. At the end of our testing our environments typically point their traffic back to the source; while we cancel the previously running MoveTables workflows.

Recently it was discovered that when we do a MoveTables Cancel it doesn't clean up Tablet controls, and it remains even when all the workflows are no longer running; and the need for tablet controls becomes irrelevant.

Reproduction Steps

Run a workflow and do a SwitchTraffic and then a ReverseTraffic; this will put tablet controls on your target shards. This can be shown with GetShard:

$ vtctlclient --server :15999 GetShard fane_import_sharded/-80
{
...
  "tablet_controls": [
    {
      "tablet_type": 1,
      "cells": [],
      "denied_tables": [
        "sbtest1",
        "sbtest2",
        "sbtest3",
        "sbtest4",
        "sbtest5",
        "sbtest6",
        "testing"
      ],
...
}

Run a Workflow Cancel to remove the running workflows

$ vtctlclient --server :15999 MoveTables cancel fane_import_sharded.import-shard-80 
cancel was successful for workflow fane_import_sharded.import-shard-80
Start State: Reads partially switched, for shards: 80-. Writes partially switched, for shards: 80-
Current State: Workflow Not Found

Note that the tablet controls remain after the workflow is removed:

$ vtctlclient --server :15999 GetShard fane_import_sharded/-80
{
...
  "tablet_controls": [
    {
      "tablet_type": 1,
      "cells": [],
      "denied_tables": [
        "sbtest1",
        "sbtest2",
        "sbtest3",
        "sbtest4",
        "sbtest5",
        "sbtest6",
        "testing"
      ],
...
}

Binary Version

Vitess 16.0.3

Operating System and Environment details

n/a

Log Fragments

n/a

vitessio / vitess