Hanging consul queries can still result in Vitess query path impact

[aquarapid]

Scenario:

• New main (1/12/22 b8964f9c2cabf8b50c4cf2a01dfb47d72fe9784f)
• Using a single consul topo server (default consul config, no special params)
• Single vtgate, unsharded keyspace
• After bootstrapping the cluster, do a kill -STOP of the consul process
• Now connect to vtgate:

  
  time mysql -u user -P 15306 -h 127.0.0.1 -p  
  Enter password: 
  Welcome to the MySQL monitor.  Commands end with ; or \g.
  ^\Aborted

real 1m14.809s user 0m0.005s sys 0m0.007s

  * i.e. had to `Ctrl-\` to stop the mysql client from hanging.
  * Collecting vtgate goroutine dumps and comparing them, it seems likely that the problem is in this goroutine stack:

0xd34699 vitess.io/vitess/go/vt/srvtopo.(*watchEntry).currentValueLocked+0x159 vitess.io/vitess/go/vt/srvtopo/watch.go:136

0xd34015 vitess.io/vitess/go/vt/srvtopo.(*resilientWatcher).getValue+0xd5 vitess.io/vitess/go/vt/srvtopo/watch.go:92

0xd356de vitess.io/vitess/go/vt/srvtopo.(*SrvKeyspaceWatcher).GetSrvKeyspace+0xbe vitess.io/vitess/go/vt/srvtopo/watch_srvkeyspace.go:72

0xd31a01 vitess.io/vitess/go/vt/srvtopo.(*Resolver).GetKeyspaceShards+0x81 vitess.io/vitess/go/vt/srvtopo/resolver.go:106

0xd329c4 vitess.io/vitess/go/vt/srvtopo.(*Resolver).ResolveDestinations+0xe4 vitess.io/vitess/go/vt/srvtopo/resolver.go:264

0xee5c85 vitess.io/vitess/go/vt/vtgate.(*vcursorImpl).ResolveDestinations+0x65 vitess.io/vitess/go/vt/vtgate/vcursor_impl.go:558

0xdea1ab vitess.io/vitess/go/vt/vtgate/engine.(*Route).paramsAnyShard+0x8b vitess.io/vitess/go/vt/vtgate/engine/route.go:572

0xde7b44 vitess.io/vitess/go/vt/vtgate/engine.(*Route).findRoute+0x1e4 vitess.io/vitess/go/vt/vtgate/engine/route.go:309

0xde75f2 vitess.io/vitess/go/vt/vtgate/engine.(*Route).executeInternal+0x52 vitess.io/vitess/go/vt/vtgate/engine/route.go:267

0xde7479 vitess.io/vitess/go/vt/vtgate/engine.(*Route).TryExecute+0xb9 vitess.io/vitess/go/vt/vtgate/engine/route.go:259

0xee4603 vitess.io/vitess/go/vt/vtgate.(*vcursorImpl).ExecutePrimitive+0x83 vitess.io/vitess/go/vt/vtgate/vcursor_impl.go:402

0xeca85a vitess.io/vitess/go/vt/vtgate.(*Executor).executePlan+0x7a vitess.io/vitess/go/vt/vtgate/plan_execute.go:213

0xeb9acb vitess.io/vitess/go/vt/vtgate.(*Executor).execute.func1+0x8b vitess.io/vitess/go/vt/vtgate/executor.go:356

0xeca052 vitess.io/vitess/go/vt/vtgate.(*Executor).newExecute+0x5b2 vitess.io/vitess/go/vt/vtgate/plan_execute.go:112

0xeb98d8 vitess.io/vitess/go/vt/vtgate.(*Executor).execute+0x158 vitess.io/vitess/go/vt/vtgate/executor.go:354

0xeb7ec6 vitess.io/vitess/go/vt/vtgate.(*Executor).Execute+0x2c6 vitess.io/vitess/go/vt/vtgate/executor.go:169

0xef0f3c vitess.io/vitess/go/vt/vtgate.(*VTGate).Execute+0x2dc vitess.io/vitess/go/vt/vtgate/vtgate.go:382

0xecc3da vitess.io/vitess/go/vt/vtgate.(*vtgateHandler).ComQuery+0x53a vitess.io/vitess/go/vt/vtgate/plugin_mysql_server.go:222

0xc154b5 vitess.io/vitess/go/mysql.(*Conn).execQuery+0x135 vitess.io/vitess/go/mysql/conn.go:1263

0xc1513d vitess.io/vitess/go/mysql.(*Conn).handleComQuery+0x33d vitess.io/vitess/go/mysql/conn.go:1248

0xc1218e vitess.io/vitess/go/mysql.(*Conn).handleNextCommand+0x1ee vitess.io/vitess/go/mysql/conn.go:885

0xc30687 vitess.io/vitess/go/mysql.(*Listener).handle+0x1427 vitess.io/vitess/go/mysql/server.go:474

0xc2f1de vitess.io/vitess/go/mysql.(*Listener).Accept.func1+0xfe vitess.io/vitess/go/mysql/server.go:286

  * Doing something similar with `etcd` and `zk` does not seem to cause a similar issue.

[mattlord]

Related to: https://github.com/vitessio/vitess/issues/9147