sql: panic `genPopulateClusterLocksWithIndex` on external tables

[msbutler]

When a PCR reader tenant's crdb_internal.cluster_locks virtual table's worker goroutine iterates all tables to accumulate the set of spans in which it will query for locks, it is attempting to get the span for tables that are external, which is invalid -- external tables by definition do not have a key span since their data is external. Thus the TableSpan() call panics.

goroutine 774254 gp=0xc0110c9180 m=12 mp=0xc0013d6008 [running]:
panic({0x67d72e0?, 0xc0154f6a20?})
        GOROOT/src/runtime/panic.go:779 +0x158 fp=0xc01643d590 sp=0xc01643d4e0 pc=0x49ca98
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3cdf747?, {0x87b8dc0, 0xc018b438f0})
        pkg/util/stop/stopper.go:226 +0x65 fp=0xc01643d5d8 sp=0xc01643d590 pc=0x131ab85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2.deferwrap4()
        pkg/util/stop/stopper.go:492 +0x28 fp=0xc01643d600 sp=0xc01643d5d8 pc=0x131c548
panic({0x67d72e0?, 0xc0154f6a20?})
        GOROOT/src/runtime/panic.go:770 +0x132 fp=0xc01643d6b0 sp=0xc01643d600 pc=0x49ca72
github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*wrapper).TableSpan(0xc019a47da0?, {{0xc011c684b0, 0xc011c684c8, {0x4}}, {0xc011c684b0}, 0x0})
        pkg/sql/catalog/tabledesc/structured.go:2303 +0x1e6 fp=0xc01643d750 sp=0xc01643d6b0 pc=0x1fa1a06
github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc.(*immutable).TableSpan(0xc8e7c90?, {{0xc011c684b0, 0xc011c684c8, {0x4}}, {0xc011c684b0}, 0x0})
        <autogenerated>:1 +0x92 fp=0xc01643d7f0 sp=0xc01643d750 pc=0x1fd6512
github.com/cockroachdb/cockroach/pkg/sql.populateClusterLocksWithFilter.genClusterLocksGenerator.func1({0x87b8dc0, 0xc018b438f0}, 0xc013f2a138, {0x470b01?, 0x1?}, 0x610f140?)
        pkg/sql/crdb_internal.go:8023 +0xb24 fp=0xc01643da30 sp=0xc01643d7f0 pc=0x3ad4a64
github.com/cockroachdb/cockroach/pkg/sql.populateClusterLocksWithFilter({0x87b8dc0?, 0xc018b438f0?}, 0x88a1458?, {0x8872f50?, 0xc0170c98c0?}, 0xc015f3b0e0, {0x0, 0x0, 0x0, 0xc011b88399})
        pkg/sql/crdb_internal.go:8233 +0x9a fp=0xc01643daa8 sp=0xc01643da30 pc=0x3ad3e7a
github.com/cockroachdb/cockroach/pkg/sql.init.genPopulateClusterLocksWithIndex.func962({0x87b8dc0, 0xc018b438f0}, {0x8832010, 0xcad1267}, 0xc013f2a138, {0x8872f50, 0xc0170c98c0}, 0xc015f3b0e0)
        pkg/sql/crdb_internal.go:8220 +0x109 fp=0xc01643db58 sp=0xc01643daa8 pc=0x398d009
github.com/cockroachdb/cockroach/pkg/sql.(*virtualDefEntry).getPlanInfo.func1.(*virtualDefEntry).makeConstrainedRowsGenerator.2({0x87b8dc0, 0xc018b438f0}, {0x8765ba0, 0xc0143ab1e0})
        pkg/sql/virtual_schema.go:836 +0x58f fp=0xc01643ddc0 sp=0xc01643db58 pc=0x3cdc08f
github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3({0x87b8dc0, 0xc018b438f0})
        pkg/sql/virtual_table.go:122 +0x104 fp=0xc01643dec0 sp=0xc01643ddc0 pc=0x3cdf5e4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2({0x6f07084?, 0xc018a5d050?})
        pkg/util/stop/stopper.go:498 +0x1f0 fp=0xc01643dfc0 sp=0xc01643dec0 pc=0x131c330
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
        pkg/util/stop/stopper.go:499 +0x28 fp=0xc01643dfe0 sp=0xc01643dfc0 pc=0x131c108
runtime.goexit({})
        src/runtime/asm_amd64.s:1695 +0x1 fp=0xc01643dfe8 sp=0xc01643dfe0 pc=0x4d90a1
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 774251
        pkg/util/stop/stopper.go:488 +0x49c

Additionally, this bug surprisingly crashes the node, rather than fails the query, as this panic appears to not be recovered because virtual table generators run their generation function in a separate goroutine.

Jira issue: CRDB-44699

[blathers-crl[bot]]

Hi @msbutler, please add branch-* labels to identify which branch(es) this C-bug affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl[bot]]

cc @cockroachdb/disaster-recovery