ERROR: internal error: found MULTIREGION_ENUM on non-multi-region database crdb_temp_system

[smcvey]

Describe the problem

A backup taken on a 24.1 Serverless cluster from crl-prod-j77 (AWS, us-east-1) was then attempted to be restored into a 24.2 OnPrem cluster (with a single locality of region=us-east-1). During restore, this stack trace was produced:

root@localhost:26257/system> RESTORE FROM LATEST IN  .... s3 details elided...

ERROR: internal error: found MULTIREGION_ENUM on non-multi-region database crdb_temp_system
SQLSTATE: XX000
DETAIL: stack trace:
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1229: func1()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:1930: func3()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:2027: func4()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:1049: exec()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1103: runTxn()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1066: TxnWithAdmissionControl()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1041: Txn()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:2014: txn()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:1928: DescsTxn()
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1190: createImportingDescriptors()
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1782: doResume()
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:1720: Resume()
github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1632: func2()
github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1633: stepThroughStateMachine()
github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:451: runJob()
github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:831: func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485: func2()
src/runtime/asm_amd64.s:1695: goexit()

HINT: You have encountered an unexpected error.

To Reproduce

• Create a single node 24.2.2 cluster, with the node having a locality of --locality=region=aws-us-east-1.
• RESTORE FROM ... (contact me separately for the AWS keys and details).

Expected behavior The restore should work as normal.

Environment:

• CockroachDB version: 24.1 backup, 24.2 restore.

Additional context Restore could not be taken.

Add any other context about the problem here.

• There's the possibility that the backup was taken while the host cluster was upgrading from 24.1 to 24.2. Although an earlier error message shows that the backup was taken on a 24.1 cluster.

Jira issue: CRDB-42475

[blathers-crl[bot]]

Hi @smcvey, please add branch-* labels to identify which branch(es) this C-bug affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}