roachtest: schemachange/random-load failed [DROP VIEW is stuck in reverting state]

[cockroach-teamcity]

roachtest.schemachange/random-load failed with artifacts on release-23.2 @ 6bc1b66344700d53e1c15c8c89cf8f8e66aa41b0:

(test_runner.go:1153).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/schemachange/random-load/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131345 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master release-blocker] - #129857 roachtest: schemachange/random-load failed [relation "[1285]" does not exist] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2.1-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42537

[rafiss]

It looks like a DROP VIEW was running/reverting for over 2 hours:

1006866750911840257 NEW SCHEMA CHANGE   DROP VIEW schemachange.public.view5302  DROP VIEW schemachange.public.view5302  roachprod   {586,753}   reverting   NULL    2024-09-26 09:09:06.015756+00   2024-09-26 09:09:10.402918+00   NULL    2024-09-26 11:14:34.136919+00   0   NULL    failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found 2   5548953609706679951 2024-09-26 11:14:34.398891+00   2024-09-26 13:22:04.398891+00   8   "{""reverting execution from '2024-09-26 09:38:56.201968' to '2024-09-26 09:38:56.468169' on 3 failed: failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found"",""reverting execution from '2024-09-26 10:10:44.724889' to '2024-09-26 10:10:45.852282' on 2 failed: failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found"",""reverting execution from '2024-09-26 11:14:34.398891' to '2024-09-26 11:14:34.663148' on 3 failed: failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found""}"    "[{""executionEndMicros"": ""1727343536468169"", ""executionStartMicros"": ""1727343536201968"", ""instanceId"": 3, ""status"": ""reverting"", ""truncatedError"": ""failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found""}, {""executionEndMicros"": ""1727345445852282"", ""executionStartMicros"": ""1727345444724889"", ""instanceId"": 2, ""status"": ""reverting"", ""truncatedError"": ""failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found""}, {""executionEndMicros"": ""1727349274663148"", ""executionStartMicros"": ""1727349274398891"", ""instanceId"": 3, ""status"": ""reverting"", ""truncatedError"": ""failed to read descriptors [586 753] for the declarative schema change state: referenced descriptor ID 753: looking up ID 753: descriptor not found""}]"

https://github.com/cockroachdb/cockroach/pull/130698 was supposed to help with this.