upgrade/upgrades: TestRoleMembersIDMigration1500Users failed

[cockroach-teamcity]

upgrade/upgrades.TestRoleMembersIDMigration1500Users failed with artifacts on release-23.1 @ 6a51c13a3a9dfa488ca1d0d638fdaec55a7c705e:

Fatal error:

panic: test timed out after 14m55s

Stack:

goroutine 107812 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2036 +0x8e
created by time.goFunc
    GOROOT/src/time/sleep.go:176 +0x32

Log preceding fatal error

``` I240917 06:51:49.690964 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9238 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:49.698776 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9239 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:49.699321 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9240 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:50.686474 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9241 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:50.692500 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9242 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:50.693098 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9243 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:51.640230 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9244 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:51.646777 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9245 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:51.647399 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9246 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:51.951706 8264 2@util/log/event_log.go:32 [T1,n1] 9247 ={"Timestamp":1726555911951701309,"EventType":"runtime_stats","MemRSSBytes":1381023744,"GoroutineCount":645,"MemStackSysBytes":9043968,"GoAllocBytes":810269608,"GoTotalBytes":1148441128,"GoStatsStaleness":0.00225017,"HeapFragmentBytes":40666200,"HeapReservedBytes":207396864,"HeapReleasedBytes":128000000,"CGoAllocBytes":99005392,"CGoTotalBytes":169283584,"CGoCallRate":0.099998355,"CPUUserPercent":109.99819,"CPUSysPercent":1.4999753,"GCPausePercent":0.004324229,"GCRunCount":636,"NetHostRecvBytes":5676,"NetHostSendBytes":5676} I240917 06:51:51.951832 8264 2@server/status/runtime_log.go:47 [T1,n1] 9248 runtime stats: 1.3 GiB RSS, 645 goroutines (stacks: 8.6 MiB), 773 MiB/1.1 GiB Go alloc/total(stale) (heap fragmentation: 39 MiB, heap reserved: 198 MiB, heap released: 122 MiB), 94 MiB/161 MiB CGO alloc/total (0.1 CGO/sec), 110.0/1.5 %(u/s)time, 0.0 %gc (636x), 5.5 KiB/5.5 KiB (r/w)net I240917 06:51:52.632326 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9249 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:52.639390 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9250 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:52.640071 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9251 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:53.573216 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9252 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:53.579353 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9253 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:53.579940 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9254 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:54.534055 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9255 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:54.540225 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9256 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:54.540801 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9257 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:55.559112 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9258 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:55.565993 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9259 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:55.566571 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9260 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:56.527021 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9261 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:56.532737 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9262 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:56.533288 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9263 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:57.507447 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9264 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:57.514731 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9265 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:57.515378 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9266 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:58.481741 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9267 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:58.589256 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9268 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:58.693161 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9269 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:59.803152 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9270 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:51:59.804169 8157 3@pebble/event.go:717 [n1,s1,pebble] 9271 [JOB 43] WAL created 000029 (recycled 000011) I240917 06:51:59.804862 102843 3@pebble/event.go:677 [n1,s1,pebble] 9272 [JOB 44] flushing 1 memtable to L0 I240917 06:51:59.804986 102843 3@pebble/event.go:697 [n1,s1,pebble] 9273 [JOB 44] flushing: sstable created 000030 I240917 06:51:59.810243 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9274 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:51:59.811243 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9275 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:51:59.864542 102843 3@pebble/event.go:681 [n1,s1,pebble] 9276 [JOB 44] flushed 1 memtable to L0 [000030] (827 K), in 0.1s (0.1s total), output rate 14 M/s I240917 06:51:59.864646 102846 3@pebble/event.go:665 [n1,s1,pebble] 9277 [JOB 46] compacting(default) L0 [000030] (827 K) + L6 [000028] (2.4 M) I240917 06:51:59.865602 102846 3@pebble/event.go:697 [n1,s1,pebble] 9278 [JOB 46] compacting: sstable created 000031 I240917 06:51:59.925490 102846 3@pebble/event.go:669 [n1,s1,pebble] 9279 [JOB 46] compacted(default) L0 [000030] (827 K) + L6 [000028] (2.4 M) -> L6 [000031] (2.5 M), in 0.1s (0.1s total), output rate 42 M/s I240917 06:51:59.926090 107635 3@pebble/event.go:701 [n1,s1,pebble] 9280 [JOB 46] sstable deleted 000028 I240917 06:51:59.926135 107635 3@pebble/event.go:701 [n1,s1,pebble] 9281 [JOB 46] sstable deleted 000030 I240917 06:52:00.694219 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9282 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:52:00.701555 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9283 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:52:00.702205 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9284 job 1004291564423151617: updated with schema change for table 33, mutation 0 I240917 06:52:01.560400 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9285 job 1004291566222868481: updated with schema change for table 23, mutation 0 I240917 06:52:01.567911 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9286 job 1004291564420136961: updated with schema change for table 4, mutation 0 I240917 06:52:01.568395 9083 sql/table.go:228 [T1,n1,client=127.0.0.1:38512,hostssl,user=root] 9287 job 1004291564423151617: updated with schema change for table 33, mutation 0 ```

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/release-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-42263

[rail]

This a flake timeout.