search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.88k stars 3.77k forks source link

sql: TestRaceWithBackfill timeout #130013

Open cockroach-teamcity opened 1 week ago

cockroach-teamcity commented 1 week ago

pkg/sql/sqltest/sql_test_shard_4/16.pkg failed on release-24.2 @ 1d7bfd246e99dc0bf7b469cf97944f0930e027f6:

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-41841

renatolabs commented 1 week ago 
-- Test timed out at 2024-09-03 10:56:23 UTC --
Received SIGTERM, printing stack traces of all goroutines:

Test timed out. Issue creator can't find the owner in this case and assigns to test-eng, I'm reassigning to Queries in case they want to investigate.

michae2 commented 1 week ago

Looks like the timeout was in TestRaceWithBackfill based on a couple of stack traces:

goroutine 80553 [semacquire, 68 minutes]:
sync.runtime_Semacquire(0xc002880e68?)
        GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc002880e60)
        GOROOT/src/sync/waitgroup.go:116 +0xa5
github.com/cockroachdb/cockroach/pkg/sql_test.runSchemaChangeWithOperations(0xc005572680, 0xc0095700d0, 0xc00519d680, {{0x1359a630, 0x1359a650, {0x1}}, {0x1359a630}, 0x0}, {0xacaccc5, 0x29}, ...)
        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/schema_changer_test.go:403 +0xb3d
github.com/cockroachdb/cockroach/pkg/sql_test.TestRaceWithBackfill(0xc005572680)
        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/schema_changer_test.go:699 +0x1526
testing.tRunner(0xc005572680, 0xb04f750)
        GOROOT/src/testing/testing.go:1689 +0x21f
created by testing.(*T).Run in goroutine 1
        GOROOT/src/testing/testing.go:1742 +0x826

...

goroutine 84918 [select, 71 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0xebd7488, 0xc009533c80})
        github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/panic: test timed out after 1h16m37s
running tests:
        TestRaceWithBackfill (1h12m19s)

...

goroutine 127700 [IO wait, 69 minutes]:
internal/poll.runtime_pollWait(0x7fe4805abd30, 0x72)
        GOROOT/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc0059fb720, 0x72, 0x0)
        GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0059fb700, {0xc004d7a000, 0x700, 0x700})
        GOROOT/src/internal/poll/fd_unix.go:164 +0x466
net.(*netFD).Read(0xc0059fb700, {0xc004d7a000, 0x700, 0x700})
        GOROOT/src/net/fd_posix.go:55 +0x4b
net.(*conn).Read(0xc00570ed70, {0xc004d7a000, 0x700, 0x700})
        GOROOT/src/net/net.go:185 +0xad
crypto/tls.(*atLeastReader).Read(0xc01649e708, {0xc004d7a000, 0x700, 0x700})
        GOROOT/src/crypto/tls/conn.go:806 +0x89
bytes.(*Buffer).ReadFrom(0xc003e689b0, {0xeb87f00, 0xc01649e708})
        GOROOT/src/bytes/buffer.go:211 +0x113
crypto/tls.(*Conn).readFromUntil(0xc003e68708, {0xeb83b80, 0xc00570ed70}, 0x5)
        GOROOT/src/crypto/tls/conn.go:828 +0x1da
crypto/tls.(*Conn).readRecordOrCCS(0xc003e68708, 0x0)
        GOROOT/src/crypto/tls/conn.go:626 +0x365
crypto/tls.(*Conn).readRecord(...)
        GOROOT/src/crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0xc003e68708, {0xc0057c2000, 0x1000, 0x0?})
        GOROOT/src/crypto/tls/conn.go:1370 +0x2dd
bufio.(*Reader).Read(0xc010714de0, {0xc003351b28, 0x5, 0x200})
        GOROOT/src/bufio/bufio.go:241 +0x4bf
io.ReadAtLeast({0xeb7b9c0, 0xc010714de0}, {0xc003351b28, 0x5, 0x200}, 0x5)
        GOROOT/src/io/io.go:335 +0xd0
io.ReadFull(...)
        GOROOT/src/io/io.go:354
github.com/lib/pq.(*conn).recvMessage(0xc003351b08, 0xc004737568)
        github.com/lib/pq/external/com_github_lib_pq/conn.go:1004 +0x1ec
github.com/lib/pq.(*conn).recv1Buf(0xc003351b08, 0xc004737568)
        github.com/lib/pq/external/com_github_lib_pq/conn.go:1059 +0x35
github.com/lib/pq.(*conn).recv1(...)
        github.com/lib/pq/external/com_github_lib_pq/conn.go:1086
github.com/lib/pq.(*conn).simpleExec(0xc003351b08, {0xacaccc5, 0x29})
        github.com/lib/pq/external/com_github_lib_pq/conn.go:664 +0x2b1
github.com/lib/pq.(*conn).Exec(0xc003351b08, {0xacaccc5, 0x29}, {0x1478a500, 0x0, 0x0})
        github.com/lib/pq/external/com_github_lib_pq/conn.go:921 +0x331
github.com/lib/pq.(*conn).ExecContext(0xc003351b08, {0xebd4038, 0x1478a500}, {0xacaccc5, 0x29}, {0x1478a500, 0x0, 0x0?})
        github.com/lib/pq/external/com_github_lib_pq/conn_go18.go:46 +0x21d
database/sql.ctxDriverExec({0xebd4038, 0x1478a500}, {0x7fe494220d40, 0xc003351b08}, {0x0, 0x0}, {0xacaccc5, 0x29}, {0x1478a500, 0x0, ...})
        GOROOT/src/database/sql/ctxutil.go:31 +0xeb
database/sql.(*DB).execDC.func2()
        GOROOT/src/database/sql/sql.go:1703 +0x266
database/sql.withLock({0xeb9da68, 0xc0059117a0}, 0xc003e65bd8)
        GOROOT/src/database/sql/sql.go:3530 +0xa7
database/sql.(*DB).execDC(0x1?, {0xebd4038, 0x1478a500}, 0xc0059117a0, 0xc003e65cf0, {0xacaccc5, 0x29}, {0x0, 0x0, 0x0})
        GOROOT/src/database/sql/sql.go:1698 +0x405
database/sql.(*DB).exec(0xc0095700d0, {0xebd4038, 0x1478a500}, {0xacaccc5, 0x29}, {0x0, 0x0, 0x0}, 0x1)
        GOROOT/src/database/sql/sql.go:1683 +0x15e
database/sql.(*DB).ExecContext.func1(0x1)
        GOROOT/src/database/sql/sql.go:1662 +0xd6
database/sql.(*DB).retry(0xc0162894c0?, 0xc004737e70)
        GOROOT/src/database/sql/sql.go:1566 +0x4b
database/sql.(*DB).ExecContext(0xc0095700d0, {0xebd4038, 0x1478a500}, {0xacaccc5, 0x29}, {0x0, 0x0, 0x0})
        GOROOT/src/database/sql/sql.go:1661 +0x185
database/sql.(*DB).Exec(...)
        GOROOT/src/database/sql/sql.go:1675
github.com/cockroachdb/cockroach/pkg/sql_test.runSchemaChangeWithOperations.func1()
        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/schema_changer_test.go:340 +0xf1
created by github.com/cockroachdb/cockroach/pkg/sql_test.runSchemaChangeWithOperations in goroutine 80553
        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/schema_changer_test.go:337 +0x216
DrewKimball commented 4 days ago

The test was running with race=true, and we also have:

initialized metamorphic constant "kv-batch-size" with value 1