search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

ccl/backupccl: TestBackupExportRequestTimeout failed #131485

Open cockroach-teamcity opened 2 months ago

cockroach-teamcity commented 2 months ago

ccl/backupccl.TestBackupExportRequestTimeout failed on release-24.2 @ e35a012c0e04d2010be362e2ef6c02686e78352a:

        testing.runTests(0xc00465a828, {0xc00464ae40, 0x6, 0x8}, {0xcba2fc0?, 0x0?, 0xcba2fc0?})
            GOROOT/src/testing/testing.go:2159 +0x445
        testing.(*M).Run(0xc004641680)
            GOROOT/src/testing/testing.go:2027 +0x68b
        github.com/cockroachdb/cockroach/pkg/ccl/backupccl_test.TestMain(0xc004641680)
            github.com/cockroachdb/cockroach/pkg/ccl/backupccl_test/pkg/ccl/backupccl/main_test.go:31 +0x158
        main.main()
            main/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/testmain.go:738 +0x4d7
        goroutine 19 [select]:
        go.opencensus.io/stats/view.(*worker).start(0xc0011b6000)
            go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:292 +0x9f
        created by go.opencensus.io/stats/view.init.0 in goroutine 1
            go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:34 +0x8d
        goroutine 31 [chan receive]:
        github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
            github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:172 +0xa8
        created by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
            github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:164 +0x1a
        goroutine 68 [syscall]:
        os/signal.signal_recv()
            GOROOT/src/runtime/sigqueue.go:152 +0x29
        os/signal.loop()
            GOROOT/src/os/signal/signal_unix.go:23 +0x13
        created by os/signal.Notify.func1.1 in goroutine 89
            GOROOT/src/os/signal/signal.go:151 +0x1f
        goroutine 88 [chan receive]:
        github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
            github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:128 +0x5e
        created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
            github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:83 +0x1a
        goroutine 89 [chan receive]:
        github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
            github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:152 +0x89
        created by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
            github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:84 +0x26
        goroutine 3380853 [chan receive]:
        github.com/cockroachdb/pebble/sstable.(*writeQueue).runWorker(0xc0201cbe40)
            github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/write_queue.go:95 +0x46
        created by github.com/cockroachdb/pebble/sstable.newWriteQueue in goroutine 3377601
            github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/write_queue.go:57 +0xc9
        goroutine 18442 [chan receive]:
        github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler.func1()
            github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:34 +0x1d
        created by github.com/bazelbuild/rules_go/go/tools/bzltestutil.RegisterTimeoutHandler in goroutine 1
            github.com/bazelbuild/rules_go/go/tools/bzltestutil/external/io_bazel_rules_go/go/tools/bzltestutil/timeout.go:33 +0x98: Leaked goroutine: goroutine 3380853 [chan receive]:
        github.com/cockroachdb/pebble/sstable.(*writeQueue).runWorker(0xc0201cbe40)
            github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/write_queue.go:95 +0x46
        created by github.com/cockroachdb/pebble/sstable.newWriteQueue in goroutine 3377601
            github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/write_queue.go:57 +0xc9
--- FAIL: TestBackupExportRequestTimeout (20.70s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-42565

dt commented 1 month ago

Looks like a missing Close() on one of our writers but hard to reproduce even in 1k runs. Marking as p3 until it recurs.