cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

roachtest: cdc/mixed-versions Cleanup() is not invoked after the test times out #104313

Closed srosenberg closed 1 year ago

srosenberg commented 1 year ago

When cdc/mixed-versions times out, it's "stuck" inside a goroutine polling for waitForResolvedTimestamps. The cancelled context doesn't result in return (from the goroutine), thereby causing spurious messages in the (build) log,

I230601 13:35:05.510614 1859568 github.com/Shopify/sarama/sarama.go:129  [kafka-producer] 19549  client/metadata fetching metadata for all topics from broker 34.85.226.188:9092
I230601 13:35:35.511739 1900680 github.com/Shopify/sarama/broker.go:201  [kafka-producer] 19550  Failed to connect to broker 34.85.226.188:9092: dial tcp 34.85.226.188:9092: i/o timeout

The dangling goroutine after the test timed out (and cluster was destroyed),

1: chan receive [1642 minutes] [Created by main.(*testRunner).runTest @ test_runner.go:1035]
    runtime      proc.go:363              gopark(func(#2309), Pointer(0x0), waitReason(#10118))
    runtime      chan.go:583              chanrecv(*hchan(#4310), Pointer(#781))
    runtime      chan.go:447              chanrecv2(*hchan(#176), Pointer(#9183))
    tests        mixed_version_cdc.go:172 (*cdcMixedVersionTester).waitForResolvedTimestamps.func1({#8635, #9182}, {#101, #5991}, 1)
    tests        versionupgrade.go:178    (*versionUpgradeTest).run(#832, {#96, #1204}, {#101, #5991})
    tests        mixed_version_cdc.go:410 runCDCMixedVersions({#96, #1204}, {#101, #5991}, {#103, #258}, {0x17, 1, 0, {#138, ...}, ...})
    tests        mixed_version_cdc.go:76  registerCDCMixedVersions.func1({#96, #1204}, {#101, #5991}, {#103, #258})
    main         test_runner.go:1050      (*testRunner).runTest.func2()
    runtime      asm_amd64.s:1594         goexit()

Jira issue: CRDB-28455

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/test-eng