search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.74k stars 3.75k forks source link

roachtest: multitenant failed: use of Span after Finish #126979

Closed cockroach-teamcity closed 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on release-24.2 @ 495d20e82f09b3436f1b00d7331146fe2042dd6a:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Error message not equal:
                    expected: "pq: query execution canceled due to statement timeout"
                    actual  : "dial tcp 104.196.204.199:29000: connect: connection refused"
    Test:           multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126313 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master release-blocker]

This test on roachdash | Improve this report!

Jira issue: CRDB-40216

cockroach-teamcity commented 1 month ago

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on release-24.2 @ d6523e58ab3c09998535e88646ca16c1ead1ae4e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Error message not equal:
                    expected: "pq: query execution canceled due to statement timeout"
                    actual  : "dial tcp 34.139.74.251:29000: connect: connection refused"
    Test:           multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126313 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on release-24.2 @ 23edce4c9826ebd4711b71c1029d2fc38a25f974:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Error message not equal:
                    expected: "pq: query execution canceled due to statement timeout"
                    actual  : "dial tcp 35.237.225.5:29000: connect: connection refused"
    Test:           multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126313 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on release-24.2 @ d4ef0343f38a35fb781b6343697b57a58131c68e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Error message not equal:
                    expected: "pq: query execution canceled due to statement timeout"
                    actual  : "dial tcp 35.231.21.22:29000: connect: connection refused"
    Test:           multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126313 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 failed with artifacts on release-24.2 @ d4ef0343f38a35fb781b6343697b57a58131c68e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:193
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_distsql.go:47
                                main/pkg/cmd/roachtest/test_runner.go:1248
                                src/runtime/asm_amd64.s:1695
    Error:          Error message not equal:
                    expected: "pq: query execution canceled due to statement timeout"
                    actual  : "dial tcp 34.148.16.48:29000: connect: connection refused"
    Test:           multitenant/distsql/instances=20/bundle=on/timeout=1
(require.go:177).EqualError: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126313 roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master release-blocker]

This test on roachdash | Improve this report!

mgartner commented 1 month ago

From this failure above:

* ERROR: a panic has occurred!
* use of Span after Finish. Span: optimizer. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:770
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
*   | github.com/cockroachdb/cockroach/pkg/util/log.ExpensiveLogEnabled
*   |   github.com/cockroachdb/cockroach/pkg/util/log/log.go:64
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).sessionEventf
*   |   github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:4339
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).close
*   |   github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1247
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx.func1
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:236
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx.func2
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:257
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   |   src/runtime/asm_amd64.s:1695
* Wraps: (4) use of Span after Finish. Span: optimizer. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
*
* ERROR: a panic has occurred!
* use of Span after Finish. Span: prepare stmt. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:770
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.WithParent
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/span_options.go:212
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.EnsureChildSpan
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/tracer.go:1627
*   | github.com/cockroachdb/cockroach/pkg/sql.(*txnState).resetForNewSQLTxn
*   |   github.com/cockroachdb/cockroach/pkg/sql/txn_state.go:216
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).newConnExecutorWithTxn
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:461
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:343
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:226
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:1269
*   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIteratorEx
*   |   github.com/cockroachdb/cockroach/pkg/sql/internal.go:886
*   | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
*   |   github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache/cache.go:133
*   | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).Get.func1
*   |   github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache/cache.go:93
*   | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
*   |   github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:389
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320
*   | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
*   |   github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:388
*   | runtime.goexit
*   |   src/runtime/asm_amd64.s:1695
* Wraps: (4) use of Span after Finish. Span: prepare stmt. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
yuzefovich commented 1 month ago

Hm, I wonder whether the problem is this diff:

diff --git a/pkg/sql/syntheticprivilegecache/cache.go b/pkg/sql/syntheticprivilegecache/cache.go
index c098f6bc904..1ab1db3ed6f 100644
--- a/pkg/sql/syntheticprivilegecache/cache.go
+++ b/pkg/sql/syntheticprivilegecache/cache.go
@@ -90,7 +90,7 @@ func (c *Cache) Get(
        }
        privDesc, err := c.c.LoadValueOutsideOfCacheSingleFlight(ctx, fmt.Sprintf("%s-%d", spo.GetPath(), desc.GetVersion()),
                func(loadCtx context.Context) (_ interface{}, retErr error) {
-                       return c.readFromStorage(ctx, txn, spo)
+                       return c.readFromStorage(loadCtx, txn, spo)
                })
        if err != nil {
                return nil, err