search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.87k stars 3.77k forks source link

cli: TestTenantZip failed #128786

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

cli.TestTenantZip failed on release-24.1.3-rc @ 9e10212477fde97f55ea4fff01797288c836575c:

=== RUN   TestTenantZip
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantZip1798464953
    test_log_scope.go:81: use -show-logs to present logs inline
    zip_tenant_test.go:116: -- test log scope end --
test logs left over in: outputs.zip/logTestTenantZip1798464953
--- FAIL: TestTenantZip (83.15s)
=== RUN   TestTenantZip/testzip_shared_process_virtualization
[debug --host=127.47.215.28:35901 --insecure=true zip --concurrency=1 --cpu-profile-duration=1s /dev/null]
    datadriven.go:144: 
        /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/cli/cli_test_/cli_test.runfiles/com_github_cockroachdb_cockroach/pkg/cli/testdata/zip/testzip_shared_process_virtualization:1:

        output didn't match expected:
        @@ -224,11 +224,13 @@
         [cluster] requesting tenant ranges: last request failed: rpc error: ...
         [cluster] requesting tenant ranges: creating error output: debug/cluster/test-tenant/tenant_ranges.err.txt... done
         [cluster] collecting the inflight traces for jobs... received response... done
         [cluster] requesting CPU profiles
         [cluster] profiles generated
        -[cluster] profile for node 1... writing binary output: debug/cluster/test-tenant/nodes/1/cpu.pprof... done
        +[cluster] profile for node 1...
        +[cluster] profile for node 1: last request failed: rpc error: ...
        +[cluster] profile for node 1: creating error output: debug/cluster/test-tenant/nodes/1/cpu.pprof.err.txt... done
         [node 1] node status... writing JSON output: debug/cluster/test-tenant/nodes/1/status.json... done
         [node 1] using SQL connection URL: postgresql://...
         [node 1] retrieving SQL data for crdb_internal.active_range_feeds... writing output: debug/cluster/test-tenant/nodes/1/crdb_internal.active_range_feeds.txt... done
         [node 1] retrieving SQL data for crdb_internal.feature_usage... writing output: debug/cluster/test-tenant/nodes/1/crdb_internal.feature_usage.txt... done
         [node 1] retrieving SQL data for crdb_internal.gossip_alerts... writing output: debug/cluster/test-tenant/nodes/1/crdb_internal.gossip_alerts.txt...
    --- FAIL: TestTenantZip/testzip_shared_process_virtualization (40.43s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/obs-prs @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-41205

dhartunian commented 1 month ago

Looks like there might be an issue with concurrent profiles being taken. This is from the clitest.log file from the run:

I240812 08:29:29.061632 1179562 server/profiler/cpuprofiler.go:132 ⋮ [T1,Vsystem,n1] 588  taking CPU profile for 10.00s
E240812 08:29:29.540146 2845617 server/status_local_file_retrieval.go:53 ⋮ [T10,Vtest-tenant,n1,status,tenant=10] 589  ‹cpu profiling already in use›
E240812 08:29:29.540146 2845617 server/status_local_file_retrieval.go:53 ⋮ [T10,Vtest-tenant,n1,status,tenant=10] 589 +(1) ‹cpu profiling already in use›
E240812 08:29:29.540146 2845617 server/status_local_file_retrieval.go:53 ⋮ [T10,Vtest-tenant,n1,status,tenant=10] 589 +Error types: (1) *errors.errorString