search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

roachtest: cdc/initial-scan-only/parquet/metamorphic failed #119295

Open cockroach-teamcity opened 7 months ago

cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ 7d0697b632066ee78735fc57e8150222d5576d0d:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240216110506&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:338
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240216110506&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-36133

wenyihu6 commented 7 months ago

The way that we are accessing the cloud storage bucket seems to be off here:https://github.com/cockroachdb/cockroach/blob/c8f84bf44b15922942a24e146d2c87af4173df31/pkg/cmd/roachtest/tests/cdc_helper.go#L243. I might hold off the investigation until next week though.

wenyihu6 commented 7 months ago

I wasn't able to repro this locally. TeamCity might be doing something different than what we expected here. We might need something similar to this https://github.com/cockroachdb/cockroach/blob/7c3d18ae4a3024ddf48a87e96098672334e1bebc/pkg/cmd/roachtest/tests/backup.go#L1004. 

15:04:39 test_runner.go:1173: running post test assertions (test-post-assertions.log)
15:04:39 test_runner.go:1186: running test teardown (test-teardown.log)
15:04:57 test_runner.go:982: metrics: https://go.crdb.dev/roachtest-grafana/wenyihu-1708095671/cdc-initial-scan-only-parquet-metamorphic/1708095801880/1708095999805
15:04:57 test_runner.go:1043: [w0] --- PASS: cdc/initial-scan-only/parquet/metamorphic (77.92s)
--- PASS: cdc/initial-scan-only/parquet/metamorphic (77.92s)
cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ 0b7ae19e2b94b851ed8812914f57032aab699811:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240217111147&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:338
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240217111147&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240218110824&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:338
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240218110824&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240219103403&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:338
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240219103403&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ a36097be277adef635f55d317579ca79b450bfef:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240220111339&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:338
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240220111339&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ e50b0ec4d3a53f81e26f3776ae3f3be55d435a9a:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240221110217&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:338
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240221110217&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.cdc/initial-scan-only/parquet/metamorphic failed with artifacts on master @ 04f0416d526a43741d22fd03966758dcccdeb79f:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:264
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                                main/pkg/cmd/roachtest/test_runner.go:1129
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240222111411&prettyPrint=false&projection=full&startOffset=&versions=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:354
                      | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).List
                      |     github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:423
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.listFilesOfTargetTable
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:101
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*metamorphicTestHelper).loadOpsToTableAndShowFingerprint
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:263
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.checkTwoChangeFeedExportContent
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_helper.go:311
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1206
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1129
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) unable to list files in gcs bucket
                    Wraps: (3) Get "https://storage.googleapis.com/storage/v1/b/cockroach-tmp/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachtest%2F20240222111411&prettyPrint=false&projection=full&startOffset=&versions=false"
                    Wraps: (4) metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write" not defined
                    Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) metadata.NotDefinedError
    Test:           cdc/initial-scan-only/parquet/metamorphic
(require.go:1360).NoError: FailNow called
(cdc.go:380).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan-only/parquet/metamorphic/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

rharding6373 commented 1 month ago

This is due to a permissions issue with the roachtest listing GCS files in GCE. I opened https://github.com/cockroachdb/cockroach/issues/127940 to track figuring out permissions and am opening a PR to unskip this test for other cloud providers so we get metamorphic test coverage back.

rharding6373 commented 1 month ago

Reopening because this test can only work in GCE, which means the test is skipped until the permissions issue is fixed.

rharding6373 commented 2 days ago

Downgraded this to P3 since it's an issue with the test infra and we're dealing with other priorities. The loss of test coverage due to skipping this test is unfortunate, but it's also been skipped longer than the test was running.