cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed #56230

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379, but there still some different errors left to solve in the comments

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@e25ee7ad4bcdeda49f40eb52c24d2d1b1aa67a86:

          | I201103 08:13:21.524996 1 workload/cli/run.go:356  creating load generator...
          | I201103 08:13:21.597243 1 workload/cli/run.go:387  creating load generator... done (took 72.246217ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=24fa1aae key=/Table/SystemConfigSpan/Start pri=0.05040495 epo=0 ts=1604391209.856504541,1 min=1604391209.226930126,0 seq=6} lock=true stat=PENDING rts=1604391209.226930126,0 wto=false max=1604391209.226930126,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | opOk
          |     8.0s        0            3.0            6.5    771.8    872.4    872.4    872.4 txnCmtErr
          |     8.0s        0            1.0            3.1     16.3     16.3     16.3     16.3 txnOk
          |     8.0s        0           25.0           20.1      2.4    268.4    469.8    469.8 txnRbk
          | BEGIN
          |   ALTER TABLE public.table309 DROP COLUMN "col309_368";
          | ROLLBACK; ERROR: column "col309_368" does not exist (SQLSTATE 42703)
          | BEGIN
          |   ALTER TABLE public.table309 ALTER COLUMN "col303_1" SET DATA TYPE public.enum370;
          | ROLLBACK; ERROR: type "public.enum370" does not exist (SQLSTATE 42704)
          | BEGIN
          |   ALTER TABLE public.table309 DROP CONSTRAINT "table303_col303_15_col303_17_col303_8_col303_13_col303_18_col303_10_col303_14_key";
          | ROLLBACK; ERROR: unimplemented: cannot drop UNIQUE constraint "table303_col303_15_col303_17_col303_8_col303_13_col303_18_col303_10_col303_14_key" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table309 ALTER COLUMN "col303_3" SET DATA TYPE varchar;
          | ROLLBACK; ERROR: ALTER COLUMN TYPE from float4 to varchar is only supported experimentally (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table309 ALTER COLUMN "col303_16" SET DATA TYPE "char";
          | ROLLBACK; ERROR: ALTER COLUMN TYPE from timestamp to "char" is only supported experimentally (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table309 DROP COLUMN "col303_16";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=3fa1992f key=/Table/SystemConfigSpan/Start pri=0.01312331 epo=0 ts=1604391209.856504541,1 min=1604391209.458920462,0 seq=7} lock=true stat=PENDING rts=1604391209.458920462,0 wto=false max=1604391209.458920462,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table309 RENAME TO public.table369;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=9dc0ce92 key=/Table/SystemConfigSpan/Start pri=0.00709184 epo=0 ts=1604391209.856504541,1 min=1604391209.302654671,0 seq=6} lock=true stat=PENDING rts=1604391209.302654671,0 wto=false max=1604391209.302654671,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP TABLE public.table309;
          | ROLLBACK; ERROR: restart transaction: failed to mark job 603982425192792065 as as successful: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=9be48adb key=/Table/SystemConfigSpan/Start pri=0.01119124 epo=0 ts=1604391209.863562534,1 min=1604391209.523850704,0 seq=5} lock=true stat=PENDING rts=1604391209.523850704,0 wto=false max=1604391209.523850704,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE INDEX IF NOT EXISTS index309_372 ON public.table309 (col303_9 DESC, col303_14 ASC, col303_17, col303_3 DESC, col303_2 DESC, col303_10, col303_12 ASC, col303_5, col303_13 ASC, col303_18, col303_6, col303_0 ASC, col303_7 DESC, col303_1 DESC, col303_11 DESC, col303_16, col303_15 DESC, col303_8 ASC);
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=26768ce1 key=/Table/SystemConfigSpan/Start pri=0.02456629 epo=0 ts=1604391209.856504541,1 min=1604391209.422275976,0 seq=6} lock=true stat=PENDING rts=1604391209.422275976,0 wto=false max=1604391209.422275976,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table309 ADD COLUMN col309_371 TIME NOT NULL;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=24fa1aae key=/Table/SystemConfigSpan/Start pri=0.05040495 epo=0 ts=1604391209.856504541,1 min=1604391209.226930126,0 seq=6} lock=true stat=PENDING rts=1604391209.226930126,0 wto=false max=1604391209.226930126,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2412951&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

Jira issue: CRDB-2965

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57487

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@25edeb8d6c2cefca0c206f86553bc75569f0f2d9:

        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2416377-1604473417-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201104 08:11:54.107946 1 workload/cli/run.go:356  creating load generator...
          | I201104 08:11:54.191787 1 workload/cli/run.go:387  creating load generator... done (took 83.840708ms)
          | Error: ***UNEXPECTED ERROR: ERROR: descriptor is being dropped (SQLSTATE XXUUU)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ), UNIQUE (col313_0 ASC), UNIQUE (col313_0 DESC), INDEX (col313_0), UNIQUE (col313_0 ASC));
          |   ALTER SEQUENCE "seq314" RENAME TO "seq315";
          | ROLLBACK; ERROR: relation "seq314" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table309 (col309_0 TIMETZ NOT NULL, col309_1 DECIMAL NULL, col309_2 BYTES NOT NULL, col309_3 CHAR NOT NULL, col309_4 DATE NOT NULL, col309_5 FLOAT4 NOT NULL, col309_6 REGPROCEDURE NOT NULL, col309_7 TIMESTAMP NOT NULL, col309_8 BIT(32) NOT NULL, col309_9 DECIMAL NOT NULL, col309_10 CHAR, col309_11 INET NOT NULL, col309_12 BIT(9) NOT NULL, col309_13 REGTYPE NOT NULL, col309_14 BOX2D NULL, PRIMARY KEY (col309_13, col309_0 ASC, col309_7 DESC, col309_2 DESC, col309_6 ASC, col309_4 ASC, col309_12 DESC, col309_9 DESC, col309_3, col309_8 ASC, col309_11, col309_5 DESC), UNIQUE (col309_11 DESC, col309_8 ASC, col309_2 ASC, col309_5, col309_14, col309_7 DESC, col309_4, col309_10 ASC, col309_1 ASC, col309_13 ASC, col309_0 ASC, col309_6 DESC, col309_3 ASC, col309_12), UNIQUE (col309_10 ASC, col309_2 ASC, col309_11, col309_8 ASC, col309_12 DESC, col309_4, col309_14 ASC, col309_0 ASC), INDEX (col309_5 ASC, col309_7 ASC, col309_8 ASC, col309_14 ASC, col309_0, col309_6 ASC, col309_4 DESC, col309_1 DESC, col309_2 ASC, col309_11, col309_9 DESC, col309_13 ASC) STORING (col309_10), UNIQUE (col309_8, col309_12 DESC, col309_2 ASC, col309_3 ASC, col309_13 ASC, col309_7 ASC, col309_9 DESC, col309_10 ASC, col309_1 DESC, col309_4 ASC), INDEX (col309_5, col309_11 DESC, col309_9, col309_8, col309_0 ASC, col309_7 DESC, col309_6 ASC, col309_3 ASC, col309_13 ASC, col309_10 ASC, col309_1 DESC, col309_2 ASC, col309_4 ASC), INDEX (col309_0, col309_2 DESC, col309_1 ASC, col309_5 DESC, col309_6, col309_11 DESC, col309_9, col309_10 DESC, col309_8 ASC, col309_14 ASC, col309_12 DESC), INDEX (col309_3, col309_10, col309_8 ASC, col309_11, col309_1 DESC) STORING (col309_14));
          |   ALTER TABLE public.table309 DROP COLUMN "col309_3";
          | ROLLBACK; ERROR: column "col309_3" is referenced by the primary key (SQLSTATE 42P10)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA schema308 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq319";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema310 AUTHORIZATION root;
          |   ALTER TABLE schema35.table320 ADD COLUMN IrrelevantColumnName string;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "schema35.table320" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE TYPE "public.enum300" AS ENUM ('sddaf');
          |   DROP SCHEMA "schema35" CASCADE;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=04257279 key=/Table/SystemConfigSpan/Start pri=0.02789539 epo=0 ts=1604477519.987135548,1 min=1604477519.127247963,0 seq=26} lock=true stat=PENDING rts=1604477519.127247963,0 wto=false max=1604477519.627247963,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER SEQUENCE "seq208" RENAME TO "seq316";
          |   CREATE SCHEMA IF NOT EXISTS schema317 AUTHORIZATION root;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE "seq325";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TABLE schema35.table312 (col312_0 BOX2D, col312_1 BIT(25), col312_2 FLOAT4 NOT NULL, col312_3 UUID NOT NULL, col312_4 INT8 NULL, col312_5 FLOAT8 NOT NULL, col312_6 BIT(38) NOT NULL, col312_7 INT8 NOT NULL, col312_8 DATE NOT NULL, col312_9 DATE NOT NULL, col312_10 INT8 NOT NULL, col312_11 REGTYPE NULL, col312_12 OID NULL, col312_13 UUID NULL, col312_14 VARCHAR NOT NULL, col312_15 BIT(4)[], col312_16 REGPROCEDURE NULL, col312_17 TIMETZ NOT NULL, col312_18 VARCHAR NOT NULL, PRIMARY KEY (col312_14, col312_3, col312_17 DESC, col312_18, col312_2 ASC, col312_8 DESC));
          | ***FAIL; Expected no errors, but got ERROR: descriptor is being dropped (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2416377&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56589

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@c456295f7d09fbd319509ae4b08fd63dae248ddf:

          | Error: Class XX - Internal Error: ERROR: cannot update progress on succeeded job (id 604548613627740161) (SQLSTATE XXUUU)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE public.table28 ALTER COLUMN "col28_5" SET NOT NULL;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=1136e79d key=/Table/SystemConfigSpan/Start pri=0.01573702 epo=0 ts=1604563995.450850602,1 min=1604563994.821395571,0 seq=11} lock=true stat=PENDING rts=1604563994.821395571,0 wto=false max=1604563995.321395571,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table28 ALTER COLUMN "col28_10" DROP DEFAULT;
          |   CREATE SEQUENCE "seq98";
          |   CREATE INDEX IF NOT EXISTS index28_99 ON public.table28 (col28_7 ASC, col28_9 ASC, col28_1 ASC, col28_18) STORING (col28_10, col28_3, col28_16, col28_14, col28_12, col28_11, col28_6, col28_2, col28_5, col28_17, col28_4, col28_13, col28_0, col28_15);
          | ROLLBACK; ERROR: index "index28_99" already contains column "col28_17" (SQLSTATE 42701)
          | BEGIN
          |   ALTER TABLE public.table28 RENAME TO public.table72;
          |   ALTER TABLE public.table72 RENAME CONSTRAINT "table28_col28_10_col28_7_col28_6_col28_13_col28_12_col28_0_col28_3_col28_4_col28_5_col28_15_col28_17_col28_14_col28_8_col28_18_col28_16_col28_9_col28_2_col28_1_col28_11_idx" TO "table28_col28_4_col28_9_col28_7_col28_8_col28_16_col28_0_col28_3_col28_11_col28_2_idx";
          | ROLLBACK; ERROR: constraint "table28_col28_10_col28_7_col28_6_col28_13_col28_12_col28_0_col28_3_col28_4_col28_5_col28_15_col28_17_col28_14_col28_8_col28_18_col28_16_col28_9_col28_2_col28_1_col28_11_idx" does not exist (SQLSTATE 42704)
          | BEGIN
          |   DROP INDEX public.table28@"table28_col28_15_col28_16_col28_12_col28_18_col28_7_col28_14_col28_17_col28_11_col28_4_col28_3_col28_10_col28_9_col28_8_idx";
          |   ALTER TABLE public.table28 ALTER COLUMN "col28_2" DROP DEFAULT;
          |   CREATE SEQUENCE "seq101";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=9c32abc6 key=/Table/SystemConfigSpan/Start pri=0.04971134 epo=0 ts=1604563995.450850602,1 min=1604563995.017638645,0 seq=28} lock=true stat=PENDING rts=1604563995.017638645,0 wto=false max=1604563995.517638645,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP TABLE public.table28;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=679a5347 key=/Table/SystemConfigSpan/Start pri=0.02126358 epo=0 ts=1604563995.450850602,1 min=1604563995.190366523,0 seq=11} lock=true stat=PENDING rts=1604563995.190366523,0 wto=false max=1604563995.690366523,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table28 (col28_0 INT4 NOT NULL, PRIMARY KEY (col28_0 DESC), INDEX (col28_0), INDEX (col28_0 DESC), UNIQUE (col28_0), INDEX (col28_0 ASC), UNIQUE (col28_0), UNIQUE (col28_0 ASC), INDEX (col28_0 ASC), INDEX (col28_0 ASC), UNIQUE (col28_0));
          |   ALTER TABLE public.table28 ALTER COLUMN "col28_9" DROP NOT NULL;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=4541dfe0 key=/Table/SystemConfigSpan/Start pri=0.00141882 epo=0 ts=1604563995.450850602,1 min=1604563995.201565573,0 seq=11} lock=true stat=PENDING rts=1604563995.201565573,0 wto=false max=1604563995.701565573,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table28 RENAME TO public.table90;
          |   ALTER TABLE public.table90 RENAME TO public.table102;
          |   CREATE SCHEMA IF NOT EXISTS schema103 AUTHORIZATION root;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE public.table28 RENAME TO public.table92;
          |   ALTER TABLE public.table92 ALTER COLUMN "col28_17" SET NOT NULL;
          |   ALTER TABLE public.table92 ALTER COLUMN "col28_16" DROP NOT NULL;
          |   DROP TABLE public.table92;
          | ROLLBACK; ERROR: cannot update progress on succeeded job (id 604548613627740161) (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2418892&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56184

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@37653515b3b6a612b9ee71c4f3bcfb9511805edb:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2204
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:129
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:73
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1357
        Wraps: (2) output in run_081238.912_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2421874-1604646259-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201106 08:12:39.752304 1 workload/cli/run.go:356  creating load generator...
          | I201106 08:12:39.833562 1 workload/cli/run.go:387  creating load generator... done (took 81.257327ms)
          | Error: ***UNEXPECTED ERROR: ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=5eaa9460 key=/Table/SystemConfigSpan/Start pri=0.01268103 epo=0 ts=1604650362.093399832,1 min=1604650361.951909917,0 seq=12} lock=true stat=PENDING rts=1604650361.951909917,0 wto=false max=1604650362.451909917,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table123 ALTER COLUMN "col123_9" DROP STORED;
          | ROLLBACK; ERROR: column "col123_9" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   ALTER TABLE public.table123 ALTER COLUMN "col123_12" DROP STORED;
          | ROLLBACK; ERROR: column "col123_12" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP INDEX public.table123@"primary";
          | ROLLBACK; ERROR: index "primary" is in use as unique constraint (SQLSTATE 2BP01)
          | BEGIN
          |   ALTER TABLE public.table123 RENAME COLUMN "col123_10" TO "col123_131";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=ff09e3cc key=/Table/SystemConfigSpan/Start pri=0.00660834 epo=0 ts=1604650362.093399832,1 min=1604650361.795957781,0 seq=11} lock=true stat=PENDING rts=1604650361.795957781,0 wto=false max=1604650362.295957781,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table123 RENAME COLUMN "col123_18" TO "col123_138";
          |   CREATE SEQUENCE "seq140";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   INSERT INTO public.table123 ("col123_0","col123_1","col123_2","col123_3","col123_4","col123_5","col123_6","col123_7","col123_8","col123_9","col123_10","col123_11","col123_12","col123_13","col123_14","col123_15","col123_16","col123_17","col123_18") VALUES ('1972-07-07':::DATE,'5e98b38a-4d87-457b-a243-ec5e04a9010f':::UUID,'-51 years -3 mons -423 days -03:47:22.342061':::INTERVAL,'1970-01-01 09:51:40.000792+00:00':::TIMESTAMPTZ,4002114157:::OID,533890969:::OID,'\x':::BYTES,'c596ad06-c9ae-4f25-b7c8-e76416d0bc48':::UUID,'1970-01-07 00:16:43.000425+00:00':::TIMESTAMPTZ,(-0.40209299325942993):::FLOAT8,'mv^&cXi':::STRING:::NAME,'15:17:49.87512-11:23:00':::TIMETZ,ARRAY[B'01000010001',B'11001110110',B'10001001111',B'00000111000'],2040721416:::OID,e'\U00002603':::STRING,(-8.732072646586586269E+29):::DECIMAL,true,(-8693):::INT8,'1977-02-02':::DATE),(NULL,'cf5e24bd-344d-44f1-b52a-79eb41cd247c':::UUID,'36 years 2 mons 209 days 10:57:59.28392':::INTERVAL,'1970-01-11 21:16:46.000724+00:00':::TIMESTAMPTZ,1085255249:::OID,2141302079:::OID,'\x9542':::BYTES,NULL,'1970-01-10 16:40:34.0006+00:00':::TIMESTAMPTZ,(-1.6990517377853394):::FLOAT8,e'k^\x03':::STRING:::NAME,'06:42:18.417367+03:56:00':::TIMETZ,ARRAY[]:::BIT(11)[],3746911174:::OID,e'(3oN\x10':::STRING,5.688317733873216594E+26:::DECIMAL,false,23329:::INT8,'1976-12-13':::DATE),('1981-11-20':::DATE,'1bc6e685-4425-4029-8a22-2fdce4e4d410':::UUID,'-60 years -2 mons -501 days -12:30:28.498913':::INTERVAL,'1970-01-05 18:34:04.000249+00:00':::TIMESTAMPTZ,451380244:::OID,3302911131:::OID,'\xf8f18d':::BYTES,'32c4e06d-ccad-451c-8d4b-71ea3c7bd435':::UUID,'1970-01-11 03:25:44.000544+00:00':::TIMESTAMPTZ,(-1.3676515817642212):::FLOAT8,e'ygmhf\x079':::STRING:::NAME,'17:57:53.19268+05:07:00':::TIMETZ,ARRAY[B'01010101111',B'11100000111',B'11100001100',B'11001111011'],175222870:::OID,'Z':::STRING,(-8.406171748560452756E+37):::DECIMAL,true,(-5919):::INT8,'1990-03-27':::DATE);
          |   CREATE TABLE public.table141 (col141_0 REGNAMESPACE, col141_1 INET, col141_2 FLOAT8, col141_3 REGPROC NULL, col141_4 BOOL NULL, col141_5 DATE NOT NULL, col141_6 NAME NOT NULL, col141_7 BYTES NOT NULL, col141_8 BOOL NOT NULL, col141_9 DATE, col141_10 FLOAT8 NULL, col141_11 INT2, col141_12 TIMESTAMP NOT NULL, col141_13 UUID, PRIMARY KEY (col141_6, col141_12 DESC, col141_8, col141_7 ASC, col141_5 ASC));
          | ***FAIL; Expected no errors, but got ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2421874&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

jayshrivastava commented 3 years ago

The schemachange after write error will be anticipated and handled after https://github.com/cockroachdb/cockroach/pull/56184 is merged in

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56589

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@61c96aaca632dfba5154d55d82c5af0732053b72:

          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1357
        Wraps: (2) output in run_075908.399_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2424980-1604732977-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201107 07:59:09.238960 1 workload/cli/run.go:356  creating load generator...
          | I201107 07:59:09.313059 1 workload/cli/run.go:387  creating load generator... done (took 74.097526ms)
          | Error: Class XX - Internal Error: ERROR: cannot update progress on succeeded job (id 605112094097670145) (SQLSTATE XXUUU)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | l325_6" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   ALTER TABLE schema248.table325 DROP COLUMN "col325_3";
          |   INSERT INTO schema248.table325 ("col325_0","col325_1","col325_2","col325_4","col325_5","col325_6","col325_7","col325_8","col325_9","col325_10") VALUES ('-4 years -5 mons -771 days -00:07:14.008593':::INTERVAL,ARRAY['f926bb5c-4bd8-4fce-a53d-f73d60f2f942':::UUID,'f9c77e97-9f64-4434-a5eb-642a5f523334':::UUID,'3bf01e4c-87b7-4e64-ab6f-e2f8b2190acf':::UUID],'\xff':::BYTES,'eed8c010-a259-46e6-ab18-79f9b3114c62':::UUID,NULL,'rK':::STRING,4030631451:::OID,384850785:::OID,0.04850411042571068:::FLOAT8,e'\x012\x15"^\x1bGbC':::STRING),('29 years 2 mons 781 days 22:44:51.139093':::INTERVAL,ARRAY['d4019b97-18d2-4f10-bb1d-f958bbe08514':::UUID,'100dbf07-abd8-4cca-8163-5408d3523d16':::UUID,'aacf9e48-97cf-406e-a346-251c95f8b63d':::UUID],'\x6f94eded9cf79ad9c2':::BYTES,'098cf97c-b68e-404e-9fa6-a2a2328719e3':::UUID,(-2.0209927558898926):::FLOAT8,'Fthag':::STRING,599089529:::OID,586997382:::OID,(-1.3184703588485718):::FLOAT8,'6':::STRING),('-40 years -11 mons -198 days -10:33:47.251684':::INTERVAL,ARRAY['9037abbf-3569-4c04-bb57-13a2af0bebf8':::UUID],'\xe36cb0f3a9921c':::BYTES,'f591e819-00c9-4ff4-99ff-b7b700854bf8':::UUID,0.0:::FLOAT8,'':::STRING,582828706:::OID,862402824:::OID,NULL,e'y!-\x06':::STRING),('-8 years -3 mons -871 days -16:33:19.638564':::INTERVAL,ARRAY['230ec623-38fc-4c1b-bdc9-30601a6910a9':::UUID,'422fd501-9169-4f07-858c-c4aaa2cafa53':::UUID,'5070c5e0-5f91-43f8-8123-cb99efa25511':::UUID],'\xcc23':::BYTES,'47410580-8165-4b83-8f2a-3e8a7ced2d14':::UUID,(-2.0312342643737793):::FLOAT8,e'.h\x10\rS':::STRING,4095793506:::OID,1257004777:::OID,(-0.07879102975130081):::FLOAT8,NULL),('-78 years -3 mons -297 days -07:53:34.758364':::INTERVAL,ARRAY['fc4fe340-d6b1-4cb4-8208-b3a4f1753fa6':::UUID,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID],'\xa948c21a495b':::BYTES,'f3a173c4-cc8d-47d7-9cfa-ad8f77091ba7':::UUID,1.2738229036331177:::FLOAT8,e'\x12D7\\;P\t(':::STRING,1114288813:::OID,0:::OID,(-0.30312103033065796):::FLOAT8,'"':::STRING);
          | ROLLBACK; ERROR: value too long for type CHAR (column "col325_6") (SQLSTATE 22001)
          | BEGIN
          |   ALTER TABLE schema248.table325 ALTER COLUMN "col325_6" SET NOT NULL;
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=a3770841 key=/Table/SystemConfigSpan/Start pri=0.00595599 epo=0 ts=1604735955.711391343,1 min=1604735955.469525271,0 seq=11} lock=true stat=PENDING rts=1604735955.469525271,0 wto=false max=1604735955.969525271,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE schema248.table325 RENAME TO schema248.table379;
          |   ALTER TABLE schema248.table379 RENAME COLUMN "col379_385" TO "col379_386";
          | ROLLBACK; ERROR: column "col379_385" does not exist (SQLSTATE 42703)
          |     7.0s        0           30.0           41.3     18.9    486.5    503.3    503.3 opOk
          |     7.0s        0            2.0            7.6    335.5    771.8    771.8    771.8 txnCmtErr
          |     7.0s        0            1.0            3.1   1610.6   1610.6   1610.6   1610.6 txnOk
          |     7.0s        0           28.0           24.3      2.1     11.0     12.6     12.6 txnRbk
          | BEGIN
          |   DROP SEQUENCE "seq308";
          |   DROP INDEX schema248.table325@"table325_col325_6_col325_10_col325_4_col325_0_idx";
          |   CREATE TABLE schema248.table325 AS SELECT col325_4, col325_1, col325_2, col325_7, col325_0, col325_6, col325_10, col325_3, col325_8, col325_6, col325_7, col325_5 FROM schema248.table325, schema248.table325, schema248.table325;
          | ROLLBACK; expected SQLSTATE(S) 42712,42P07, and got ERROR: source name "table325" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE schema248.table325 ALTER COLUMN "col325_388" SET NOT NULL;
          | ROLLBACK; ERROR: column "col325_388" does not exist (SQLSTATE 42703)
          | BEGIN
          |   ALTER TABLE schema248.table325 ALTER COLUMN "col325_2" SET NOT NULL;
          |   DROP TABLE schema248.table325;
          | ROLLBACK; ERROR: cannot update progress on succeeded job (id 605112094097670145) (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2424980&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56184

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@be84ae4a59e6a5e50694c1756e05b0ba7c3a54d8:

          | stderr:
          | I201108 08:13:34.212333 1 workload/cli/run.go:356  creating load generator...
          | I201108 08:13:34.304991 1 workload/cli/run.go:387  creating load generator... done (took 92.656955ms)
          | Error: ***UNEXPECTED ERROR: ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ;
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA IF NOT EXISTS schema59 AUTHORIZATION root;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA schema43 AUTHORIZATION root;
          |   CREATE SCHEMA IF NOT EXISTS schema60 AUTHORIZATION root;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table45 (col45_0 INT2 NOT NULL, col45_1 INT2 NOT NULL, col45_2 REGCLASS NOT NULL, col45_3 INTERVAL NOT NULL, col45_4 FLOAT4 NOT NULL, col45_5 TIME, col45_6 OID NULL, col45_7 BIT(7) NOT NULL, col45_8 REGTYPE, col45_9 FLOAT4 NULL, col45_10 FLOAT4, col45_11 REGPROC, col45_12 FLOAT4 NOT NULL, col45_13 "char" NULL, PRIMARY KEY (col45_1 DESC, col45_2 DESC), INDEX (col45_4, col45_6 DESC, col45_12, col45_5, col45_10 DESC) STORING (col45_0, col45_3, col45_7, col45_8), UNIQUE (col45_13 ASC, col45_7 DESC, col45_2 ASC, col45_1 DESC, col45_11 DESC, col45_8 ASC, col45_9, col45_10, col45_5 DESC, col45_6, col45_3 DESC, col45_0, col45_4, col45_12), UNIQUE (col45_7, col45_1 DESC, col45_10 ASC, col45_13, col45_5 DESC, col45_11 ASC, col45_2 ASC, col45_8 ASC, col45_12 DESC, col45_0 DESC), INDEX (col45_8 DESC, col45_4, col45_9 DESC, col45_13 DESC, col45_12 ASC, col45_1 DESC, col45_5 ASC, col45_0 DESC, col45_3) STORING (col45_6, col45_7, col45_10, col45_11), UNIQUE (col45_0, col45_5 DESC, col45_6 ASC, col45_8 ASC, col45_3 ASC, col45_2 DESC, col45_12 ASC, col45_9 DESC, col45_1) STORING (col45_4, col45_11), UNIQUE (col45_8 ASC, col45_7 DESC, col45_4 DESC, col45_10 ASC, col45_3 DESC, col45_9 ASC, col45_0, col45_1, col45_11 DESC, col45_6 DESC, col45_13, col45_12 DESC, col45_5 DESC, col45_2 ASC));
          | COMMIT;  
          | BEGIN
          |   DROP SEQUENCE "seq19";
          |   CREATE SEQUENCE "seq62";
          |   ALTER TABLE public.table64 ADD COLUMN IrrelevantColumnName string;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table64" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE INDEX index58_65 ON schema3.table58 (col58_3 ASC, col58_0 DESC) STORING (col58_7, col58_2, col58_1, col58_6, col58_5);
          | ROLLBACK; ERROR: index "index58_65" already contains column "col58_2" (SQLSTATE 42701)
          | BEGIN
          |   ALTER TABLE schema3.table58 ALTER COLUMN "col58_67" SET NOT NULL;
          | ROLLBACK; ERROR: column "col58_67" does not exist (SQLSTATE 42703)
          | BEGIN
          |   DROP SCHEMA "schema3" CASCADE;
          |   CREATE SEQUENCE "seq68";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER SEQUENCE "seq73" RENAME TO "seq74";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   INSERT INTO schema3.table58 ("col58_0","col58_1","col58_2","col58_3","col58_4","col58_5","col58_6","col58_7") VALUES (1550797638:::OID,1.429585860844619:::FLOAT8,3538062852:::OID,'23:22:59.00352':::TIME,1615876049:::OID,9327:::INT8,2243354635:::OID,71770754:::OID),(2759577239:::OID,1.6081506693282912:::FLOAT8,319970030:::OID,'00:00:00':::TIME,2303151568:::OID,(-1):::INT8,1476157203:::OID,0:::OID);
          |   CREATE TABLE schema3.table72 (col72_0 TIME NOT NULL, col72_1 NAME NOT NULL, col72_2 UUID NOT NULL, col72_3 BIT(18), col72_4 FLOAT4 NOT NULL, col72_5 DATE, col72_6 REGNAMESPACE NOT NULL, col72_7 BYTES, col72_8 DECIMAL NOT NULL, PRIMARY KEY (col72_2, col72_1 ASC, col72_0 DESC), INDEX (col72_1 DESC, col72_2, col72_0, col72_6 ASC, col72_8 ASC, col72_5 ASC, col72_4), INDEX (col72_6 DESC, col72_5 DESC), INDEX (col72_3, col72_2 DESC, col72_6 DESC, col72_8 ASC, col72_5 ASC, col72_7, col72_0), UNIQUE (col72_7 DESC, col72_5, col72_1, col72_4 DESC, col72_0 ASC), UNIQUE (col72_8), INDEX (col72_8 DESC, col72_1, col72_7) STORING (col72_4, col72_5, col72_6), FAMILY (col72_4, col72_8), FAMILY (col72_2), FAMILY (col72_5), FAMILY (col72_3), FAMILY (col72_6), FAMILY (col72_1, col72_0), FAMILY (col72_7));
          | ***FAIL; Expected no errors, but got ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2426238&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@f2b53d1080f1ccc503263dd178ee31b4878d2360:

          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=a1deaad3 key=/Table/SystemConfigSpan/Start pri=0.00332277 epo=0 ts=1604996009.003118996,1 min=1604996008.481483057,0 seq=7} lock=true stat=PENDING rts=1604996008.481483057,0 wto=false max=1604996008.481483057,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE schema1.table188 RENAME TO schema1.table222;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=ffdc1ed1 key=/Table/SystemConfigSpan/Start pri=0.00950908 epo=0 ts=1604996008.859975551,2 min=1604996008.454586764,0 seq=6} lock=true stat=PENDING rts=1604996008.454586764,0 wto=false max=1604996008.454586764,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   ALTER TABLE schema1.table188 ALTER COLUMN "col188_1" DROP STORED;
          | ROLLBACK; ERROR: column "col188_1" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   ALTER TABLE schema1.table188 RENAME CONSTRAINT "table188_col188_2_col188_11_col188_14_col188_10_col188_1_col188_6_col188_13_col188_12_col188_8_col188_7_col188_5_col188_9_col188_3_col188_4_idx" TO "index188_227";
          | ROLLBACK; ERROR: constraint "table188_col188_2_col188_11_col188_14_col188_10_col188_1_col188_6_col188_13_col188_12_col188_8_col188_7_col188_5_col188_9_col188_3_col188_4_idx" does not exist (SQLSTATE 42704)
          | BEGIN
          |   CREATE SEQUENCE "seq228";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE schema1.table188 ALTER COLUMN "col188_2" DROP NOT NULL;
          | ROLLBACK; ERROR: column "col188_2" is in a primary index (SQLSTATE 42P16)
          | BEGIN
          |   DROP TABLE schema1.table188;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=422a795e key=/Table/SystemConfigSpan/Start pri=0.05882680 epo=0 ts=1604996008.859975551,2 min=1604996008.859975551,0 seq=6} lock=true stat=PENDING rts=1604996008.859975551,0 wto=false max=1604996008.859975551,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE schema1.table188 ALTER COLUMN "col188_11" SET NOT NULL;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=5771de77 key=/Table/SystemConfigSpan/Start pri=0.00500362 epo=0 ts=1604996009.028525017,1 min=1604996008.621483799,0 seq=6} lock=true stat=PENDING rts=1604996008.621483799,0 wto=false max=1604996008.621483799,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema232 AUTHORIZATION root;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   DROP SEQUENCE "seq210";
          |   CREATE TABLE public.table233 AS SELECT col188_14, col188_6, col188_3, col188_4, col188_2, col188_12, col188_8, col188_5, col188_0, col188_1, col188_10, col188_11, col188_13 FROM schema1.table188;
          |   CREATE TYPE "schema1.enum234" AS ENUM ('fdssdf', 'asa');
          |   DROP SEQUENCE "seq236";
          | ROLLBACK; ERROR: relation "seq236" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TYPE "schema72.enum221" AS ENUM ('d');
          |   CREATE TABLE schema1.table229 (col229_0 DECIMAL NOT NULL, col229_1 OID, col229_2 REGPROC NOT NULL, col229_3 "char", col229_4 OID, col229_5 FLOAT4[], PRIMARY KEY (col229_2, col229_0 ASC), INDEX (col229_1 DESC, col229_3, col229_4, col229_0, col229_2), UNIQUE (col229_3));
          |   ALTER TABLE schema1.table188 ADD COLUMN col188_235 TIME NULL;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=6df61957 key=/Table/SystemConfigSpan/Start pri=0.00048401 epo=0 ts=1604996009.173371665,1 min=1604996008.724212890,0 seq=22} lock=true stat=PENDING rts=1604996008.724212890,0 wto=false max=1604996008.724212890,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2430130&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56184

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@f2b53d1080f1ccc503263dd178ee31b4878d2360:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2226
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:129
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:73
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1357
        Wraps: (2) output in run_172155.153_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2431438-1605024874-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201110 17:21:56.024942 1 workload/cli/run.go:356  creating load generator...
          | I201110 17:21:56.111321 1 workload/cli/run.go:387  creating load generator... done (took 86.377537ms)
          | Error: ***UNEXPECTED ERROR: ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 1047380080:::OID,2581704626:::OID,(-0.12716174125671387):::FLOAT8,B'01010101000100110111011110','00:00:00+15:59:00':::TIMETZ,'07f83fb5-9f71-47e8-b24a-0253952ecb61':::UUID,'1993-08-30':::DATE,'1975-03-31':::DATE,'\x638893d3b6cf82':::BYTES,'37 years 10 mons 995 days 12:54:06.850656':::INTERVAL,2447000860:::OID,'1970-01-12 10:51:12.000035':::TIMESTAMP,B'0000000000000000000000000000000000000000000000000000000000000000',1202255848:::INT8);
          | ROLLBACK; ERROR: bit string length 64 does not match type BIT(41) (SQLSTATE 22026)
          |     2.0s        0           42.0           41.0     14.2    201.3    268.4    268.4 opOk
          |     2.0s        0            7.0            8.5    604.0   1140.9   1140.9   1140.9 txnCmtErr
          |     2.0s        0            0.0            2.5      0.0      0.0      0.0      0.0 txnOk
          |     2.0s        0           28.0           21.5      2.9     29.4     35.7     35.7 txnRbk
          | BEGIN
          |   CREATE SEQUENCE "seq88";
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   DROP SEQUENCE "seq52";
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=eef6dd4c key=/Table/SystemConfigSpan/Start pri=0.00740243 epo=0 ts=1605028917.807859546,1 min=1605028917.564633904,0 seq=11} lock=true stat=PENDING rts=1605028917.564633904,0 wto=false max=1605028918.064633904,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE INDEX index48_97 ON public.table48 (col48_4 DESC, col48_11 DESC) STORING (col48_1, col48_17, col48_0, col48_8, col48_18, col48_7, col48_12, col48_15, col48_6, col48_2, col48_16);
          | ROLLBACK; ERROR: index "index48_97" already contains column "col48_17" (SQLSTATE 42701)
          | BEGIN
          |   INSERT INTO public.table48 ("col48_0","col48_1","col48_2","col48_3","col48_4","col48_5","col48_6","col48_7","col48_8","col48_9","col48_10","col48_11","col48_12","col48_13","col48_14","col48_15","col48_16","col48_17","col48_18") VALUES (B'01011000011001000100101010110100101001001',NULL,'BOX(-0.28674123623536024 -1.3673733447611331,0.6146156096442191 -1.1700235598081012)':::BOX2D,781630756:::OID,e'\x0e\'.R':::STRING:::NAME,811143341:::OID,0:::OID,0.05736084282398224:::FLOAT8,B'10001110000010000110001001','03:17:46.042789+00:01:00':::TIMETZ,'bbf417f9-a377-43bc-8711-264862f553b7':::UUID,'1982-04-10':::DATE,'1991-03-17':::DATE,'\xe7':::BYTES,'-42 years -4 mons -758 days -04:29:30.616142':::INTERVAL,NULL,'1970-01-10 09:24:50.000376':::TIMESTAMP,B'10001110100101100100111000000100010100010',1:::INT8),(B'01100010010110000100010000010110111111111',2.621255450938645577E+37:::DECIMAL,'BOX(-0.7751573381953424 0.14408485757818568,2.193165131178402 0.9336560497334027)':::BOX2D,2021175673:::OID,'b@"':::STRING:::NAME,1354054701:::OID,0:::OID,0.6713987588882446:::FLOAT8,B'10110010101010110110011101','15:13:51.449759+05:46:00':::TIMETZ,'180a3f69-7b4e-483e-8b15-468d870f6446':::UUID,'1975-03-05':::DATE,'1974-07-08':::DATE,'\xd421b281850d':::BYTES,'-1 years -5 mons -854 days -23:54:59.695348':::INTERVAL,0:::OID,'1970-01-07 19:19:16.000332':::TIMESTAMP,B'01101110010010001011001010011111111010101',24832985:::INT8),(B'00110100100000011000101000001001000010011',(-7.42853988115019900E+32):::DECIMAL,'BOX(-0.3548929755240329 0.3842478013938248,0.24072470125070233 1.1777177564259227)':::BOX2D,1745672044:::OID,e'\'':::STRING:::NAME,3023903317:::OID,436077371:::OID,(-1.5103683471679688):::FLOAT8,B'01111001101000010111010100','21:54:35.459572+00:39:00':::TIMETZ,'fcf55ec4-0be4-40ce-8e19-44d87a2c1319':::UUID,'1993-02-14':::DATE,'1981-03-21':::DATE,'\x112609cd33':::BYTES,'-18 years -1 mons -861 days -14:49:05.975467':::INTERVAL,4219162875:::OID,'1970-01-02 03:21:53.000471':::TIMESTAMP,B'11000101101010010010100101111001100001000',(-82520509):::INT8);
          |   ALTER TABLE public.table48 ADD COLUMN col48_99 INTERVAL;
          | ***FAIL; Expected no errors, but got ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2431438&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57487

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@df21e5065a79d1dd5b0fd76018724dd26660beb2:

          | I201111 08:11:49.617448 1 workload/cli/run.go:356  creating load generator...
          | I201111 08:11:49.692840 1 workload/cli/run.go:387  creating load generator... done (took 75.390395ms)
          | Error: ***UNEXPECTED ERROR: ERROR: descriptor is being dropped (SQLSTATE XXUUU)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | RetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schema2 54 0} {schemachange 52 1} {seq25 61 0}], old versions still in use (SQLSTATE 40001)
          | BEGIN
          |   DROP SCHEMA "schema2" CASCADE;
          |   ALTER SEQUENCE "seq25" RENAME TO "seq61";
          |   CREATE SCHEMA IF NOT EXISTS schema62 AUTHORIZATION root;
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schema2 54 0} {schemachange 52 1} {seq25 61 0}], old versions still in use (SQLSTATE 40001)
          | BEGIN
          |   DROP SEQUENCE "seq25";
          |   DROP SEQUENCE "seq13";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=c0d063f8 key=/Table/SystemConfigSpan/Start pri=0.00592079 epo=0 ts=1605082310.067381802,1 min=1605082309.972144601,0 seq=16} lock=true stat=ABORTED rts=1605082309.972144601,0 wto=false max=1605082309.972144601,0 (SQLSTATE 40001)
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |     1.0s        0           39.0           42.0     17.8    218.1    352.3    352.3 opOk
          |     1.0s        0            4.6            5.0    604.0    906.0    906.0    906.0 txnCmtErr
          |     1.0s        0            1.9            2.0     52.4    385.9    385.9    385.9 txnOk
          |     1.0s        0           24.2           26.0      3.9    209.7    260.0    260.0 txnRbk
          | BEGIN
          |   CREATE SEQUENCE "seq42";
          |   CREATE TYPE "schema2.enum47" AS ENUM ('fsad');
          |   CREATE SEQUENCE "seq51";
          |   DROP SCHEMA "schema2" CASCADE;
          |   DROP SEQUENCE "seq13";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=d7bbee23 key=/Table/SystemConfigSpan/Start pri=0.00089370 epo=0 ts=1605082310.633381563,1 min=1605082309.750289592,0 seq=48} lock=true stat=PENDING rts=1605082309.750289592,0 wto=false max=1605082309.750289592,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE schema35.table36 (col36_0 BYTES NOT NULL, col36_1 INT2 NOT NULL, col36_2 INTERVAL NULL, col36_3 FLOAT8, col36_4 VARCHAR NOT NULL, col36_5 INT8 NOT NULL, col36_6 INT4 NULL, col36_7 BYTES NOT NULL, PRIMARY KEY (col36_4 ASC, col36_5 DESC, col36_0 ASC, col36_1 ASC, col36_7 DESC), UNIQUE (col36_1 ASC, col36_3 ASC, col36_5 DESC, col36_0, col36_4 ASC, col36_7 ASC), UNIQUE (col36_0 DESC, col36_4, col36_5, col36_7 ASC, col36_3 DESC, col36_6 DESC, col36_2 DESC), INDEX (col36_7 ASC, col36_4 ASC, col36_3, col36_6 DESC, col36_0 ASC, col36_5, col36_2 ASC, col36_1), UNIQUE (col36_5 DESC, col36_0 ASC, col36_2 DESC, col36_3 DESC), FAMILY (col36_7), FAMILY (col36_4), FAMILY (col36_2), FAMILY (col36_5, col36_3), FAMILY (col36_1, col36_0), FAMILY (col36_6));
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema35.table36" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "schema2" CASCADE;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=68e0c14a key=/Table/SystemConfigSpan/Start pri=0.00370760 epo=0 ts=1605082310.720621263,2 min=1605082310.178854269,0 seq=1} lock=true stat=PENDING rts=1605082310.178854269,0 wto=false max=1605082310.178854269,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE "seq48";
          |   CREATE TABLE IF NOT EXISTS schema2.table66 (col66_0 REGPROC NULL, col66_1 INET NOT NULL, col66_2 INTERVAL, col66_3 INT4, col66_4 CHAR NULL, col66_5 DECIMAL NOT NULL, col66_6 CHAR, col66_7 BIT(33), col66_8 REGPROC NOT NULL, PRIMARY KEY (col66_5 DESC, col66_8 DESC, col66_1 DESC), INDEX (col66_7 ASC, col66_8 ASC, col66_5, col66_4 DESC) STORING (col66_2, col66_3), INDEX (col66_8 ASC, col66_0, col66_2 DESC, col66_5 DESC, col66_3 DESC, col66_4 DESC), UNIQUE (col66_3) STORING (col66_2, col66_6));
          | ***FAIL; Expected no errors, but got ERROR: descriptor is being dropped (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2433350&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) Related: - #56513 roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-56248](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-56248) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56184

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@2d34be73030a0d114b84fc2785672eda072c5a50:

          | t;
          |   DROP VIEW schema225.view226;
          | ROLLBACK; ERROR: relation "schema225.view226" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SEQUENCE "seq229";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   ALTER TABLE public.table228 ALTER COLUMN "col228_0" DROP NOT NULL;
          | ROLLBACK; ERROR: column "col228_0" is in a primary index (SQLSTATE 42P16)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema208 AUTHORIZATION root;
          |   ALTER TABLE public.table228 ALTER COLUMN "col228_0" DROP STORED;
          | ROLLBACK; ERROR: column "col228_0" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   ALTER TABLE public.table228 DROP CONSTRAINT "table228_col228_0_key";
          | ROLLBACK; ERROR: unimplemented: cannot drop UNIQUE constraint "table228_col228_0_key" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table228 ALTER COLUMN "col228_0" DROP STORED;
          | ROLLBACK; ERROR: column "col228_0" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   ALTER TABLE public.table228 ALTER COLUMN "col228_0" DROP NOT NULL;
          | ROLLBACK; ERROR: column "col228_0" is in a primary index (SQLSTATE 42P16)
          | BEGIN
          |   CREATE SEQUENCE "seq235";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE public.table228 RENAME COLUMN "col228_0" TO "col228_231";
          |   ALTER TABLE public.table228 ALTER COLUMN "col228_2" DROP STORED;
          | ROLLBACK; ERROR: column "col228_2" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   CREATE SEQUENCE "seq180";
          |   CREATE SEQUENCE "seq191";
          |   DROP SEQUENCE "seq180";
          |   DROP SEQUENCE "seq190";
          | COMMIT;  
          | BEGIN
          |   CREATE TABLE public.table222 (col222_0 INT4 NOT NULL, col222_1 FLOAT8, col222_2 BOOL, col222_3 REGCLASS NOT NULL, col222_4 UUID NULL, col222_5 DECIMAL, col222_6 "char", col222_7 REGPROC, col222_8 BIT(31), col222_9 FLOAT8, col222_10 VARCHAR, col222_11 BIT(19) NOT NULL, col222_12 BOX2D NULL, UNIQUE (col222_11 ASC), INDEX (col222_9 ASC, col222_2 ASC, col222_11 DESC, col222_0 DESC, col222_4 ASC, col222_5, col222_7 DESC) STORING (col222_1, col222_6, col222_8), INDEX (col222_3 ASC, col222_12 ASC, col222_0 ASC, col222_9 ASC), UNIQUE (col222_8 ASC, col222_1 ASC, col222_0, col222_12, col222_5, col222_11 DESC, col222_10 DESC, col222_2, col222_7, col222_9), INDEX (col222_5, col222_9, col222_4 DESC, col222_2 ASC, col222_12, col222_0 DESC, col222_3 ASC, col222_7 DESC, col222_11 DESC), FAMILY (col222_9), FAMILY (col222_8), FAMILY (col222_7, col222_6, col222_3, col222_0, col222_10), FAMILY (col222_1, col222_12), FAMILY (col222_5, col222_2, col222_11, col222_4));
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=8d5180aa key=/Table/SystemConfigSpan/Start pri=0.00415877 epo=0 ts=1605254932.576730025,1 min=1605254932.292015278,0 seq=7} lock=true stat=PENDING rts=1605254932.292015278,0 wto=false max=1605254932.792015278,0 (SQLSTATE 40001)
          | BEGIN
          |   INSERT INTO public.table228 ("col228_0","col228_1","col228_2") VALUES (true,(-0.23978972434997559):::FLOAT8,ARRAY['1975-08-16':::DATE,'1993-07-30':::DATE,'1971-07-23':::DATE]),(true,(-0.8790212273597717):::FLOAT8,ARRAY['1982-10-08':::DATE]),(true,(-0.16100792586803436):::FLOAT8,ARRAY['1981-02-14':::DATE,'1982-01-09':::DATE,'1986-04-03':::DATE,'1996-10-11':::DATE]),(false,0.3714587688446045:::FLOAT8,ARRAY['1978-05-05':::DATE]);
          | ROLLBACK; ERROR: duplicate key value (col228_0)=(true) violates unique constraint "primary" (SQLSTATE 23505)
          | BEGIN
          |   INSERT INTO public.table228 ("col228_0","col228_1","col228_2") VALUES (true,0.361959844827652:::FLOAT8,ARRAY['infinity':::DATE,'1971-07-05':::DATE,'1996-06-15':::DATE,'1996-09-15':::DATE,'1979-02-01':::DATE]);
          |   ALTER TABLE public.table228 ADD COLUMN col228_2 TIME;
          | ***FAIL; Expected one of SQLSTATES 42701, but got ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2439944&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) Related: - #56513 roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-56248](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-56248) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57487

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@6b254f2d4a8e1470c1a7d5bd7c3f746d69de0797:

          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq80";
          |   DROP VIEW public.view82;
          | ROLLBACK; ERROR: relation "public.view82" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TYPE "schema83.enum84" AS ENUM ('daddaa', 'asfaa', 'd', 'sfds');
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA schema72 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq85";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA IF NOT EXISTS schema81 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=1bc6999f key=/Table/SystemConfigSpan/Start pri=0.00896542 epo=0 ts=1605341424.817230103,2 min=1605341424.258231355,0 seq=6} lock=true stat=PENDING rts=1605341424.258231355,0 wto=false max=1605341424.258231355,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema8 AUTHORIZATION root;
          | ROLLBACK; ERROR: schema "schema8" already exists (SQLSTATE 42P06)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema87 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=2a819433 key=/Table/SystemConfigSpan/Start pri=0.02036644 epo=0 ts=1605341424.876528409,1 min=1605341424.494296635,0 seq=6} lock=true stat=PENDING rts=1605341424.494296635,0 wto=false max=1605341424.494296635,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema97 AUTHORIZATION root;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER VIEW public.view98 RENAME TO public.view99;
          | ROLLBACK; ERROR: relation "public.view98" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq100";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA schema89 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=46ebc455 key=/Table/SystemConfigSpan/Start pri=0.00947756 epo=0 ts=1605341424.876528409,1 min=1605341424.348050956,0 seq=6} lock=true stat=PENDING rts=1605341424.348050956,0 wto=false max=1605341424.348050956,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table101 RENAME TO public.table102;
          | ROLLBACK; ERROR: relation "public.table101" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TYPE "schema50.enum94" AS ENUM ('s', 'a', 'saafa', 'af', 'fdsfas', 'fadda');
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=8821fbd0 key=/Table/SystemConfigSpan/Start pri=0.01165038 epo=0 ts=1605341424.876528409,1 min=1605341424.696822805,0 seq=9} lock=true stat=PENDING rts=1605341424.696822805,0 wto=false max=1605341425.196822805,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP VIEW schema8.view105;
          | ROLLBACK; ERROR: descriptor is being dropped (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2442839&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) Related: - #56513 roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-56248](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-56248) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57487

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@5d8f1d3dbc356573a1fb2d1d6bf564bf0370daff:

          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | max=1605428753.470292672,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema488 AUTHORIZATION root;
          |   CREATE TYPE "schema488.enum499" AS ENUM ('aadsfd', 'afas', 'dds', 'faadd');
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=918732fd key=/Table/SystemConfigSpan/Start pri=0.01943251 epo=0 ts=1605428754.323558760,2 min=1605428753.143279777,0 seq=22} lock=true stat=PENDING rts=1605428753.143279777,0 wto=false max=1605428753.643279777,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP SCHEMA "schema342" CASCADE;
          |   CREATE SCHEMA schema500 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq501";
          |   DROP SCHEMA "schema504" CASCADE;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE schema342.table505 ALTER COLUMN "col505_506" SET NOT NULL;
          | ROLLBACK; ERROR: relation "schema342.table505" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema508 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=171da496 key=/Table/SystemConfigSpan/Start pri=0.12313202 epo=0 ts=1605428754.606730020,1 min=1605428753.763053703,0 seq=6} lock=true stat=PENDING rts=1605428753.763053703,0 wto=false max=1605428753.763053703,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE VIEW schema342.view516 AS SELECT "IrrelevantColumnName" FROM schema342.table513;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "schema342.table513" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SEQUENCE "seq518";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA schema519 AUTHORIZATION root;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER SEQUENCE "seq335" RENAME TO "seq510";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=1f718ba0 key=/Table/SystemConfigSpan/Start pri=0.04388234 epo=0 ts=1605428754.373991943,1 min=1605428753.902941151,0 seq=6} lock=true stat=PENDING rts=1605428753.902941151,0 wto=false max=1605428753.902941151,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE "seq509";
          |   DROP SCHEMA "schema342" CASCADE;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=7dc77c52 key=/Table/SystemConfigSpan/Start pri=0.00447226 epo=0 ts=1605428754.545091262,2 min=1605428754.139192897,0 seq=9} lock=true stat=PENDING rts=1605428754.139192897,0 wto=false max=1605428754.139192897,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS schema342.table512 (col512_0 REGCLASS, col512_1 REGPROC NOT NULL, col512_2 REGTYPE NOT NULL, col512_3 BIT(25) NOT NULL, col512_4 DECIMAL NOT NULL, col512_5 REGCLASS NOT NULL, col512_6 VARCHAR NOT NULL, col512_7 TIMETZ NOT NULL, col512_8 BYTES NOT NULL, col512_9 DATE NOT NULL, col512_10 DATE NOT NULL, col512_11 DATE NOT NULL, col512_12 TIMESTAMP NOT NULL, col512_13 TIMESTAMP NOT NULL, PRIMARY KEY (col512_9, col512_1 ASC, col512_6, col512_3 DESC, col512_7 DESC, col512_13 DESC, col512_10 DESC, col512_4, col512_5 DESC, col512_2, col512_12, col512_11, col512_8 ASC), INDEX (col512_5, col512_0, col512_4, col512_1 DESC, col512_2), INDEX (col512_2 DESC, col512_10, col512_7 ASC, col512_13 DESC, col512_6 DESC, col512_1 DESC, col512_4 ASC));
          | ***FAIL; Expected no errors, but got ERROR: descriptor is being dropped (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2444337&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) Related: - #56513 roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-56248](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-56248) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@b4bf42c4fe10155717ac48f2e68ad8b501ca0b1a:

          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_074526.516_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2445673-1605509892-22-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201116 07:45:27.214146 1 workload/cli/run.go:356  creating load generator...
          | I201116 07:45:27.288896 1 workload/cli/run.go:387  creating load generator... done (took 74.748896ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=e1bac91f key=/Table/SystemConfigSpan/Start pri=0.02362072 epo=0 ts=1605512727.829280112,1 min=1605512727.435364183,0 seq=6} lock=true stat=PENDING rts=1605512727.435364183,0 wto=false max=1605512727.435364183,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | orted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SEQUENCE "seq40";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE TABLE public.table42 (col42_0 REGCLASS NOT NULL, col42_1 REGCLASS NULL, col42_2 BOOL NOT NULL, col42_3 TIME NOT NULL, col42_4 INT8 NOT NULL, col42_5 INTERVAL NOT NULL, col42_6 INET NOT NULL, col42_7 STRING NOT NULL, col42_8 TIMETZ NOT NULL, col42_9 "char"[] NOT NULL, col42_10 INET NOT NULL, col42_11 TIMESTAMPTZ NOT NULL, col42_12 TIMESTAMPTZ NOT NULL, col42_13 INT4[] NOT NULL, col42_14 BIT, col42_15 BIT(16) NOT NULL, col42_16 BIT(44) NOT NULL, col42_17 UUID NOT NULL, col42_18 DECIMAL NOT NULL, PRIMARY KEY (col42_17 ASC, col42_18 ASC, col42_7 ASC, col42_4 DESC, col42_11 ASC, col42_10, col42_6, col42_12 DESC, col42_2 DESC, col42_3 DESC, col42_15), UNIQUE (col42_2 ASC, col42_14 DESC, col42_8, col42_4, col42_7 ASC, col42_0 ASC, col42_11, col42_12, col42_1 DESC, col42_5, col42_15 DESC, col42_3 ASC, col42_17 ASC, col42_6 ASC, col42_18 DESC), INDEX (col42_10 DESC, col42_16, col42_18) STORING (col42_1, col42_5, col42_8, col42_9, col42_13, col42_14), FAMILY (col42_17), FAMILY (col42_0, col42_16, col42_4), FAMILY (col42_5, col42_7, col42_3), FAMILY (col42_13), FAMILY (col42_2), FAMILY (col42_9), FAMILY (col42_8, col42_10), FAMILY (col42_6), FAMILY (col42_18), FAMILY (col42_12, col42_1, col42_11, col42_15), FAMILY (col42_14));
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=052403ba key=/Table/SystemConfigSpan/Start pri=0.00766395 epo=0 ts=1605512727.645315343,1 min=1605512727.289582867,0 seq=7} lock=true stat=PENDING rts=1605512727.289582867,0 wto=false max=1605512727.789582867,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE schema43.table44 ALTER COLUMN "IrrelevantColumnName" DROP DEFAULT;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "schema43.table44" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table46 ALTER COLUMN "IrrelevantColumnName" DROP NOT NULL;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table46" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table45 DROP COLUMN "IrrelevantColumnName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table45" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TABLE public.table33 (col33_0 CHAR NOT NULL, col33_1 BOOL NOT NULL, col33_2 DECIMAL NOT NULL, col33_3 REGNAMESPACE NULL, col33_4 REGPROC NOT NULL, col33_5 INT2 NOT NULL, col33_6 "char" NOT NULL, PRIMARY KEY (col33_2, col33_6 DESC, col33_5, col33_4 DESC, col33_0 ASC, col33_1 ASC), UNIQUE (col33_0 ASC), UNIQUE (col33_3 DESC, col33_1 ASC, col33_6 DESC, col33_0 ASC, col33_4, col33_5 ASC, col33_2 ASC), UNIQUE (col33_1 DESC, col33_5 ASC, col33_3 DESC, col33_6 ASC), UNIQUE (col33_6 DESC, col33_4 ASC, col33_3 DESC, col33_0 DESC, col33_2 DESC), INDEX (col33_4, col33_6 ASC, col33_0 DESC, col33_2 DESC), UNIQUE (col33_0, col33_3 DESC), UNIQUE (col33_5, col33_1 ASC), UNIQUE (col33_5 DESC), INDEX (col33_3 DESC, col33_4 ASC, col33_0 ASC, col33_6 DESC));
          |   ALTER TABLE public.table49 DROP COLUMN "IrrelevantColumnName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table49" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema50 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=e1bac91f key=/Table/SystemConfigSpan/Start pri=0.02362072 epo=0 ts=1605512727.829280112,1 min=1605512727.435364183,0 seq=6} lock=true stat=PENDING rts=1605512727.435364183,0 wto=false max=1605512727.435364183,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2445673&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) Related: - #56513 roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-56248](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-56248) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@06944cb1996ed9ccfa036118b810e46673fe0ff4:

          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | AMILY (col138_16, col138_12), FAMILY (col138_0), FAMILY (col138_10), FAMILY (col138_15, col138_8, col138_11), FAMILY (col138_7, col138_2));
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=25eef775 key=/Table/SystemConfigSpan/Start pri=0.01265123 epo=0 ts=1605686734.988713583,1 min=1605686734.796377347,0 seq=7} lock=true stat=PENDING rts=1605686734.796377347,0 wto=false max=1605686735.296377347,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE "seq134";
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=615efa62 key=/Table/SystemConfigSpan/Start pri=0.01058728 epo=0 ts=1605686734.988713583,1 min=1605686734.973457449,0 seq=8} lock=true stat=PENDING rts=1605686734.973457449,0 wto=false max=1605686735.473457449,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum140" AS ENUM ('a', 'das', 'd', 'fsdads');
          |   ALTER VIEW public.view144 RENAME TO public.view145;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.view144" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema84 AUTHORIZATION root;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   CREATE SCHEMA schema83 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq152";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=de1fbbec key=/Table/SystemConfigSpan/Start pri=0.00033313 epo=0 ts=1605686735.198832735,1 min=1605686734.363229630,0 seq=21} lock=true stat=PENDING rts=1605686734.363229630,0 wto=false max=1605686734.863229630,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema98 AUTHORIZATION root;
          |   CREATE TYPE "public.enum154" AS ENUM ('fafd', 's', 'faasss', 'a', 'sfsss', 'sfsfs');
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=8dd27567 key=/Table/SystemConfigSpan/Start pri=0.02223537 epo=0 ts=1605686735.276048769,1 min=1605686734.549356701,0 seq=22} lock=true stat=PENDING rts=1605686734.549356701,0 wto=false max=1605686735.049356701,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq156";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema124 AUTHORIZATION root;
          |   ALTER SEQUENCE "seq107" RENAME TO "seq157";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=a5a41dcc key=/Table/SystemConfigSpan/Start pri=0.02067620 epo=0 ts=1605686735.276048769,2 min=1605686734.562701041,0 seq=19} lock=true stat=PENDING rts=1605686734.562701041,0 wto=false max=1605686734.562701041,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema139 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=62948267 key=/Table/SystemConfigSpan/Start pri=0.01178132 epo=0 ts=1605686735.361804421,2 min=1605686734.825325148,0 seq=6} lock=true stat=PENDING rts=1605686734.825325148,0 wto=false max=1605686734.825325148,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2453711&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@8ff64f5d897a9c7b39ca437193a11a7300cecbc1:

          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 1_4));
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=587ba800 key=/Table/SystemConfigSpan/Start pri=0.00018408 epo=0 ts=1605773026.761237167,1 min=1605773026.525975416,0 seq=20} lock=true stat=PENDING rts=1605773026.525975416,0 wto=false max=1605773027.025975416,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema14 AUTHORIZATION root;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=206b300b key=/Table/SystemConfigSpan/Start pri=0.01996579 epo=0 ts=1605773026.825085594,1 min=1605773026.526398583,0 seq=13} lock=true stat=PENDING rts=1605773026.526398583,0 wto=false max=1605773027.026398583,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "schema3.enum4" AS ENUM ('asadsd', 'sfassd', 'sfddas', 'afada', 'sdfaa', 'daasaf');
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   DROP VIEW public.view26;
          | ROLLBACK; ERROR: relation "public.view26" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SEQUENCE "seq28";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=0beffc88 key=/Table/SystemConfigSpan/Start pri=0.00429596 epo=0 ts=1605773026.526796047,1 min=1605773026.526612263,0 seq=8} lock=true stat=PENDING rts=1605773026.526612263,0 wto=false max=1605773027.026612263,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE "seq1";
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   CREATE TYPE "public.enum27" AS ENUM ('dsdaf', 's', 'saaada', 'd', 'sfddas', 'f');
          |   CREATE TABLE IF NOT EXISTS public.table29 (col29_0 TIME NULL, col29_1 REGPROC NULL, col29_2 BOOL NOT NULL, col29_3 BIT(42) NULL, col29_4 FLOAT8 NOT NULL, UNIQUE (col29_2 DESC, col29_0 DESC, col29_4 DESC, col29_3 ASC, col29_1 ASC));
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=5b5c7efe key=/Table/SystemConfigSpan/Start pri=0.04888289 epo=0 ts=1605773026.526796047,1 min=1605773026.526755089,0 seq=16} lock=true stat=PENDING rts=1605773026.526755089,0 wto=false max=1605773027.026755089,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          |   DROP VIEW public.view32;
          | ROLLBACK; ERROR: relation "public.view32" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SEQUENCE "seq8";
          |   ALTER VIEW public.view30 RENAME TO public.view31;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.view30" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SEQUENCE "seq1";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=3638b886 key=/Table/SystemConfigSpan/Start pri=0.01527844 epo=0 ts=1605773026.863939732,1 min=1605773026.526810942,0 seq=11} lock=true stat=PENDING rts=1605773026.526810942,0 wto=false max=1605773027.026810942,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SCHEMA schema33 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=fc7177f4 key=/Table/SystemConfigSpan/Start pri=0.00809827 epo=0 ts=1605773026.999043514,1 min=1605773026.643696988,0 seq=6} lock=true stat=PENDING rts=1605773026.643696988,0 wto=false max=1605773026.643696988,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2457321&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@609cad52785f49338f7ceff289486b87db4cecb6:

          | stdout:
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table3 ALTER COLUMN "IrrelevantColumnName" DROP DEFAULT;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table3" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   ALTER SEQUENCE "seq6" RENAME TO "seq7";
          | ROLLBACK; ERROR: relation "seq6" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   ALTER TABLE public.table15 DROP COLUMN "IrrelevantColumnName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table15" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE "seq14";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=ce93528d key=/Table/SystemConfigSpan/Start pri=0.01744009 epo=0 ts=1605858617.973329508,1 min=1605858617.969861408,0 seq=8} lock=true stat=PENDING rts=1605858617.969861408,0 wto=false max=1605858618.469861408,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum4" AS ENUM ('asfffa');
          |   CREATE SEQUENCE "seq17";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=cbeca287 key=/Table/SystemConfigSpan/Start pri=0.00586971 epo=0 ts=1605858617.973329508,1 min=1605858617.970579403,0 seq=17} lock=true stat=PENDING rts=1605858617.970579403,0 wto=false max=1605858618.470579403,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   CREATE SEQUENCE "seq26";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE schema1.table23 RENAME CONSTRAINT "index23_24" TO "index23_25";
          | ROLLBACK; ERROR: relation "schema1.table23" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TYPE "public.enum32" AS ENUM ('afs');
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TYPE "public.enum10" AS ENUM ('affdsa', 'sada', 'aa', 'sf', 'ffsda', 'sfa');
          |   CREATE SCHEMA schema33 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=750b700a key=/Table/SystemConfigSpan/Start pri=0.00761035 epo=0 ts=1605858618.336904277,1 min=1605858617.968846766,0 seq=15} lock=true stat=PENDING rts=1605858617.968846766,0 wto=false max=1605858617.968846766,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2460636&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@7b8288b26c3aad649ed6e2d89f679d46b5f3988d:

          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | N
          |   ALTER TABLE public.table100 ADD COLUMN col100_161 BYTES NULL;
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          |     3.0s        0           39.0           44.0     19.9    335.5    436.2    436.2 opOk
          |     3.0s        0            6.0            6.7    838.9   1040.2   1040.2   1040.2 txnCmtErr
          |     3.0s        0            3.0            2.0    385.9   2281.7   2281.7   2281.7 txnOk
          |     3.0s        0           30.0           26.7      1.4     23.1     39.8     39.8 txnRbk
          | BEGIN
          |   DROP SEQUENCE "seq134";
          |   CREATE SEQUENCE "seq159";
          |   ALTER TABLE public.table100 RENAME CONSTRAINT "table100_col100_4_col100_8_col100_5_col100_6_col100_10_col100_12_col100_9_key" TO "index100_168";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=1694ca18 key=/Table/SystemConfigSpan/Start pri=0.00050520 epo=0 ts=1606203990.147754862,1 min=1606203989.800204204,0 seq=24} lock=true stat=ABORTED rts=1606203989.800204204,0 wto=false max=1606203989.800204204,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table100 ALTER COLUMN "col100_7" DROP DEFAULT;
          |   ALTER TABLE public.table100 ALTER COLUMN "col100_14" DROP DEFAULT;
          |   DROP SEQUENCE "seq134";
          |   ALTER TABLE public.table100 ALTER COLUMN "col100_13" DROP STORED;
          | ROLLBACK; ERROR: column "col100_13" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   CREATE VIEW public.view165 AS SELECT public.table100.col100_13, public.table100.col100_3, public.table100.col100_9, public.table100.col100_7, public.table100.col100_12, public.table100.col100_8, public.table100.col100_10 FROM public.table100;
          |   CREATE TABLE public.table173 AS SELECT public.table100.col100_6, public.table100.col100_2, public.table100.col100_13, public.table100.col100_3, public.table100.col100_14 FROM public.table100, public.table100, public.table100;
          | ROLLBACK; expected SQLSTATE(S) 42712, and got ERROR: source name "table100" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE schema175.table176 RENAME TO schema175.table177;
          | ROLLBACK; expected SQLSTATE(S) 3F000,42P01, and got ERROR: relation "schema175.table176" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TYPE "public.enum160" AS ENUM ('fsssfa');
          |   DROP INDEX public.table100@"table100_col100_0_col100_5_col100_10_col100_3_col100_11_key";
          | ROLLBACK; ERROR: index "table100_col100_0_col100_5_col100_10_col100_3_col100_11_key" is in use as unique constraint (SQLSTATE 2BP01)
          | BEGIN
          |   CREATE VIEW public.view181 AS SELECT public.table100.col100_7, public.table100.col100_8, public.table100.col100_11, public.table100.col100_2, public.table100.col100_9, public.table100.col100_8, public.table100.col100_14, public.table100.col100_5, public.table100.col100_10, public.table100.col100_6, public.table100.col100_13, public.table100.col100_1, public.table100.col100_12, public.table100.col100_9, public.table100.col100_0, public.table100.col100_3, public.table100.col100_2, public.table100.col100_4, public.table100.col100_7, public.table100.col100_12, public.table100.col100_7, public.table100.col100_3, public.table100.col100_13, public.table100.col100_1, public.table100.col100_11, public.table100.col100_5, public.table100.col100_14, public.table100.col100_8 FROM public.table100, public.table100, public.table100;
          | ROLLBACK; expected SQLSTATE(S) 42701,42712, and got ERROR: source name "table100" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE public.table100 RENAME COLUMN "col100_5" TO "col100_171";
          |   CREATE SCHEMA schema174 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=6c52b48e key=/Table/SystemConfigSpan/Start pri=0.01874128 epo=0 ts=1606203990.324580358,1 min=1606203989.830358115,0 seq=16} lock=true stat=ABORTED rts=1606203989.830358115,0 wto=false max=1606203989.830358115,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2468817&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@d251e175421d9303492a4923fb933515987163b6:

          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:135
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:79
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_075135.817_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2471879-1606287802-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201125 07:51:36.541621 1 workload/cli/run.go:357 ⋮ creating load generator...
          | I201125 07:51:36.618994 1 workload/cli/run.go:388 ⋮ creating load generator... done (took 77.371294ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=1e7dc33a key=/Table/SystemConfigSpan/Start pri=0.00078985 epo=0 ts=1606290697.157839974,1 min=1606290696.993835252,0 seq=18} lock=true stat=ABORTED rts=1606290696.993835252,0 wto=false max=1606290696.993835252,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  CONSTRAINT "table4_col4_11_col4_5_col4_14_col4_7_col4_1_col4_2_col4_10_col4_9_col4_0_col4_3_col4_13_col4_6_key";
          | ROLLBACK; ERROR: unimplemented: cannot drop UNIQUE constraint "table4_col4_11_col4_5_col4_14_col4_7_col4_1_col4_2_col4_10_col4_9_col4_0_col4_3_col4_13_col4_6_key" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
          |   INSERT INTO public.table4 ("col4_0","col4_1","col4_2","col4_3","col4_4","col4_5","col4_6","col4_7","col4_8","col4_9","col4_10","col4_11","col4_12","col4_13","col4_14","col4_15") VALUES ('':::STRING:::NAME,0.1904248548208427:::FLOAT8,'d7a6:113c:623d:99c5:8bd6:ec7b:a4fe:23fd/123':::INET,NULL,0:::OID,(-0.6299803773797832):::FLOAT8,4016812750:::OID,'06:24:26.775007':::TIME,'eed4:793d:cc3e:7b3f:3317:cdda:a808:6c2c/126':::INET,B'0010001010000001011110',1187174771:::OID,'@':::STRING,4199524626:::OID,1280372513:::OID,(-9193896051552473655):::INT8,'\x3d2a51313c62':::BYTES),(e'\x020':::STRING:::NAME,0.7516474514666238:::FLOAT8,'40da:b71a:c557:48ed:217a:84f9:d6b:ea46/22':::INET,B'101000100',2250857123:::OID,(-0.09165599521853596):::FLOAT8,3888553868:::OID,'00:00:00':::TIME,'3c86:3e49:9366:43e9:b160:93d9:822d:357b/5':::INET,B'0100000101100101100100',1187711936:::OID,',':::STRING,3220639729:::OID,NULL,(-1694402882330872557):::INT8,'\x5477bfdcddb0':::BYTES),('gL!n':::STRING:::NAME,NULL,'4e71:75f1:2386:a3a3:d6a7:79b4:4ec6:65a3/56':::INET,B'100011110',4282815463:::OID,0.832410478679039:::FLOAT8,1355294509:::OID,'19:39:59.660154':::TIME,'91.149.121.123/15':::INET,B'0001110011101100011110',127451352:::OID,e'\x17\x01.':::STRING,2884649567:::OID,2136545199:::OID,5153450319245005425:::INT8,'\x37671751636c1dffd5':::BYTES),(e'\r_':::STRING:::NAME,0.9417568874365374:::FLOAT8,'163.143.6.123/17':::INET,B'001011000',3151384822:::OID,0.18408607711936031:::FLOAT8,0:::OID,'00:15:30.562946':::TIME,'0.0.0.0/0':::INET,B'0100000011011000100100',3236701439:::OID,'A@^"':::STRING,0:::OID,1457612898:::OID,(-319575828475728610):::INT8,'\x818284773b5d6ba194':::BYTES),('X':::STRING,0.753563081456349:::FLOAT8,'417b:409e:937c:f797:1c5:d3aa:9268:e712/3':::INET,NULL,276245418:::OID,(-0.6738089229586208):::FLOAT8,4099672847:::OID,'16:20:58.259441':::TIME,'111.129.212.183/24':::INET,B'0011001000011000101100',1461920795:::OID,e'\U00002603':::STRING,595721938:::OID,3778126447:::OID,(-6900951957850802108):::INT8,'\x87b613':::BYTES);
          |   SELECT 'validating all objects';
          |   CREATE VIEW public.view43 AS SELECT public.table4.col4_4, public.table4.col4_5, public.table4.col4_8, public.table4.col4_12, public.table4.col4_11, public.table4.col4_3, public.table4.col4_14, public.table4.col4_10, public.table4.col4_1, public.table4.col4_9 FROM public.table4;
          | ROLLBACK; expected SQLSTATE(S) 0A000, and got ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table4 ALTER COLUMN "col4_9" DROP DEFAULT;
          |   CREATE SEQUENCE "seq46";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TYPE "public.enum28" AS ENUM ('dda', 'fsdss', 'adfa', 'adda', 'dddaa');
          |   CREATE SEQUENCE "seq48";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TABLE public.table50 AS SELECT public.table4.col4_15 FROM public.table4;
          |   ALTER TABLE public.table4 ALTER COLUMN "col4_51" DROP DEFAULT;
          | ROLLBACK; expected SQLSTATE(S) 42703, and got ERROR: column "col4_51" does not exist (SQLSTATE 42703)
          | BEGIN
          |   CREATE SCHEMA schema37 AUTHORIZATION root;
          |   ALTER TABLE public.table4 RENAME COLUMN "col4_11" TO "col4_42";
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=1e7dc33a key=/Table/SystemConfigSpan/Start pri=0.00078985 epo=0 ts=1606290697.157839974,1 min=1606290696.993835252,0 seq=18} lock=true stat=ABORTED rts=1606290696.993835252,0 wto=false max=1606290696.993835252,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2471879&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@fe5da5c2df069dc7f820f5c3e2f3e03f1cb7b661:

          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TYPE "public.enum18" AS ENUM ('adsda', 'ssas', 's');
          |   ALTER TABLE public.table28 ALTER COLUMN "IrrelevantColumnName" DROP NOT NULL;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table28" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table29 ALTER COLUMN "col29_30" SET DATA TYPE char;
          | ROLLBACK; ERROR: relation "public.table29" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SCHEMA schema1 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq31";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SEQUENCE "seq2";
          |   ALTER SEQUENCE "seq2" RENAME TO "seq22";
          |   SELECT 'validating all objects';
          |   CREATE TYPE "public.enum34" AS ENUM ('dsd', 'sf', 'a', 's', 'd');
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=4a151a25 key=/Table/SystemConfigSpan/Start pri=0.02424712 epo=0 ts=1606377023.119192047,1 min=1606377023.085016626,0 seq=31} lock=true stat=PENDING rts=1606377023.085016626,0 wto=false max=1606377023.585016626,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA schema5 AUTHORIZATION root;
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "schema5" CASCADE;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=f6265695 key=/Table/SystemConfigSpan/Start pri=0.01997533 epo=0 ts=1606377023.482811564,1 min=1606377023.085352837,0 seq=30} lock=true stat=PENDING rts=1606377023.085352837,0 wto=false max=1606377023.585352837,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum4" AS ENUM ('sasfdf');
          |   CREATE SCHEMA IF NOT EXISTS schema16 AUTHORIZATION root;
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=d540e481 key=/Table/SystemConfigSpan/Start pri=0.00236165 epo=0 ts=1606377023.585707175,1 min=1606377023.085548018,0 seq=22} lock=true stat=PENDING rts=1606377023.085548018,0 wto=false max=1606377023.585548018,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table10 (col10_0 FLOAT8 NOT NULL, col10_1 BYTES, col10_2 BYTES NOT NULL, col10_3 INT8 NULL, col10_4 UUID[] NOT NULL, col10_5 REGNAMESPACE NULL, col10_6 "char" NOT NULL, col10_7 INTERVAL NOT NULL, col10_8 BOX2D NOT NULL, col10_9 DATE, col10_10 STRING NOT NULL, col10_11 FLOAT4 NOT NULL, col10_12 TIME NOT NULL, col10_13 INT8 NOT NULL, col10_14 BOOL, col10_15 FLOAT4, col10_16 TIMESTAMPTZ NOT NULL, PRIMARY KEY (col10_7 ASC, col10_6, col10_16 ASC, col10_10 ASC));
          |   ALTER TABLE public.table10 RENAME COLUMN "col10_15" TO "col10_20";
          |   CREATE SCHEMA schema25 AUTHORIZATION root;
          |   ALTER TABLE public.table10 RENAME COLUMN "col10_5" TO "col10_38";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=b6b78cfc key=/Table/SystemConfigSpan/Start pri=0.01041465 epo=0 ts=1606377023.585707175,2 min=1606377023.085957447,0 seq=39} lock=true stat=PENDING rts=1606377023.085957447,0 wto=false max=1606377023.585957447,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   CREATE SCHEMA schema12 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=ee5cbecb key=/Table/SystemConfigSpan/Start pri=0.00914489 epo=0 ts=1606377023.677143525,2 min=1606377023.259288100,0 seq=6} lock=true stat=PENDING rts=1606377023.259288100,0 wto=false max=1606377023.259288100,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2475210&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@361215163c597bd1460bba65ca3298f37e29aacc:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/scbench/randomload/nodes=3/ops=10000/conc=20/run_1
    cluster.go:2214,schemachange_random_load.go:135,schemachange_random_load.go:79,test_runner.go:760: output in run_080917.870_n1_workload_run_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2477206-1606461694-22-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned: exit status 20
        (1) attached stack trace
          -- stack trace:
          | main.(*cluster).RunE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
          | main.(*cluster).Run
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:135
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:79
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_080917.870_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2477206-1606461694-22-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201127 08:09:18.624104 1 workload/cli/run.go:357 ⋮ creating load generator...
          | I201127 08:09:18.698831 1 workload/cli/run.go:388 ⋮ creating load generator... done (took 74.725344ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=63e1222b key=/Table/SystemConfigSpan/Start pri=0.00579036 epo=0 ts=1606464558.822410474,1 min=1606464558.699804883,0 seq=6} lock=true stat=PENDING rts=1606464558.699804883,0 wto=false max=1606464558.699804883,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | BEGIN
          |   DROP SCHEMA "schema1" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: unknown schema "schema1" (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table5 ADD COLUMN IrrelevantColumnName string;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table5" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SEQUENCE "seq6";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=e8105b3a key=/Table/SystemConfigSpan/Start pri=0.00284566 epo=0 ts=1606464558.700755473,1 min=1606464558.700071284,0 seq=8} lock=true stat=PENDING rts=1606464558.700071284,0 wto=false max=1606464559.200071284,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE schema9.table10 ALTER COLUMN "IrrelevantColumnName" DROP NOT NULL;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "schema9.table10" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema8 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=63e1222b key=/Table/SystemConfigSpan/Start pri=0.00579036 epo=0 ts=1606464558.822410474,1 min=1606464558.699804883,0 seq=6} lock=true stat=PENDING rts=1606464558.699804883,0 wto=false max=1606464558.699804883,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2477206&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@54cdc5fec0e7dc835af7d2fc4231b52d49a71bf8:

          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_075720.445_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2479006-1606547424-22-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201128 07:57:21.248401 1 workload/cli/run.go:357 ⋮ creating load generator...
          | I201128 07:57:21.315259 1 workload/cli/run.go:388 ⋮ creating load generator... done (took 66.857537ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=8651becf key=/Table/SystemConfigSpan/Start pri=0.01395573 epo=0 ts=1606550242.018604636,1 min=1606550241.869040072,0 seq=6} lock=true stat=PENDING rts=1606550241.869040072,0 wto=false max=1606550241.869040072,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | saction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA schema10 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq19";
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq31";
          |   ALTER SEQUENCE "seq31" RENAME TO "seq40";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=d88e9cb9 key=/Table/SystemConfigSpan/Start pri=0.06471006 epo=0 ts=1606550241.481791548,1 min=1606550241.316751825,0 seq=43} lock=true stat=PENDING rts=1606550241.316751825,0 wto=false max=1606550241.816751825,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE "seq4";
          |   CREATE SEQUENCE "seq46";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table5 (col5_0 FLOAT8, col5_1 DATE, col5_2 BOOL NULL, col5_3 BIT(42) NOT NULL, col5_4 DATE NULL, col5_5 "char" NULL, col5_6 REGNAMESPACE NOT NULL, col5_7 TIMETZ, col5_8 DECIMAL NOT NULL, col5_9 REGNAMESPACE NOT NULL, col5_10 REGTYPE NOT NULL, col5_11 INT2, col5_12 BOOL NOT NULL, col5_13 INET NULL, col5_14 DATE NULL, col5_15 CHAR NOT NULL, col5_16 UUID NOT NULL, col5_17 INT8 NOT NULL, col5_18 BOX2D NULL, PRIMARY KEY (col5_15 ASC, col5_10 ASC, col5_12 DESC, col5_16 ASC, col5_8 DESC, col5_17 ASC, col5_6), INDEX (col5_9 DESC, col5_11 DESC, col5_2 DESC, col5_4 DESC, col5_7 DESC, col5_6 ASC, col5_1 DESC, col5_5 ASC, col5_17 ASC), UNIQUE (col5_6 DESC, col5_17 ASC, col5_8 DESC, col5_14 DESC, col5_16, col5_7, col5_3 DESC, col5_18 ASC, col5_12 ASC, col5_9 DESC) STORING (col5_4, col5_11), UNIQUE (col5_17 ASC, col5_1 DESC, col5_6, col5_7, col5_10 DESC), INDEX (col5_14 ASC, col5_3 ASC, col5_7 DESC, col5_2, col5_8 DESC, col5_12, col5_9 DESC, col5_15, col5_0 DESC, col5_10, col5_18, col5_11 DESC, col5_16 DESC, col5_17 DESC, col5_13, col5_4 ASC, col5_5 DESC), INDEX (col5_3 DESC, col5_9 DESC, col5_12 DESC, col5_15) STORING (col5_1, col5_2, col5_4, col5_5, col5_7, col5_11, col5_14, col5_18), INDEX (col5_4 DESC, col5_18, col5_0 DESC, col5_11 DESC, col5_2, col5_17, col5_1 DESC, col5_6, col5_13, col5_8, col5_14 DESC, col5_16 ASC), INDEX (col5_8 ASC, col5_9, col5_5 ASC, col5_15 DESC, col5_1, col5_3, col5_7 DESC, col5_0 ASC, col5_10, col5_6, col5_17 DESC, col5_16 ASC, col5_4 DESC, col5_14 DESC, col5_12 ASC) STORING (col5_11, col5_18), UNIQUE (col5_9 DESC, col5_7 ASC, col5_13 ASC, col5_5, col5_4, col5_18 ASC, col5_15, col5_3 ASC, col5_10, col5_2 DESC, col5_11 DESC, col5_6 ASC, col5_17 DESC, col5_8 ASC, col5_14));
          |   CREATE SCHEMA IF NOT EXISTS schema16 AUTHORIZATION root;
          |   DROP INDEX public.table5@"table5_col5_6_col5_17_col5_8_col5_14_col5_16_col5_7_col5_3_col5_18_col5_12_col5_9_key";
          | ROLLBACK; ERROR: index "table5_col5_6_col5_17_col5_8_col5_14_col5_16_col5_7_col5_3_col5_18_col5_12_col5_9_key" is in use as unique constraint (SQLSTATE 2BP01)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema21 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq48";
          |   DROP SCHEMA "schema21" CASCADE;
          |   DROP SEQUENCE "seq48";
          |   CREATE SCHEMA IF NOT EXISTS schema49 AUTHORIZATION root;
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=56716955 key=/Table/SystemConfigSpan/Start pri=0.00054943 epo=0 ts=1606550241.847108955,1 min=1606550241.316331314,0 seq=64} lock=true stat=PENDING rts=1606550241.316331314,0 wto=false max=1606550241.816331314,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT;  
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema52 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=8651becf key=/Table/SystemConfigSpan/Start pri=0.01395573 epo=0 ts=1606550242.018604636,1 min=1606550241.869040072,0 seq=6} lock=true stat=PENDING rts=1606550241.869040072,0 wto=false max=1606550241.869040072,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2479006&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@76935fd62c5a76f88b754cd3f9a5bfb3ccf1d8c2:

        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2480413-1606719755-22-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201130 08:08:18.181510 1 workload/cli/run.go:357 ⋮ creating load generator...
          | I201130 08:08:18.251786 1 workload/cli/run.go:388 ⋮ creating load generator... done (took 70.274769ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=2c393da2 key=/Table/SystemConfigSpan/Start pri=0.00837734 epo=0 ts=1606723699.193175194,2 min=1606723698.374475693,0 seq=15} lock=true stat=PENDING rts=1606723698.374475693,0 wto=false max=1606723698.374475693,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | REATE TABLE schema58.table59 (col59_0 BIT(25) NULL, col59_1 TIMESTAMPTZ NOT NULL, col59_2 VARCHAR NULL, col59_3 STRING NOT NULL, col59_4 DATE, col59_5 BIT(45) NOT NULL, col59_6 VARCHAR NOT NULL, col59_7 REGCLASS NOT NULL, col59_8 BYTES NOT NULL, col59_9 REGTYPE NULL, PRIMARY KEY (col59_1 DESC, col59_3 ASC, col59_8 ASC, col59_6 ASC), INDEX (col59_5, col59_0 ASC, col59_8 DESC, col59_2, col59_4), UNIQUE (col59_0 DESC, col59_6, col59_8 ASC, col59_3 ASC, col59_7, col59_9 DESC, col59_2 DESC, col59_5 ASC), UNIQUE (col59_4 ASC, col59_9 DESC, col59_1, col59_7 DESC, col59_0, col59_2 ASC, col59_8 DESC, col59_3 ASC), UNIQUE (col59_8 ASC, col59_6 ASC, col59_2 DESC, col59_9, col59_0 ASC), UNIQUE (col59_8 DESC, col59_3) STORING (col59_0, col59_4, col59_7), UNIQUE (col59_3 DESC, col59_6 DESC, col59_7, col59_2 ASC, col59_9 ASC, col59_0 DESC, col59_1, col59_4, col59_8 DESC));
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema58.table59" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq54";
          |   DROP SCHEMA "schema62" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: unknown schema "schema62" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TYPE "public.enum51" AS ENUM ('ss', 'adff', 'dd', 'aadsd', 'dfssfd', 'fsdaf');
          |   CREATE TYPE "public.enum63" AS ENUM ('aaa', 'sasdff', 'sfaf', 'fadds', 'a', 'dsa');
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=03c2fab4 key=/Table/SystemConfigSpan/Start pri=0.02842629 epo=0 ts=1606723698.643974897,1 min=1606723698.635900674,0 seq=18} lock=true stat=PENDING rts=1606723698.635900674,0 wto=false max=1606723699.135900674,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER SEQUENCE "seq21" RENAME TO "seq52";
          |   ALTER SEQUENCE "seq52" RENAME TO "seq61";
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=9d4bfcf4 key=/Table/SystemConfigSpan/Start pri=0.00345358 epo=0 ts=1606723698.643974897,1 min=1606723698.616648263,0 seq=21} lock=true stat=PENDING rts=1606723698.616648263,0 wto=false max=1606723699.116648263,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema41 AUTHORIZATION root;
          |   DROP SEQUENCE "seq21";
          |   SELECT 'validating all objects';
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=04799b46 key=/Table/SystemConfigSpan/Start pri=0.01756754 epo=0 ts=1606723698.970210729,1 min=1606723698.643974897,0 seq=24} lock=true stat=PENDING rts=1606723698.643974897,0 wto=false max=1606723699.143974897,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum22" AS ENUM ('fas');
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA schema50 AUTHORIZATION root;
          |   CREATE SEQUENCE "seq65";
          |   DROP SEQUENCE "seq65";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=db16b0f3 key=/Table/SystemConfigSpan/Start pri=0.01862324 epo=0 ts=1606723699.060393227,1 min=1606723698.464345958,0 seq=43} lock=true stat=PENDING rts=1606723698.464345958,0 wto=false max=1606723698.964345958,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum55" AS ENUM ('afdfa', 'afdsfs', 'affas', 'd', 'sas');
          |   CREATE SCHEMA IF NOT EXISTS schema64 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=2c393da2 key=/Table/SystemConfigSpan/Start pri=0.00837734 epo=0 ts=1606723699.193175194,2 min=1606723698.374475693,0 seq=15} lock=true stat=PENDING rts=1606723698.374475693,0 wto=false max=1606723698.374475693,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2480413&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@22906f72e795f9f2c69828e65194f4177833ffbb:

          -- stack trace:
          | main.(*cluster).RunE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
          | main.(*cluster).Run
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:135
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:79
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_081516.856_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2483915-1606806534-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201201 08:15:17.590541 1 workload/cli/run.go:357 ⋮ creating load generator...
          | I201201 08:15:17.675122 1 workload/cli/run.go:388 ⋮ creating load generator... done (took 84.58ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=4d537753 key=/Table/SystemConfigSpan/Start pri=0.00647125 epo=0 ts=1606810518.703489187,1 min=1606810518.543872186,0 seq=16} lock=true stat=ABORTED rts=1606810518.543872186,0 wto=false max=1606810518.543872186,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 984-10-03':::DATE,775374766:::OID,707411852:::OID,'BOX(-0.6488833066339699 -0.18885067729257568,0.34076428930839964 0.8010428806020481)':::BOX2D,'b3384b99-cb48-48db-a121-49a941f7a43c':::UUID,'04:05:40.353217':::TIME,(-5197):::INT8,0:::OID,3378988708:::OID,B'1010010101010000111001100100100111010001001000',0.1645319014787674:::FLOAT8),('BOX(-0.4707503929679413 -0.9098127738538041,0.11245774638252681 0.8530711431761989)':::BOX2D,1089149579:::INT8,'04:03:59.234935+11:21:00':::TIMETZ,'-27 years -1 mons -835 days -08:06:41.397077':::INTERVAL,'1983-09-13':::DATE,43459221:::OID,977580468:::OID,'BOX(-1.621821569560744 -1.1195902791652277,-0.6703148557079944 0.196299282738371)':::BOX2D,'9743babe-c4ce-4956-bdbf-a3ac63b3422d':::UUID,'17:06:50.457427':::TIME,(-24767):::INT8,665851281:::OID,2448934551:::OID,B'0101010000011011000000101111001000011000111011',2.1298155784606934:::FLOAT8),('BOX(-1.5111181453228497 -1.1973434576824058,0.6955979721993029 -0.13012140909077297)':::BOX2D,(-2104024565):::INT8,'13:00:55.286972-07:52:00':::TIMETZ,'53 years 9 mons 313 days 14:58:58.703716':::INTERVAL,'1985-08-09':::DATE,2393539416:::OID,1788124250:::OID,'BOX(0.7990524102524375 -0.5616301563004872,0.8819278278081939 0.43371306796295)':::BOX2D,'8c0d6f6b-7181-4968-be2d-80d7c8b903a6':::UUID,'10:50:01.61855':::TIME,(-1):::INT8,0:::OID,3992173434:::OID,B'1111000001110110011111010001110001110111001111',1.0:::FLOAT8),('BOX(-0.2766729586918272 -2.3412814285726244,-0.20130393438491917 0.6388621213887528)':::BOX2D,(-1368534191):::INT8,'00:00:00+15:59:00':::TIMETZ,'55 years 6 mons 815 days 04:45:10.408112':::INTERVAL,'1985-03-20':::DATE,2224270588:::OID,1272320116:::OID,'BOX(-10 -10,10 10)':::BOX2D,'a5e4418b-4fb9-4a07-8616-ab26104ec775':::UUID,'10:25:01.05473':::TIME,(-1152):::INT8,0:::OID,313958433:::OID,B'1000010100011000100001110011010010111100101010',(-1.1200512647628784):::FLOAT8);
          | ROLLBACK; ERROR: bit string length 64 does not match type BIT(46) (SQLSTATE 22026)
          | BEGIN
          |   ALTER TABLE public.table28 ALTER COLUMN "col28_6" SET DATA TYPE timestamp;
          | ROLLBACK; ERROR: cannot convert REGNAMESPACE to TIMESTAMP (SQLSTATE 42846)
          | BEGIN
          |   ALTER SEQUENCE "seq39" RENAME TO "seq39";
          |   ALTER SEQUENCE "seq39" RENAME TO "seq64";
          |   CREATE TYPE "public.enum66" AS ENUM ('as', 'dd', 'ff', 'fdsd', 'd');
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=291d526e key=/Table/SystemConfigSpan/Start pri=0.00620158 epo=0 ts=1606810518.611997734,1 min=1606810518.505706386,0 seq=21} lock=true stat=PENDING rts=1606810518.505706386,0 wto=false max=1606810519.005706386,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum55" AS ENUM ('f');
          |   CREATE VIEW public.view68 AS SELECT public.table28.col28_8, public.table28.col28_5, public.table28.col28_13, public.table28.col28_10, public.table28.col28_9, public.table28.col28_11, public.table28.col28_6, public.table28.col28_2, public.table28.col28_3, public.table28.col28_13, public.table28.col28_0, public.table28.col28_2, public.table28.col28_12, public.table28.col28_5, public.table28.col28_8, public.table28.col28_9, public.table28.col28_7, public.table28.col28_13, public.table28.col28_12, public.table28.col28_0, public.table28.col28_4, public.table28.col28_14, public.table28.col28_10, public.table28.col28_5, public.table28.col28_3 FROM public.table28, public.table28, public.table28;
          | ROLLBACK; expected SQLSTATE(S) 42701,42712, and got ERROR: source name "table28" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   DROP TABLE public.table28;
          |   CREATE SCHEMA IF NOT EXISTS schema65 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "sql txn" meta={id=4d537753 key=/Table/SystemConfigSpan/Start pri=0.00647125 epo=0 ts=1606810518.703489187,1 min=1606810518.543872186,0 seq=16} lock=true stat=ABORTED rts=1606810518.543872186,0 wto=false max=1606810518.543872186,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2483915&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@5af065eeb4f520ee93901e75fbef5d877f06585c:

          | stderr:
          | I201202 07:49:56.565504 1 workload/cli/run.go:357 ⋮ creating load generator...
          | I201202 07:49:56.656033 1 workload/cli/run.go:388 ⋮ creating load generator... done (took 90.528215ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=57b17861 key=/Table/SystemConfigSpan/Start pri=0.00889665 epo=0 ts=1606895399.310041933,1 min=1606895398.791392091,0 seq=6} lock=true stat=PENDING rts=1606895398.791392091,0 wto=false max=1606895398.791392091,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          |  FAMILY (col87_10), FAMILY (col87_2, col87_3, col87_11), FAMILY (col87_14), FAMILY (col87_9), FAMILY (col87_13), FAMILY (col87_0), FAMILY (col87_6), FAMILY (col87_12, col87_4, col87_7));
          |   ALTER TABLE public.table87 RENAME COLUMN "col87_13" TO "col87_94";
          |   ALTER TABLE public.table87 RENAME TO public.table98;
          | COMMIT;  
          | BEGIN
          |   CREATE TABLE schema123.table158 (col158_0 CHAR NOT NULL, col158_1 DATE, col158_2 FLOAT8[] NULL, col158_3 CHAR, col158_4 REGPROC NOT NULL, col158_5 BOOL NOT NULL, col158_6 INTERVAL NOT NULL, col158_7 TIME NOT NULL, col158_8 TIMESTAMP NOT NULL, col158_9 OID NOT NULL, col158_10 REGCLASS NULL, col158_11 INT8 NULL, col158_12 INT2 NOT NULL, col158_13 VARCHAR NOT NULL, col158_14 TIMESTAMPTZ NOT NULL, col158_15 VARCHAR NOT NULL, col158_16 DATE NOT NULL, col158_17 "char" NOT NULL, PRIMARY KEY (col158_14 ASC, col158_6, col158_15, col158_0, col158_9, col158_4 ASC, col158_16 DESC, col158_12 ASC, col158_13 ASC, col158_5 ASC, col158_7, col158_8 DESC, col158_17), INDEX (col158_9, col158_16, col158_4 DESC, col158_1 DESC, col158_6 ASC) STORING (col158_3, col158_10), INDEX (col158_12, col158_3 DESC, col158_4 DESC, col158_17 ASC, col158_6 DESC, col158_13, col158_14, col158_16), UNIQUE (col158_6 ASC, col158_13, col158_3, col158_16 DESC, col158_7 DESC), INDEX (col158_1, col158_13 ASC, col158_9 ASC, col158_3 ASC, col158_14, col158_15 ASC, col158_10, col158_16, col158_17 ASC, col158_6 DESC, col158_0 ASC, col158_7, col158_5 ASC), UNIQUE (col158_1 DESC, col158_8 DESC, col158_6 DESC, col158_3 DESC, col158_14), INDEX (col158_14 DESC, col158_15, col158_13, col158_11 ASC, col158_12 DESC, col158_17 DESC, col158_9, col158_0, col158_6 ASC, col158_1 DESC, col158_3 ASC, col158_16 DESC, col158_7 DESC, col158_10, col158_5 ASC, col158_8 ASC));
          |   CREATE TABLE IF NOT EXISTS schema123.table164 (col164_0 VARCHAR, col164_1 OID NOT NULL, col164_2 DECIMAL NOT NULL, col164_3 REGCLASS NOT NULL, PRIMARY KEY (col164_3 ASC, col164_1 ASC, col164_2 DESC), INDEX (col164_3 ASC, col164_2 ASC, col164_0 ASC), UNIQUE (col164_0), INDEX (col164_0 DESC, col164_1 ASC, col164_2 DESC, col164_3), UNIQUE (col164_2 ASC, col164_0), FAMILY (col164_3), FAMILY (col164_1, col164_2), FAMILY (col164_0));
          |   DROP INDEX schema123.table164@"table164_col164_0_key";
          | ROLLBACK; ERROR: index "table164_col164_0_key" is in use as unique constraint (SQLSTATE 2BP01)
          | BEGIN
          |   ALTER TABLE public.table98 DROP CONSTRAINT "primary";
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq173";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE public.table98 RENAME TO public.table150;
          |   ALTER TABLE public.table150 ALTER COLUMN "col150_175" SET NOT NULL;
          | ROLLBACK; ERROR: column "col150_175" does not exist (SQLSTATE 42703)
          | BEGIN
          |   CREATE SEQUENCE "seq176";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          |     3.0s        0           34.0           38.7     14.7    335.5    419.4    419.4 opOk
          |     3.0s        0            3.0            5.3    738.2   1040.2   1040.2   1040.2 txnCmtErr
          |     3.0s        0            3.0            2.0    805.3   2550.1   2550.1   2550.1 txnOk
          |     3.0s        0           37.0           33.7      4.7     21.0     33.6     33.6 txnRbk
          | BEGIN
          |   ALTER TABLE public.table98 ALTER COLUMN "col87_7" SET NOT NULL;
          |   ALTER TABLE public.table98 RENAME TO public.table178;
          |   CREATE SEQUENCE "seq180";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE public.table98 ALTER COLUMN "col87_5" DROP DEFAULT;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=57b17861 key=/Table/SystemConfigSpan/Start pri=0.00889665 epo=0 ts=1606895399.310041933,1 min=1606895398.791392091,0 seq=6} lock=true stat=PENDING rts=1606895398.791392091,0 wto=false max=1606895398.791392091,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2488192&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@677f6f89f97d492f5eb443fcdb326d5695ddc5d7:

          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_074623.689_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2491302-1606978739-22-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201203 07:46:24.411657 1 workload/cli/run.go:359  creating load generator...
          | I201203 07:46:24.491707 1 workload/cli/run.go:390  creating load generator... done (took 80.048901ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=2720f8b0 key=/Table/SystemConfigSpan/Start pri=0.02795318 epo=0 ts=1606981585.488312421,1 min=1606981585.172597503,0 seq=6} lock=true stat=PENDING rts=1606981585.172597503,0 wto=false max=1606981585.172597503,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | 2_6, col62_13, col62_10 DESC), INDEX (col62_11 ASC, col62_7 DESC, col62_10 ASC, col62_0 DESC, col62_14, col62_4 DESC, col62_9 ASC, col62_6 ASC, col62_15 DESC, col62_8 DESC, col62_12, col62_13 DESC), INDEX (col62_7 DESC, col62_13 ASC, col62_0 ASC, col62_14, col62_4 DESC, col62_6, col62_15 ASC, col62_2 ASC, col62_8 DESC, col62_10 DESC) STORING (col62_1), INDEX (col62_11, col62_12, col62_6 DESC, col62_4 ASC, col62_8, col62_2 DESC, col62_15) STORING (col62_1, col62_5, col62_7, col62_9, col62_13), UNIQUE (col62_6 DESC, col62_12 DESC, col62_7, col62_8 ASC, col62_10, col62_15 ASC, col62_13 DESC, col62_2 ASC, col62_9));
          |   ALTER TABLE public.table62 RENAME COLUMN "col62_15" TO "col62_69";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=3f787a98 key=/Table/SystemConfigSpan/Start pri=0.04472858 epo=0 ts=1606981585.311412560,1 min=1606981584.741204785,0 seq=27} lock=true stat=PENDING rts=1606981584.741204785,0 wto=false max=1606981585.241204785,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table71 ADD COLUMN IrrelevantColumnName string;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table71" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TYPE "schema54.enum55" AS ENUM ('ss');
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=da896ec2 key=/Table/SystemConfigSpan/Start pri=0.00870726 epo=0 ts=1606981585.330674366,1 min=1606981585.103797904,0 seq=9} lock=true stat=PENDING rts=1606981585.103797904,0 wto=false max=1606981585.603797904,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS schema77.table78 (col78_0 TIMESTAMP NOT NULL, col78_1 INT4 NOT NULL, col78_2 INET NOT NULL, col78_3 BYTES NOT NULL, col78_4 UUID, col78_5 REGNAMESPACE NOT NULL, col78_6 DECIMAL NOT NULL, PRIMARY KEY (col78_2 DESC, col78_6, col78_0 ASC, col78_3 ASC, col78_1, col78_5 DESC), INDEX (col78_3 ASC, col78_5, col78_6 DESC, col78_4, col78_2 ASC), UNIQUE (col78_0 ASC, col78_4 DESC, col78_1));
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema77.table78" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE "seq79";
          | COMMIT;  ***FAIL: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=2d624306 key=/Table/SystemConfigSpan/Start pri=0.05176365 epo=0 ts=1606981585.330674366,1 min=1606981585.052497277,0 seq=8} lock=true stat=PENDING rts=1606981585.052497277,0 wto=false max=1606981585.552497277,0 (SQLSTATE 40001)
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |     1.0s        0           50.0           54.0     13.1     58.7     71.3     71.3 opOk
          |     1.0s        0           10.2           11.0    335.5    637.5    704.6    704.6 txnCmtErr
          |     1.0s        0            3.7            4.0    121.6    419.4    419.4    419.4 txnOk
          |     1.0s        0           28.7           31.0      3.1     14.7     24.1     24.1 txnRbk
          | BEGIN
          |   ALTER TABLE public.table67 RENAME TO public.table68;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table67" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema81 AUTHORIZATION root;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=2720f8b0 key=/Table/SystemConfigSpan/Start pri=0.02795318 epo=0 ts=1606981585.488312421,1 min=1606981585.172597503,0 seq=6} lock=true stat=PENDING rts=1606981585.172597503,0 wto=false max=1606981585.172597503,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2491302&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) this will be fixed by the types refactor in https://github.com/cockroachdb/cockroach/pull/57491

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@ce3f9b29fee565a2994ca84d2ecd20db7fe59d0b:

        Wraps: (2) output in run_075133.613_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2498144-1607151733-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201205 07:51:34.330289 1 workload/cli/run.go:359  creating load generator...
          | I201205 07:51:34.394125 1 workload/cli/run.go:390  creating load generator... done (took 63.834939ms)
          | Error: ***UNEXPECTED SUCCESS
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   INSERT INTO public.table6 (IrrelevantColumnName) VALUES ("IrrelevantValue");
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table6" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   DROP VIEW public.view10 CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.view10" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table16 RENAME COLUMN "IrrelevantColumnName" TO "OtherIrrelevantName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table16" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS public AUTHORIZATION root;
          |   CREATE SCHEMA schema8 AUTHORIZATION root;
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=c9ccbf3a key=/Table/SystemConfigSpan/Start pri=0.02864394 epo=0 ts=1607154694.481357208,1 min=1607154694.395427231,0 seq=13} lock=true stat=PENDING rts=1607154694.395427231,0 wto=false max=1607154694.895427231,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE public.table13 (col13_0 CHAR, col13_1 TIME NOT NULL, col13_2 INT8 NOT NULL, col13_3 TIMETZ NOT NULL, col13_4 BOOL NOT NULL, col13_5 INT4 NOT NULL, col13_6 REGPROC NOT NULL, col13_7 BIT(8) NOT NULL, col13_8 INT4 NOT NULL, col13_9 REGTYPE NOT NULL, col13_10 "char" NOT NULL, col13_11 FLOAT8 NOT NULL, PRIMARY KEY (col13_7, col13_10 ASC, col13_6, col13_5, col13_8 DESC, col13_2 ASC, col13_4, col13_3, col13_1 DESC, col13_11 ASC, col13_9), INDEX (col13_7, col13_10 ASC, col13_3 DESC, col13_8, col13_0), INDEX (col13_9 DESC, col13_11 DESC, col13_2, col13_6, col13_10 DESC, col13_5 DESC, col13_4, col13_0 ASC, col13_7 ASC, col13_8 ASC, col13_3 ASC, col13_1 DESC), UNIQUE (col13_8 DESC, col13_2, col13_11 DESC, col13_10 ASC, col13_5 ASC, col13_6, col13_9, col13_0 DESC), UNIQUE (col13_4 ASC, col13_6, col13_2 ASC, col13_0, col13_7 DESC, col13_9, col13_3), INDEX (col13_7 ASC, col13_11 ASC, col13_8 DESC, col13_2, col13_4 ASC) STORING (col13_0), INDEX (col13_4, col13_7, col13_6 ASC, col13_8 DESC, col13_2, col13_0, col13_3 ASC, col13_9 ASC, col13_10, col13_1 ASC, col13_11, col13_5 DESC), UNIQUE (col13_2 DESC, col13_3 DESC, col13_1 ASC, col13_4 DESC, col13_0, col13_8 DESC, col13_7 DESC, col13_6 ASC, col13_9, col13_5 ASC, col13_11 ASC), UNIQUE (col13_7 ASC, col13_0 DESC, col13_6, col13_11), INDEX (col13_6 ASC, col13_10 ASC, col13_5 ASC, col13_9, col13_0 DESC, col13_4, col13_7 DESC));
          |   CREATE SEQUENCE IF NOT EXISTS public.seq17;
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=c88e7282 key=/Table/SystemConfigSpan/Start pri=0.00020633 epo=0 ts=1607154694.420156348,1 min=1607154694.395874474,0 seq=15} lock=true stat=PENDING rts=1607154694.395874474,0 wto=false max=1607154694.895874474,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum23" AS ENUM ('faf', 'adsad', 's');
          | Expected SQLSTATE(S) 3F000, but got no errors
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2498144&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57476

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@1ed669de1c3b77798fade7ad9f056edc0bf27c36:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
          | main.(*cluster).Run
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:135
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:79
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_075228.457_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2500105-1607324629-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201207 07:52:29.165782 1 workload/cli/run.go:359  creating load generator...
          | I201207 07:52:29.221848 1 workload/cli/run.go:390  creating load generator... done (took 56.065308ms)
          | Error: ***UNEXPECTED SUCCESS
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ol12038_16 BIT(37) NOT NULL, col12038_17 STRING NOT NULL, col12038_18 REGNAMESPACE, UNIQUE (col12038_8 ASC, col12038_2 ASC, col12038_0, col12038_9, col12038_4, col12038_10 DESC, col12038_18 ASC, col12038_7, col12038_16, col12038_13 ASC, col12038_12 ASC, col12038_1, col12038_5) STORING (col12038_6, col12038_11, col12038_14, col12038_15), INDEX (col12038_4, col12038_1 DESC, col12038_5 DESC, col12038_0, col12038_11 DESC, col12038_8 ASC, col12038_9, col12038_18 DESC, col12038_16 DESC, col12038_6 ASC), UNIQUE (col12038_10, col12038_7, col12038_14 ASC, col12038_18, col12038_12 DESC, col12038_16, col12038_17 DESC, col12038_15, col12038_0 ASC, col12038_2 ASC, col12038_4, col12038_13 DESC, col12038_1 DESC), INDEX (col12038_17, col12038_11 ASC, col12038_7, col12038_16 DESC, col12038_14 ASC, col12038_12, col12038_5 ASC, col12038_0 DESC, col12038_6 DESC, col12038_18 DESC, col12038_4 ASC, col12038_9 DESC, col12038_10 ASC, col12038_1 DESC) STORING (col12038_3, col12038_15));
          |   ALTER SEQUENCE schema57.seq4680 RENAME TO schema57.seq12044;
          |   DROP TABLE schema57.table2985 CASCADE;
          |   SELECT 'validating all objects';
          |   ALTER TABLE schema1.table2979 ALTER COLUMN "col2979_1" SET NOT NULL;
          | COMMIT; 
          | BEGIN
          |   ALTER TABLE public.table12038 ALTER COLUMN "col12038_8" SET NOT NULL;
          |   ALTER TABLE public.table12038 ALTER COLUMN "col12038_13" SET NOT NULL;
          |   ALTER TABLE public.table12266 RENAME COLUMN "IrrelevantColumnName" TO "OtherIrrelevantName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table12266" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table5726 RENAME CONSTRAINT "table5726_col5726_9_col5726_2_col5726_6_col5726_5_col5726_12_col5726_14_col5726_0_col5726_16_key" TO "index5726_12245";
          |   CREATE TABLE schema57.table12274 AS SELECT public.table12038.col12038_9, public.table12038.col12038_4, public.table12038.col12038_2, public.table12038.col12038_18, public.table4683.col4683_3, public.table4683.col4683_1, public.table4683.col4683_0, public.table4683.col4683_2, public.table12038.col12038_11, public.table12038.col12038_1 FROM public.table12038, public.table4683, public.table12038;
          | ROLLBACK; expected SQLSTATE(S) 42712, and got ERROR: source name "table12038" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE public.table12038 RENAME COLUMN "col12038_2" TO "col12038_12246";
          |   INSERT INTO public.table5726 (col5726_0,col5726_1,col5726_2,col5726_3,col5726_4,col5726_5,col5726_6,col5726_7,col5726_8,col5726_9,col5726_10,col5726_11,col5726_12,col5726_13,col5726_14,col5726_15,col5726_16,col5726_17) VALUES (930415411:::INT8,489369052:::INT8,771633825:::OID,'\x94af8277a47723':::BYTES,'72 years 9 mons 235 days 07:59:07.540897':::INTERVAL,'z1kJ':::STRING:::NAME,'!':::STRING,e'\x063,b*\x14':::STRING,'-35 years -1 mons -877 days -19:51:51.655843':::INTERVAL,'294276-12-31 23:59:59.999999':::TIMESTAMP,757898593:::OID,e'm6\x16v\x19~':::STRING:::NAME,B'0100100100101100101001110010110101011001','R':::STRING,1745203299:::OID,(-2.60227632522583):::FLOAT8,B'01111101111100110110100010001',e'l\x0b\x05\x0fd\x05n':::STRING:::NAME),((-1948212752):::INT8,(-537015906):::INT8,18245588:::OID,'\xff':::BYTES,'41 years 9 mons 951 days 01:49:30.645802':::INTERVAL,e'\\\x10.A%':::STRING:::NAME,' ':::STRING,e'\r@&,2S':::STRING,'14 years 11 mons 722 days 03:39:13.033147':::INTERVAL,'1975-10-11 18:50:41.000828':::TIMESTAMP,4228661784:::OID,e'\x18/:7\x13<Zl|':::STRING:::NAME,B'1011011000110001101100011000001001001111','P':::STRING,943347780:::OID,'NaN':::FLOAT8,B'01101101000011101000010001101','Z?X*':::STRING:::NAME),((-1629198518):::INT8,1007405633:::INT8,0:::OID,'\x':::BYTES,'-55 years -7 mons -732 days -07:23:55.66857':::INTERVAL,'P':::STRING:::NAME,e'\\':::STRING,e'nV;j\bZ':::STRING,'1 day':::INTERVAL,'1981-06-07 10:11:18.000025':::TIMESTAMP,370637285:::OID,'X':::STRING,B'0000000000000000000000000000000000000000','k':::STRING,1085311576:::OID,(-2.0210633277893066):::FLOAT8,B'11101100101101111000010100110',NULL);
          |   ALTER TABLE public.table4683 RENAME TO public.table12292;
          | Expected SQLSTATE(S) 0A000, but got no errors
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2500105&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57476

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@9b2aeea6ca553f79dd737052ac23c91763a0b713:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
          | main.(*cluster).Run
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:135
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:79
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_075324.676_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2503533-1607411015-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201208 07:53:25.479632 1 workload/cli/run.go:359  creating load generator...
          | I201208 07:53:25.561310 1 workload/cli/run.go:390  creating load generator... done (took 81.675578ms)
          | Error: ***UNEXPECTED SUCCESS
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | T, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
          |   CREATE VIEW public.view12727 AS SELECT schema1.table9275.col9275_13, schema1.table9275.col9275_6, schema1.table9275.col9275_18, schema1.table9275.col9275_15, schema1.table9275.col9275_9, schema1.table9275.col9275_3, schema1.table9275.col9275_4, schema1.table9275.col9275_0, schema1.table9275.col9275_16, schema1.table9275.col9275_17, schema1.table9275.col9275_15, schema1.table9275.col9275_2, schema1.table9275.col9275_5, schema1.table9275.col9275_6, schema1.table9275.col9275_11, schema1.table9275.col9275_10, schema1.table9275.col9275_18 FROM schema1.table9275, schema1.table9275;
          | ROLLBACK; expected SQLSTATE(S) 42701,42712, and got ERROR: source name "table9275" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE schema1.table9275 ADD COLUMN col9275_12701 TIMESTAMP NULL;
          |   ALTER TABLE schema1.table9275 RENAME TO schema1.table9275;
          |   CREATE VIEW public.view12734 AS SELECT schema1.table9275.col9275_18, schema1.table9275.col9275_7, schema1.table9275.col9275_13, schema1.table9275.col9275_6, schema1.table9275.col9275_15, schema1.table9275.col9275_11, schema1.table9275.col9275_6, schema1.table9275.col9275_0, schema1.table9275.col9275_2, schema1.table9275.col9275_12, schema1.table9275.col9275_17, schema1.table9275.col9275_5, schema1.table9275.col9275_1, schema1.table9275.col9275_10, schema1.table9275.col9275_3, schema1.table9275.col9275_18, schema1.table9275.col9275_7, schema1.table9275.col9275_14, schema1.table9275.col9275_8, schema1.table9275.col9275_13, schema1.table9275.col9275_9, schema1.table9275.col9275_16 FROM schema1.table9275, schema1.table9275;
          | ROLLBACK; expected SQLSTATE(S) 42701,42712, and got ERROR: source name "table9275" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   ALTER TABLE schema1.table9275 ALTER COLUMN "col9275_12738" DROP DEFAULT;
          | ROLLBACK; expected SQLSTATE(S) 42703, and got ERROR: column "col9275_12738" does not exist (SQLSTATE 42703)
          | BEGIN
          |   ALTER TABLE schema1.table9275 DROP CONSTRAINT "table9275_col9275_14_col9275_9_col9275_17_col9275_10_col9275_1_col9275_3_col9275_13_col9275_11_col9275_7_col9275_8_col9275_5_col9275_0_col9275_15_col9275_18_key";
          | ROLLBACK; ERROR: unimplemented: cannot drop UNIQUE constraint "table9275_col9275_14_col9275_9_col9275_17_col9275_10_col9275_1_col9275_3_col9275_13_col9275_11_col9275_7_col9275_8_col9275_5_col9275_0_col9275_15_col9275_18_key" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS schema1.table12724 (col12724_0 REGNAMESPACE NULL, col12724_1 "char", col12724_2 REGTYPE NOT NULL, col12724_3 REGTYPE, col12724_4 INT8, col12724_5 REGTYPE NOT NULL, col12724_6 NAME NOT NULL, col12724_7 DECIMAL, col12724_8 BOX2D NOT NULL, col12724_9 BOOL, col12724_10 REGPROC NOT NULL, col12724_11 BIT(34) NULL, col12724_12 INT8, col12724_13 TIMESTAMPTZ NULL, col12724_14 UUID, col12724_15 TIMESTAMP NULL, col12724_16 NAME NOT NULL, PRIMARY KEY (col12724_8 DESC, col12724_2), FAMILY (col12724_0, col12724_9, col12724_6, col12724_14), FAMILY (col12724_16), FAMILY (col12724_8, col12724_5, col12724_7, col12724_13, col12724_2), FAMILY (col12724_3, col12724_4), FAMILY (col12724_11), FAMILY (col12724_15), FAMILY (col12724_12, col12724_1), FAMILY (col12724_10));
          |   INSERT INTO schema1.table12724 (col12724_0,col12724_1,col12724_2,col12724_3,col12724_4,col12724_5,col12724_6,col12724_7,col12724_8,col12724_9,col12724_10,col12724_11,col12724_12,col12724_13,col12724_14,col12724_15,col12724_16) VALUES (267490935:::OID,NULL,0:::OID,374489190:::OID,(-2755576570555515518):::INT8,89884584:::OID,e'y\x01rAI}38':::STRING:::NAME,5435340633362160.764:::DECIMAL,'BOX(-1.4308082999756229 -1.043836813729765,-1.160334465404027 -0.5401147946398116)':::BOX2D,false,188430331:::OID,B'1011101101111010010101111111110001',(-5884666928402398534):::INT8,NULL,'711e21ff-a7b7-4db5-aa2b-641c000b712f':::UUID,'2029-06-26 17:20:17.000538':::TIMESTAMP,e'0P\\\x05>eq\x7f':::STRING:::NAME);
          |   DROP SCHEMA "schema10116" CASCADE;
          | Expected SQLSTATE(S) 0A000, but got no errors
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2503533&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@258ef5765d0205f487b1481a9c26059db2a70362:

          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=17ab4c89 key=/Table/SystemConfigSpan/Start pri=0.00814991 epo=0 ts=1607502723.913592105,1 min=1607502714.140793310,0 seq=6} lock=true stat=PENDING rts=1607502714.140793310,0 wto=false max=1607502714.140793310,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE "public.enum559" AS ENUM ('d', 'afsd', 'dss');
          |   CREATE SCHEMA public AUTHORIZATION root;
          | ROLLBACK; expected SQLSTATE(S) 42P06, and got ERROR: schema "public" already exists (SQLSTATE 42P06)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT; 
          | BEGIN
          |   CREATE TYPE "public.enum563" AS ENUM ('aassaa', 'adsa', 'adfaa', 'saffdd', 'f', 'd');
          |   CREATE SCHEMA IF NOT EXISTS public AUTHORIZATION root;
          |   CREATE TABLE schema566.table567 (col567_0 TIMETZ NOT NULL, col567_1 CHAR NULL, col567_2 INT2 NOT NULL, col567_3 UUID NOT NULL, col567_4 UUID NOT NULL, col567_5 OID NOT NULL, col567_6 TIMESTAMPTZ NOT NULL, col567_7 DECIMAL, col567_8 REGNAMESPACE NOT NULL, col567_9 REGTYPE NOT NULL, col567_10 REGTYPE NOT NULL, PRIMARY KEY (col567_4 DESC, col567_9, col567_2, col567_0 ASC, col567_10, col567_8 ASC, col567_6, col567_5), UNIQUE (col567_1, col567_2, col567_4, col567_5, col567_3), UNIQUE (col567_1 ASC, col567_5 DESC, col567_6 ASC, col567_2, col567_7 DESC, col567_4 ASC, col567_9), UNIQUE (col567_4 ASC, col567_10 DESC, col567_3 ASC, col567_2 DESC, col567_8 DESC, col567_7, col567_1 ASC, col567_0 DESC, col567_5 DESC, col567_9, col567_6), UNIQUE (col567_1 DESC, col567_9 DESC, col567_5, col567_8 DESC, col567_10 DESC, col567_3 ASC, col567_6 DESC, col567_2 DESC, col567_4), UNIQUE (col567_7, col567_1 ASC, col567_3 DESC, col567_10 DESC), INDEX (col567_6, col567_2 DESC, col567_3 ASC, col567_7 DESC));
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema566.table567" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "schema577" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: unknown schema "schema577" (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table575 RENAME TO public.table576;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table575" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE IF NOT EXISTS public.seq585;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |    11.0s        0           13.0            3.5      5.5     37.7     37.7     37.7 opOk
          |    11.0s        0            2.0            0.6  10200.5  10200.5  10200.5  10200.5 txnCmtErr
          |    11.0s        0            2.0            0.3   8321.5  10200.5  10200.5  10200.5 txnOk
          |    11.0s        0           13.0            2.4      2.4     27.3     67.1     67.1 txnRbk
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE TYPE "schema2.enum3" AS ENUM ('saaf');
          |   DROP INDEX schema649.table650@"index650_651";
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE SCHEMA schema20 AUTHORIZATION root;
          | ROLLBACK; expected SQLSTATE(S) 42P06, and got ERROR: schema "schema20" already exists (SQLSTATE 42P06)
          | BEGIN
          |   DROP SCHEMA "schema20" CASCADE;
          | ***FAIL; Expected no errors, but got ERROR: unknown schema "schema20" (SQLSTATE 3F000)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2506254&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57476

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@c41a2ed503cd31d95bc3ee4365663811772c8bd3:

        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2509111-1607583926-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201210 07:56:01.619629 1 workload/cli/run.go:359  creating load generator...
          | I201210 07:56:01.676631 1 workload/cli/run.go:390  creating load generator... done (took 57.001143ms)
          | Error: ***UNEXPECTED SUCCESS
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | hema85" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq81;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TYPE "public.enum87" AS ENUM ('das', 'dfdf', 'ds');
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=b77c343a key=/Table/SystemConfigSpan/Start pri=0.00931621 epo=0 ts=1607586962.943296421,2 min=1607586961.996936544,0 seq=9} lock=true stat=PENDING rts=1607586961.996936544,0 wto=false max=1607586962.496936544,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq70;
          |   CREATE SEQUENCE IF NOT EXISTS public.seq84;
          |   ALTER VIEW public.view94 RENAME TO public.view95;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.view94" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE TYPE "public.enum91" AS ENUM ('ds', 'f', 'dff', 'sffad');
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=571fc665 key=/Table/SystemConfigSpan/Start pri=0.02196794 epo=0 ts=1607586962.943296421,2 min=1607586961.959778573,0 seq=9} lock=true stat=PENDING rts=1607586961.959778573,0 wto=false max=1607586962.459778573,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT; 
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq65;
          |   SELECT 'validating all objects';
          |   CREATE SEQUENCE IF NOT EXISTS public.seq96;
          |   CREATE TYPE "public.enum105" AS ENUM ('s', 'fdaa', 'ddfadf', 'affa', 'dfsf', 'dsddfd');
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=926fad58 key=/Table/SystemConfigSpan/Start pri=0.02125585 epo=0 ts=1607586962.943296421,2 min=1607586961.970658567,0 seq=25} lock=true stat=PENDING rts=1607586961.970658567,0 wto=false max=1607586962.470658567,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table69 (col69_0 REGTYPE NOT NULL, col69_1 DECIMAL NULL, col69_2 REGPROCEDURE NOT NULL, col69_3 VARCHAR NOT NULL, col69_4 STRING[] NOT NULL, col69_5 INT2 NOT NULL, col69_6 TIMESTAMPTZ NOT NULL, PRIMARY KEY (col69_0 ASC, col69_3 ASC, col69_5, col69_6, col69_2 ASC), INDEX (col69_0, col69_2, col69_3 ASC), UNIQUE (col69_1 ASC, col69_2 ASC, col69_5, col69_3 ASC, col69_0 ASC), UNIQUE (col69_1, col69_6, col69_5 ASC, col69_3, col69_2), FAMILY (col69_3), FAMILY (col69_5, col69_0), FAMILY (col69_4), FAMILY (col69_2), FAMILY (col69_6, col69_1));
          |   INSERT INTO public.table69 (col69_0,col69_1,col69_2,col69_3,col69_4,col69_5,col69_6) VALUES (2983400296:::OID,6966082705.410801493:::DECIMAL,1138980865:::OID,'a':::STRING,ARRAY['(.!':::STRING,'&_j':::STRING,'':::STRING],4520:::INT8,'2023-07-20 03:17:11.000031+00:00':::TIMESTAMPTZ),(1554715671:::OID,NULL,1070323081:::OID,e'O\x1fiK&S':::STRING,ARRAY['r&':::STRING,e'7^BsXl\x12.!':::STRING,'"':::STRING,e'\x18':::STRING,'"':::STRING,'%':::STRING],(-1):::INT8,'1999-08-24 22:03:13.000349+00:00':::TIMESTAMPTZ),(2968438030:::OID,(-61252473.1420808195):::DECIMAL,2581268386:::OID,e'\x00':::STRING,ARRAY[e'\x03\x13\\YI\x01\x0f+':::STRING,'':::STRING,e'\x19\x7f(\x1d\x19Z':::STRING],(-23145):::INT8,'2019-01-25 01:21:58.000824+00:00':::TIMESTAMPTZ);
          |   INSERT INTO public.table69 (col69_0,col69_1,col69_2,col69_3,col69_4,col69_5,col69_6) VALUES (2555204519:::OID,(-402227588250445.8788):::DECIMAL,2532119227:::OID,')':::STRING,ARRAY[]:::STRING[],(-26184):::INT8,'2011-04-28 09:36:04.000025+00:00':::TIMESTAMPTZ),(3147407899:::OID,16.38123265478540372:::DECIMAL,0:::OID,e'A\x03-\\\n\x1e\x1c\x17':::STRING,ARRAY[e'$\x13\x03{I\x05h':::STRING,e'\x11':::STRING,'a':::STRING],17616:::INT8,'2031-02-24 11:24:47.000754+00:00':::TIMESTAMPTZ);
          |   CREATE VIEW public.view107 AS SELECT public.table69.col69_5, public.table69.col69_0 FROM public.table69;
          | Expected SQLSTATE(S) 0A000, but got no errors
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2509111&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57493

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@960b4cfc54c8b78df56e62f07d7a07b986ceacff:

          |    65.0s        0            0.0            1.1      0.0      0.0      0.0      0.0 opOk
          |    65.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    65.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    65.0s        0            0.0            0.9      0.0      0.0      0.0      0.0 txnRbk
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |    66.0s        0            0.0            1.1      0.0      0.0      0.0      0.0 opOk
          |    66.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    66.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    66.0s        0            0.0            0.9      0.0      0.0      0.0      0.0 txnRbk
          |    67.0s        0            0.0            1.1      0.0      0.0      0.0      0.0 opOk
          |    67.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    67.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    67.0s        0            0.0            0.9      0.0      0.0      0.0      0.0 txnRbk
          |    68.0s        0            0.0            1.1      0.0      0.0      0.0      0.0 opOk
          |    68.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    68.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    68.0s        0            0.0            0.9      0.0      0.0      0.0      0.0 txnRbk
          | BEGIN
          |   ALTER TABLE public.table5 ALTER COLUMN "col5_3152" DROP NOT NULL;
          | ROLLBACK; expected SQLSTATE(S) 42703, and got ERROR: column "col5_3152" does not exist (SQLSTATE 42703)
          | BEGIN
          |   INSERT INTO public.table5 (col5_0) VALUES ('8':::STRING),('<':::STRING);
          |   ALTER TABLE public.table5 DROP CONSTRAINT "primary";
          | ROLLBACK; ERROR: unimplemented: schema change statement cannot follow a statement that has written in the same transaction (SQLSTATE 0A000)
          |    69.0s        0            1.0            1.1      2.0      2.0      2.0      2.0 opOk
          |    69.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    69.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    69.0s        0            2.0            0.9      1.0      2.9      2.9      2.9 txnRbk
          |    70.0s        0            0.0            1.1      0.0      0.0      0.0      0.0 opOk
          |    70.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    70.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    70.0s        0            0.0            0.9      0.0      0.0      0.0      0.0 txnRbk
          | BEGIN
          |   ALTER TABLE public.table5 RENAME COLUMN "col5_0" TO "col5_436";
          |   CREATE SCHEMA schema701 AUTHORIZATION root;
          |   ALTER TABLE public.table5 RENAME COLUMN "col5_436" TO "col5_704";
          |   ALTER TABLE public.table5 DROP CONSTRAINT "primary";
          |   ALTER TABLE schema3221.table3222 ALTER COLUMN "col3222_3223" DROP STORED;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |    71.0s        0            2.0            1.1   2281.7  33286.0  33286.0  33286.0 opOk
          |    71.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnCmtErr
          |    71.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
          |    71.0s        0            1.0            0.9      0.3      0.3      0.3      0.3 txnRbk
          | BEGIN
          |   ALTER TABLE public.table5 DROP CONSTRAINT "primary";
          |   ALTER TABLE public.table5 RENAME TO public.table3225;
          | ***FAIL; Expected no errors, but got ERROR: requested table does not have a primary key (SQLSTATE 55C02)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2515122&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@53b08837cf5e76504e437bad80c97e05989d2c60:

          |     1.0s        0            2.8            3.0    436.2    570.4    570.4    570.4 txnCmtErr
          |     1.0s        0            1.9            2.0     54.5     65.0     65.0     65.0 txnOk
          |     1.0s        0           12.1           13.0      3.9     16.8     25.2     25.2 txnRbk
          | BEGIN
          |   SELECT 'validating all objects';
          |   ALTER TABLE schema102.table103 ALTER COLUMN "col103_104" DROP STORED;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   CREATE TABLE IF NOT EXISTS public.table2 (col2_0 REGPROCEDURE NULL, col2_1 REGNAMESPACE, col2_2 DATE NULL, col2_3 INTERVAL, col2_4 FLOAT4 NOT NULL, col2_5 OID, col2_6 BIT(26) NULL, INDEX (col2_2) STORING (col2_0, col2_5), UNIQUE (col2_3 ASC, col2_4 ASC, col2_2 ASC, col2_0 ASC, col2_5) STORING (col2_1, col2_6), UNIQUE (col2_6, col2_0 ASC, col2_4 ASC, col2_1 DESC, col2_3 ASC), UNIQUE (col2_6 ASC, col2_5, col2_4, col2_2, col2_0 ASC, col2_3 DESC, col2_1 ASC), UNIQUE (col2_1, col2_0 DESC) STORING (col2_4, col2_5, col2_6), UNIQUE (col2_4 DESC, col2_2 ASC, col2_5, col2_1 DESC, col2_6 DESC), INDEX (col2_2, col2_1 ASC, col2_6, col2_0 ASC, col2_3 DESC), INDEX (col2_6 DESC, col2_0 ASC, col2_1, col2_5 DESC, col2_4, col2_2, col2_3 DESC));
          |   ALTER TABLE public.table2 ALTER COLUMN "col2_6" SET NOT NULL;
          |   ALTER TABLE public.table2 RENAME COLUMN "col2_3" TO "col2_20";
          |   ALTER TABLE schema113.table114 ALTER COLUMN "col114_115" SET NOT NULL;
          | ROLLBACK; ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)
          | BEGIN
          |   ALTER TABLE public.table38 RENAME COLUMN "IrrelevantColumnName" TO "OtherIrrelevantName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table38" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table35 ADD COLUMN IrrelevantColumnName string;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table35" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT; 
          | BEGIN
          |   CREATE TABLE schema118.table119 (col119_0 BIT(12) NOT NULL, PRIMARY KEY (col119_0), INDEX (col119_0 DESC), UNIQUE (col119_0), INDEX (col119_0 DESC), INDEX (col119_0 ASC));
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema118.table119" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table123 RENAME COLUMN "IrrelevantColumnName" TO "OtherIrrelevantName";
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table123" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   DROP SCHEMA "schema125" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: unknown schema "schema125" (SQLSTATE 3F000)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq122;
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=a27ef149 key=/Table/SystemConfigSpan/Start pri=0.03207196 epo=0 ts=1607932281.387909867,1 min=1607932281.340872769,0 seq=8} lock=true stat=PENDING rts=1607932281.340872769,0 wto=false max=1607932281.840872769,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema117 AUTHORIZATION root;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq131;
          | Expected SQLSTATE(S) 3F000, but got no errors
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2516757&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/58067

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@3018f66208f68fb90c77e3fad01f395a8f10ca8b:

          |   401.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnOk
          |   401.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnRbk
          |   402.0s        0            0.0            0.8      0.0      0.0      0.0      0.0 opOk
          |   402.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnCmtErr
          |   402.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnOk
          |   402.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnRbk
          |   403.0s        0            0.0            0.8      0.0      0.0      0.0      0.0 opOk
          |   403.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnCmtErr
          |   403.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnOk
          |   403.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnRbk
          |   404.0s        0            0.0            0.8      0.0      0.0      0.0      0.0 opOk
          |   404.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnCmtErr
          |   404.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnOk
          |   404.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnRbk
          |   405.0s        0            0.0            0.8      0.0      0.0      0.0      0.0 opOk
          |   405.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnCmtErr
          |   405.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnOk
          |   405.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnRbk
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema26038 AUTHORIZATION root;
          |   CREATE SCHEMA IF NOT EXISTS schema26047 AUTHORIZATION root;
          |   CREATE SEQUENCE IF NOT EXISTS schema26053.seq26054;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema26053.seq26054" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS schema26035.seq26036;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema26035.seq26036" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table13586 ADD COLUMN col13586_26846 BOOL NOT NULL;
          | ROLLBACK; expected SQLSTATE(S) 23502, and got ERROR: null value in column "col13586_26846" violates not-null constraint (SQLSTATE 23502)
          | BEGIN
          |   ALTER TABLE public.table11136 RENAME CONSTRAINT "index11136_16886" TO "index11136_26850";
          | ROLLBACK; ERROR: constraint "index11136_16886" does not exist (SQLSTATE 42704)
          | BEGIN
          |   ALTER TABLE public.table11136 DROP COLUMN "col11136_6";
          | ROLLBACK; expected SQLSTATE(S) 42P10, and got ERROR: column "col11136_6" is referenced by the primary key (SQLSTATE 42P10)
          | BEGIN
          |   ALTER TABLE public.table11136 RENAME COLUMN "col11136_8" TO "col11136_26851";
          | ROLLBACK; expected SQLSTATE(S) 2BP01, and got ERROR: cannot rename column "col11136_8" because view "view21694" depends on it (SQLSTATE 2BP01)
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   406.0s        0            2.0            0.8     71.3    121.6    121.6    121.6 opOk
          |   406.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnCmtErr
          |   406.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnOk
          |   406.0s        0            6.0            0.7      9.4     17.8     17.8     17.8 txnRbk
          | BEGIN
          |   CREATE INDEX IF NOT EXISTS index13586_26847 ON public.table13586 (col11136_7 DESC, col11136_6, col11136_5, col11136_0 ASC, col13586_16864, col11136_11 DESC, col11136_1 ASC, col11136_8 DESC);
          |   ALTER TABLE public.table13586 RENAME COLUMN "col11136_0" TO "col13586_26868";
          |   ALTER TABLE public.table13586 RENAME COLUMN "col11136_0" TO "col13586_26886";
          | ***FAIL; Expected no errors, but got ERROR: column "col11136_0" does not exist (SQLSTATE 42703)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2519773&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@de91557c0634ab3797356d445ba39e37d45d8205:

          | I201216 08:08:18.834576 1 workload/cli/run.go:359  creating load generator...
          | I201216 08:08:18.896333 1 workload/cli/run.go:390  creating load generator... done (took 61.755933ms)
          | Error: ***UNEXPECTED ERROR: ERROR: unknown schema "schema31" (SQLSTATE 3F000)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ublic.table33" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE TYPE public.enum28 AS ENUM ('dassf');
          |   CREATE TYPE schema34.enum35 AS ENUM ('sdsd', 's', 'ffafsf', 'd');
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema34.enum35" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq36;
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=b06f1d4a key=/Table/SystemConfigSpan/Start pri=0.00508507 epo=0 ts=1608106099.234678569,1 min=1608106099.105810451,0 seq=8} lock=true stat=PENDING rts=1608106099.105810451,0 wto=false max=1608106099.605810451,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq14;
          |   CREATE SCHEMA IF NOT EXISTS schema21 AUTHORIZATION root;
          |   SELECT 'validating all objects';
          |   DROP SCHEMA "schema21" CASCADE;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TABLE public.table27 (col27_0 BOOL NOT NULL, col27_1 BOX2D NOT NULL, col27_2 INET NOT NULL, col27_3 FLOAT4 NOT NULL, col27_4 INT8[] NULL, col27_5 INT8 NOT NULL, col27_6 DATE NOT NULL, col27_7 REGPROC NOT NULL, col27_8 DECIMAL NOT NULL, col27_9 BOX2D NOT NULL, col27_10 DATE NOT NULL, col27_11 REGCLASS NOT NULL, col27_12 OID NOT NULL, col27_13 BYTES NOT NULL, PRIMARY KEY (col27_2, col27_0 ASC, col27_7 DESC, col27_9, col27_13 ASC, col27_12 DESC, col27_6 ASC, col27_11 DESC, col27_10 ASC, col27_3 ASC, col27_1 DESC, col27_8 DESC, col27_5), UNIQUE (col27_13, col27_3 ASC, col27_11, col27_2 ASC, col27_0 ASC), UNIQUE (col27_9 ASC, col27_5 ASC, col27_11 DESC, col27_1 DESC, col27_8, col27_0 DESC, col27_12, col27_7 DESC), INDEX (col27_7 DESC, col27_9 DESC, col27_8, col27_12 ASC, col27_3, col27_11, col27_5 DESC, col27_0 ASC), UNIQUE (col27_10 DESC, col27_12 ASC, col27_2 ASC, col27_1 DESC, col27_9 ASC, col27_8 DESC, col27_7 DESC, col27_0 ASC, col27_11, col27_13, col27_5, col27_6 ASC, col27_3), INDEX (col27_7 DESC, col27_3 ASC, col27_1 ASC, col27_2, col27_12 DESC, col27_11 DESC, col27_5 ASC, col27_0 DESC, col27_13 DESC, col27_9, col27_6 ASC), INDEX (col27_12 ASC, col27_3 DESC, col27_5, col27_6, col27_2), FAMILY (col27_2), FAMILY (col27_8), FAMILY (col27_12, col27_5, col27_6, col27_9), FAMILY (col27_0, col27_11), FAMILY (col27_4), FAMILY (col27_10, col27_13, col27_3, col27_7), FAMILY (col27_1));
          |   ALTER TABLE public.table27 ALTER COLUMN "col27_12" DROP STORED;
          | ROLLBACK; expected SQLSTATE(S) 42611, and got ERROR: column "col27_12" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   CREATE SCHEMA schema26 AUTHORIZATION root;
          |   ALTER TABLE public.table43 ALTER COLUMN IrrelevantColumnName SET NOT NULL;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.table43" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SCHEMA IF NOT EXISTS schema41 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=41482aae key=/Table/SystemConfigSpan/Start pri=0.05269917 epo=0 ts=1608106099.630709388,1 min=1608106099.031475086,0 seq=6} lock=true stat=PENDING rts=1608106099.031475086,0 wto=false max=1608106099.031475086,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA schema44 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=1afdebee key=/Table/SystemConfigSpan/Start pri=0.00876812 epo=0 ts=1608106099.630709388,1 min=1608106098.933940708,0 seq=6} lock=true stat=PENDING rts=1608106098.933940708,0 wto=false max=1608106098.933940708,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP SCHEMA "schema31" CASCADE;
          | ***FAIL; Expected no errors, but got ERROR: unknown schema "schema31" (SQLSTATE 3F000)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2522960&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57993

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@eda9189cecbbc279f1857f6e6b992bdfd363397e:

          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | expected SQLSTATE(S) 3F000, and got ERROR: cannot create "schema43.enum44" because the target database or schema does not exist (SQLSTATE 3F000)
          | BEGIN
          |   ALTER TABLE public.table6 ALTER COLUMN col6_47 SET DEFAULT "IrrelevantValue";
          | ROLLBACK; expected SQLSTATE(S) 42703, and got ERROR: column "col6_47" does not exist (SQLSTATE 42703)
          | BEGIN
          |   DROP INDEX public.table6@"index6_45" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 42704, and got ERROR: index "index6_45" does not exist (SQLSTATE 42704)
          | BEGIN
          |   ALTER TABLE public.table6 ALTER COLUMN "col6_1" DROP STORED;
          | ROLLBACK; expected SQLSTATE(S) 42611, and got ERROR: column "col6_1" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq49;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE VIEW public.view52 AS SELECT public.table6.col6_1, "IrrelevantColumnName" FROM public.table6, schema25.table51;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "schema25.table51" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table6 DROP COLUMN "col6_50";
          | ROLLBACK; expected SQLSTATE(S) 42703, and got ERROR: column "col6_50" does not exist (SQLSTATE 42703)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq40;
          |   ALTER TABLE public.table6 ALTER COLUMN "col6_1" DROP STORED;
          | ROLLBACK; expected SQLSTATE(S) 42611, and got ERROR: column "col6_1" is not a computed column (SQLSTATE 42611)
          | BEGIN
          |   DROP VIEW IF EXISTS public.view41 CASCADE;
          |   ALTER TABLE public.table6 DROP COLUMN "col6_54";
          | ROLLBACK; expected SQLSTATE(S) 42703, and got ERROR: column "col6_54" does not exist (SQLSTATE 42703)
          | BEGIN
          |   ALTER TABLE public.table6 RENAME COLUMN "col6_1" TO "col6_1";
          |   ALTER TABLE public.table6 ALTER COLUMN "col6_0" SET DATA TYPE REGPROC;
          | ROLLBACK; expected SQLSTATE(S) 0A000,42846, and got ERROR: cannot convert INTERVAL to REGPROC (SQLSTATE 42846)
          | BEGIN
          |   DROP TABLE IF EXISTS public.table6;
          |   CREATE TABLE IF NOT EXISTS public.table55 (col55_0 STRING NOT NULL, col55_1 TIME NULL, col55_2 TIMESTAMPTZ, col55_3 REGCLASS NOT NULL, col55_4 VARCHAR NULL, col55_5 TIME, col55_6 INT8[] NULL, col55_7 OID NULL, col55_8 TIMESTAMPTZ, col55_9 BOX2D NULL, col55_10 INT8 NOT NULL, col55_11 CHAR NOT NULL, col55_12 NAME NULL, col55_13 TIMESTAMPTZ NOT NULL, col55_14 STRING NULL, INDEX (col55_9 ASC, col55_14 DESC, col55_7 ASC, col55_11, col55_0 ASC, col55_2 DESC, col55_10, col55_4 ASC, col55_5 ASC, col55_1), UNIQUE (col55_0, col55_12 ASC, col55_8 ASC, col55_1 DESC, col55_2, col55_10 DESC, col55_3 DESC, col55_14 ASC, col55_9 DESC, col55_7, col55_13 DESC, col55_4 DESC), UNIQUE (col55_1 DESC, col55_5 ASC, col55_11, col55_14 ASC, col55_3, col55_12, col55_7 DESC, col55_2 DESC, col55_0 DESC, col55_8 ASC, col55_4, col55_13 DESC), UNIQUE (col55_3, col55_14 DESC, col55_7 ASC, col55_13, col55_2 DESC, col55_10 DESC) STORING (col55_0, col55_1, col55_4, col55_9, col55_11, col55_12), UNIQUE (col55_4, col55_1 ASC, col55_14 DESC, col55_12 DESC, col55_2, col55_0, col55_3 DESC, col55_9 DESC) STORING (col55_6, col55_13), FAMILY (col55_11, col55_6), FAMILY (col55_5), FAMILY (col55_2, col55_4, col55_8), FAMILY (col55_9), FAMILY (col55_7, col55_10), FAMILY (col55_14), FAMILY (col55_13, col55_12), FAMILY (col55_3, col55_1), FAMILY (col55_0));
          |   ALTER TABLE public.table55 ALTER COLUMN "col55_0" DROP NOT NULL;
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=0386d78c key=/Table/SystemConfigSpan/Start pri=0.02394393 epo=0 ts=1608191700.270351333,1 min=1608191699.926105736,0 seq=29} lock=true stat=PENDING rts=1608191699.926105736,0 wto=false max=1608191700.426105736,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER SEQUENCE schema25.seq57 RENAME TO public.seq58;
          | ROLLBACK; expected SQLSTATE(S) 42602,42P01, and got ERROR: relation "schema25.seq57" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER TABLE public.table6 ALTER COLUMN "col6_0" SET DATA TYPE INTERVAL;
          | Expected SQLSTATE(S) 0A000,42846, but got no errors
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2525484&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/58068 Upon inspecting the artifiacts, this is the error that occurs:

Error: could not generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@46919380225dba7122130c338744b561d7eb6c56:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_080343.103_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2528188-1608274662-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201218 08:03:43.857694 1 workload/cli/run.go:359  creating load generator...
          | I201218 08:03:43.932291 1 workload/cli/run.go:390  creating load generator... done (took 74.595194ms)
          | Error: could not generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | , col98_103 "char", col98_104 INT2 NULL, col98_105 INET NULL, col98_106 BIT(36), col98_107 INET NULL, col98_108 DATE, col98_109 "char" NOT NULL, col98_110 TIMETZ NOT NULL, col98_111 INET, PRIMARY KEY (col98_101 ASC, col98_109 ASC, col98_110 ASC), INDEX (col98_102 ASC, col98_107, col98_101 ASC, col98_109 ASC, col98_99, col98_100, col98_111 DESC, col98_105 DESC, col98_103, col98_106), UNIQUE (col98_100, col98_106 DESC, col98_105 ASC, col98_101 ASC, col98_104, col98_110, col98_99, col98_109 ASC, col98_108 DESC, col98_102, col98_107 ASC, col98_103) STORING (col98_111));
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=b3eedf99 key=/Table/SystemConfigSpan/Start pri=0.03566802 epo=0 ts=1608278624.383569428,1 min=1608278624.175421583,0 seq=7} lock=true stat=PENDING rts=1608278624.175421583,0 wto=false max=1608278624.675421583,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq94;
          |   SELECT 'validating all objects';
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=76d97259 key=/Table/SystemConfigSpan/Start pri=0.02890313 epo=0 ts=1608278624.383569428,1 min=1608278624.054751218,0 seq=8} lock=true stat=PENDING rts=1608278624.054751218,0 wto=false max=1608278624.554751218,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE TYPE public.enum112 AS ENUM ('aas', 'sfa', 'dasd', 'asa', 'd');
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=b75b0202 key=/Table/SystemConfigSpan/Start pri=0.00735042 epo=0 ts=1608278624.383569428,1 min=1608278624.085378339,0 seq=9} lock=true stat=PENDING rts=1608278624.085378339,0 wto=false max=1608278624.585378339,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE public.table61 AS SELECT public.table61.col61_62, public.table61.col61_65, public.table61.col61_69, public.table61.col61_64, public.table61.col61_68, public.table61.col61_70, public.table61.col61_67, public.table61.col61_66, public.table61.col61_63, public.table61.col61_66, public.table61.col61_69, public.table61.col61_71, public.table61.col61_68, public.table61.col61_64 FROM public.table61, public.table61, public.table61;
          | ROLLBACK; expected SQLSTATE(S) 42701,42712,42P07, and got ERROR: source name "table61" specified more than once (missing AS clause) (SQLSTATE 42712)
          | BEGIN
          |   DROP SEQUENCE IF EXISTS public.seq24;
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   SELECT 'validating all objects';
          |   DROP SEQUENCE IF EXISTS public.seq24;
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=40b2d783 key=/Table/SystemConfigSpan/Start pri=0.03067534 epo=0 ts=1608278624.459310774,1 min=1608278624.131570088,0 seq=11} lock=true stat=PENDING rts=1608278624.131570088,0 wto=false max=1608278624.631570088,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE schema28.enum87 AS ENUM ('s', 'as', 'dfa', 'fffds', 'dad', 'd');
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=8d1462b2 key=/Table/SystemConfigSpan/Start pri=0.02087508 epo=0 ts=1608278624.459310774,1 min=1608278624.417629634,0 seq=9} lock=true stat=PENDING rts=1608278624.417629634,0 wto=false max=1608278624.917629634,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table61 DROP CONSTRAINT "table61_col61_67_col61_63_col61_62_col61_64_col61_68_key";
          | ROLLBACK; expected SQLSTATE(S) 0A000, and got ERROR: unimplemented: cannot drop UNIQUE constraint "table61_col61_67_col61_63_col61_62_col61_64_col61_68_key" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2528188&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57487

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@a782ee8d93a23fc53eedada3c51893c19f7bb41e:

          | main.(*cluster).Run
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
          | main.runSchemaChangeRandomLoad
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:135
          | main.registerRandomLoadBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange_random_load.go:79
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (2) output in run_081127.826_n1_workload_run_schemachange
        Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2530481-1608361532-23-n3cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20 returned
          | stderr:
          | I201219 08:11:28.569498 1 workload/cli/run.go:359  creating load generator...
          | I201219 08:11:28.654325 1 workload/cli/run.go:390  creating load generator... done (took 84.82619ms)
          | Error: ***UNEXPECTED ERROR: ERROR: descriptor is being dropped (SQLSTATE XXUUU)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | ansactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=86fee25f key=/Table/SystemConfigSpan/Start pri=0.01151166 epo=0 ts=1608365489.714108512,1 min=1608365489.607999555,0 seq=8} lock=true stat=PENDING rts=1608365489.607999555,0 wto=false max=1608365490.107999555,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE SCHEMA IF NOT EXISTS schema199 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=4050413f key=/Table/SystemConfigSpan/Start pri=0.03499552 epo=0 ts=1608365490.061107940,1 min=1608365489.514084754,0 seq=6} lock=true stat=PENDING rts=1608365489.514084754,0 wto=false max=1608365489.514084754,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP SCHEMA "schema1" CASCADE;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=22c92cdf key=/Table/SystemConfigSpan/Start pri=0.00215308 epo=0 ts=1608365490.043686637,2 min=1608365489.375203840,0 seq=1} lock=true stat=PENDING rts=1608365489.375203840,0 wto=false max=1608365489.375203840,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT; 
          | BEGIN
          |   CREATE SCHEMA schema173 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=41bac246 key=/Table/SystemConfigSpan/Start pri=0.01169359 epo=0 ts=1608365490.061107940,1 min=1608365489.881455025,0 seq=6} lock=true stat=PENDING rts=1608365489.881455025,0 wto=false max=1608365489.881455025,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TYPE public.enum158 AS ENUM ('sss', 's');
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TABLE schema1.table204 (col204_205 INT2 NULL, col204_206 INET NOT NULL, col204_207 INT8 NOT NULL, col204_208 FLOAT4 NULL, col204_209 REGPROC NOT NULL, col204_210 OID NOT NULL, col204_211 OID NOT NULL, col204_212 NAME, col204_213 BIT(25) NULL, col204_214 DATE NULL, col204_215 REGTYPE NOT NULL, col204_216 TIMESTAMP NOT NULL, col204_217 DECIMAL NOT NULL, col204_218 REGPROCEDURE NOT NULL, PRIMARY KEY (col204_207 ASC), INDEX (col204_211, col204_218 DESC, col204_216 ASC, col204_214 DESC, col204_212 DESC, col204_213 ASC, col204_209 ASC, col204_208 DESC, col204_207 DESC, col204_205 ASC, col204_215, col204_206 ASC, col204_210), UNIQUE (col204_217 ASC, col204_210 DESC, col204_212 ASC, col204_216 ASC) STORING (col204_208, col204_211, col204_214, col204_215, col204_218), INDEX (col204_206, col204_207, col204_208 DESC, col204_212, col204_213) STORING (col204_205, col204_211, col204_215, col204_217, col204_218), INDEX (col204_209 ASC, col204_213 DESC, col204_208 DESC, col204_217 DESC, col204_215, col204_205, col204_207 DESC), UNIQUE (col204_206 DESC, col204_207 DESC, col204_208 ASC, col204_217 ASC, col204_205 ASC, col204_212 ASC, col204_214) STORING (col204_210, col204_211, col204_215, col204_218), INDEX (col204_205 DESC, col204_216 ASC, col204_217, col204_209 ASC, col204_214 DESC, col204_210 ASC, col204_213 ASC), UNIQUE (col204_217, col204_214 ASC, col204_218, col204_210 DESC, col204_213 DESC, col204_205 ASC, col204_216, col204_206 ASC, col204_215 ASC, col204_211 ASC, col204_209, col204_207), UNIQUE (col204_218 DESC, col204_208 ASC, col204_209 DESC, col204_212), UNIQUE (col204_209 ASC, col204_217 DESC, col204_218 ASC, col204_210 DESC, col204_212, col204_213 DESC), FAMILY (col204_208), FAMILY (col204_207), FAMILY (col204_209), FAMILY (col204_212), FAMILY (col204_218), FAMILY (col204_205), FAMILY (col204_210), FAMILY (col204_215), FAMILY (col204_211), FAMILY (col204_217, col204_213, col204_206), FAMILY (col204_216, col204_214));
          | ***FAIL; Expected no errors, but got ERROR: descriptor is being dropped (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2530481&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(jayshrivastava) tracked by https://github.com/cockroachdb/cockroach/issues/57487

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@75efa1fe6f2096adc9db474026a2b7235e53c388:

          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | TATE(S) 42P01, and got ERROR: relation "schema1.view100" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   ALTER VIEW public.view101 RENAME TO public.view102;
          | ROLLBACK; expected SQLSTATE(S) 42P01, and got ERROR: relation "public.view101" does not exist (SQLSTATE 42P01)
          | BEGIN
          |   CREATE SEQUENCE IF NOT EXISTS public.seq103;
          |   DROP SCHEMA "schema105" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: unknown schema "schema105" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE SCHEMA schema91 AUTHORIZATION root;
          |   SELECT 'validating all objects';
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=70698a34 key=/Table/SystemConfigSpan/Start pri=0.00560610 epo=0 ts=1608451869.362117691,1 min=1608451869.176819969,0 seq=13} lock=true stat=PENDING rts=1608451869.176819969,0 wto=false max=1608451869.676819969,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema97 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=670918f0 key=/Table/SystemConfigSpan/Start pri=0.00339788 epo=0 ts=1608451869.427772388,1 min=1608451869.036618411,0 seq=6} lock=true stat=PENDING rts=1608451869.036618411,0 wto=false max=1608451869.036618411,0 (SQLSTATE 40001)
          | BEGIN
          |   SELECT 'validating all objects';
          | COMMIT; 
          | BEGIN
          |   DROP SEQUENCE IF EXISTS schema1.seq95;
          |   DROP SCHEMA "schema1" CASCADE;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=f04b4696 key=/Table/SystemConfigSpan/Start pri=0.00382997 epo=0 ts=1608451869.384227375,2 min=1608451869.026637181,0 seq=1} lock=true stat=PENDING rts=1608451869.026637181,0 wto=false max=1608451869.026637181,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema104 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=081262b3 key=/Table/SystemConfigSpan/Start pri=0.02717571 epo=0 ts=1608451869.427772388,1 min=1608451869.064944229,0 seq=6} lock=true stat=PENDING rts=1608451869.064944229,0 wto=false max=1608451869.064944229,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE SCHEMA schema109 AUTHORIZATION root;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=eea9135c key=/Table/SystemConfigSpan/Start pri=0.00557643 epo=0 ts=1608451869.502855958,1 min=1608451868.790994739,0 seq=6} lock=true stat=PENDING rts=1608451868.790994739,0 wto=false max=1608451868.790994739,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE TABLE public.table48 (col48_49 BIT(41) NOT NULL, PRIMARY KEY (col48_49), UNIQUE (col48_49), INDEX (col48_49 ASC), INDEX (col48_49), INDEX (col48_49), UNIQUE (col48_49 ASC), INDEX (col48_49), UNIQUE (col48_49 DESC), INDEX (col48_49 DESC));
          |   DROP TABLE IF EXISTS public.table48;
          | COMMIT; 
          | BEGIN
          |   SELECT 'validating all objects';
          |   CREATE TABLE IF NOT EXISTS schema1.table110 (col110_111 BOX2D NOT NULL, PRIMARY KEY (col110_111 DESC), INDEX (col110_111));
          | COMMIT; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): "sql txn" meta={id=6ceef8db key=/Table/SystemConfigSpan/Start pri=0.00010082 epo=0 ts=1608451869.384227375,3 min=1608451868.944467626,0 seq=7} lock=true stat=PENDING rts=1608451868.944467626,0 wto=false max=1608451869.444467626,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP SCHEMA "public" CASCADE;
          | ROLLBACK; expected SQLSTATE(S) 3F000, and got ERROR: cannot drop schema "public" (SQLSTATE 3F000)
          | BEGIN
          |   CREATE TYPE schema1.enum113 AS ENUM ('fd', 'sfd');
          | ***FAIL; Expected no errors, but got ERROR: descriptor is being dropped (SQLSTATE XXUUU)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2531670&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@cd62a02833eeb02d14944668d09ee54806927a2b:

          |  "workerId": 0,
          |  "clientTimestamp": "08:13:45.933756",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table161 RENAME TO public.table162"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table161\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:13:45.829774",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE INDEX \"IrrelevantName\" ON public.table157 (\"IrrelevantColumn\" ASC)"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table157\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:13:46.053126",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects'",
          |   "ALTER TABLE public.table156 ALTER COLUMN IrrelevantColumnName SET NOT NULL"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table156\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:13:46.061056",
          |  "ops": [
          |   "BEGIN",
          |   "INSERT INTO schema31.table185 (IrrelevantColumnName) VALUES (\"IrrelevantValue\")"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: descriptor is being dropped (SQLSTATE XXUUU)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2538659&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@152c3b30ae1dcb7a7cefc6aa7fa86c2dd264bd2e:

          |   "CREATE TYPE schema222.enum268 AS ENUM ('dfdfss', 'ddaffa', 'a', 'dfsdfa')",
          |   "SELECT 'validating all objects'",
          |   "CREATE SEQUENCE IF NOT EXISTS schema222.seq336",
          |   "CREATE TYPE public.enum355 AS ENUM ('ffsafa')",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=0d782b10 key=/Table/SystemConfigSpan/Start pri=0.01885328 epo=0 ts=1608883617.072909427,1 min=1608883616.863940033,0 seq=26} lock=true stat=PENDING rts=1608883616.863940033,0 wto=false max=1608883617.363940033,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:06:56.904483",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TABLE IF NOT EXISTS schema222.table319 (col319_320 VARCHAR NULL, col319_321 TIMETZ, col319_322 FLOAT4 NOT NULL, col319_323 INET NOT NULL, col319_324 NAME NOT NULL, col319_325 TIMESTAMPTZ NOT NULL, col319_326 INT2 NOT NULL, col319_327 TIME NOT NULL, col319_328 \"char\" NULL, col319_329 DATE NOT NULL, col319_330 BYTES NOT NULL, col319_331 REGNAMESPACE NOT NULL, PRIMARY KEY (col319_330 DESC, col319_322 ASC, col319_323 DESC, col319_324 ASC, col319_326 ASC, col319_327), UNIQUE (col319_322 ASC, col319_320) STORING (col319_321, col319_325), INDEX (col319_321 DESC, col319_320 ASC, col319_329 ASC, col319_331 ASC), UNIQUE (col319_322 DESC, col319_330 DESC, col319_326, col319_329 ASC, col319_325 ASC, col319_321 ASC, col319_327 DESC, col319_324, col319_323 DESC, col319_320), INDEX (col319_324 ASC, col319_323, col319_326 ASC, col319_330 DESC, col319_322 ASC, col319_327, col319_329 DESC, col319_325 ASC, col319_320 ASC) STORING (col319_321, col319_328, col319_331), UNIQUE (col319_326), UNIQUE (col319_329 DESC, col319_320, col319_330, col319_327 ASC, col319_322 DESC, col319_321 DESC, col319_325 DESC, col319_324, col319_326, col319_328, col319_331 DESC, col319_323), INDEX (col319_329, col319_327 DESC, col319_330 DESC, col319_328 ASC, col319_331, col319_320 DESC, col319_323 DESC, col319_322 DESC, col319_324, col319_325 ASC, col319_326, col319_321 ASC), INDEX (col319_327, col319_326, col319_329), INDEX (col319_325 ASC, col319_329, col319_322, col319_326 DESC, col319_328) STORING (col319_320, col319_321))",
          |   "ALTER TABLE schema222.table319 DROP COLUMN \"col319_330\""
          |  ],
          |  "expectedExecErrors": "42P10",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col319_330\" is referenced by the primary key (SQLSTATE 42P10)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:06:56.951879",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TYPE schema365.enum366 AS ENUM ('f', 'sad', 'fd', 'ffdsad', 'fff', 'sfsssd')"
          |  ],
          |  "expectedExecErrors": "3F000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot create \"schema365.enum366\" because the target database or schema does not exist (SQLSTATE 3F000)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:06:57.299403",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA schema333 AUTHORIZATION root"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2543440&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@4e3a9ccdb9cac89ae32d6248bb9b119b5bc250f7:

          |   "DROP TABLE public.table432 CASCADE"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table432\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:12:48.408175",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects'",
          |   "DROP SEQUENCE IF EXISTS public.seq265",
          |   "CREATE TYPE schema154.enum433 AS ENUM ('ddasss', 'f')",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=b5dd5078 key=/Table/SystemConfigSpan/Start pri=0.00679563 epo=0 ts=1609056768.464104866,1 min=1609056768.408123015,0 seq=20} lock=true stat=PENDING rts=1609056768.408123015,0 wto=false max=1609056768.908123015,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:12:48.303382",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER SEQUENCE public.seq265 RENAME TO public.seq420",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=e4f87361 key=/Table/SystemConfigSpan/Start pri=0.04347954 epo=0 ts=1609056768.464104866,1 min=1609056768.303289207,0 seq=12} lock=true stat=PENDING rts=1609056768.303289207,0 wto=false max=1609056768.803289207,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:12:48.464173",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA schema418 AUTHORIZATION root",
          |   "SELECT 'validating all objects'"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2544831&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@08c89a597a06520c30faf01965f9c74fe9b9854f:

          | 49",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA IF NOT EXISTS schema67 AUTHORIZATION root",
          |   "DROP SCHEMA \"public\" CASCADE"
          |  ],
          |  "expectedExecErrors": "3F000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:05:50.242746",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TABLE IF NOT EXISTS schema67.table245 (col245_246 FLOAT8 NOT NULL, col245_247 REGTYPE NULL, col245_248 VARCHAR NULL, col245_249 BIT(48) NOT NULL, col245_250 STRING NOT NULL, col245_251 INTERVAL NOT NULL, col245_252 \"char\" NOT NULL, col245_253 REGCLASS NULL, col245_254 TIME NULL, col245_255 BYTES NULL, col245_256 DECIMAL NOT NULL, col245_257 BYTES, col245_258 INTERVAL NULL, col245_259 BIT(22) NOT NULL, col245_260 TIMESTAMP, col245_261 INT4 NOT NULL, col245_262 INT4 NULL, PRIMARY KEY (col245_259, col245_249, col245_261 ASC, col245_246), INDEX (col245_261 DESC, col245_247, col245_262 DESC, col245_253 ASC, col245_257 DESC, col245_255 ASC, col245_249 DESC, col245_250 DESC, col245_248, col245_252, col245_259 DESC, col245_246, col245_258, col245_254 ASC, col245_256 ASC, col245_251), INDEX (col245_261, col245_252 DESC, col245_256, col245_255, col245_258, col245_254 DESC, col245_257 ASC), UNIQUE (col245_261 ASC, col245_256, col245_246 DESC, col245_258 ASC, col245_254 DESC, col245_251 DESC, col245_257 DESC, col245_259 DESC, col245_260 ASC, col245_247 DESC, col245_249 ASC, col245_255, col245_250 DESC, col245_253 ASC, col245_252 DESC, col245_262 DESC, col245_248 ASC), UNIQUE (col245_259 ASC, col245_256 ASC, col245_246 ASC, col245_262, col245_254 DESC, col245_251 DESC, col245_255 DESC, col245_260, col245_250 DESC, col245_258 DESC, col245_261, col245_248 ASC, col245_249 DESC) STORING (col245_247), UNIQUE (col245_254 ASC, col245_257 DESC, col245_260, col245_262, col245_246 ASC, col245_261 DESC, col245_256, col245_249 ASC, col245_255) STORING (col245_247, col245_251, col245_252, col245_258), INDEX (col245_259 DESC, col245_258 DESC, col245_248 DESC, col245_251 ASC, col245_260 ASC, col245_261, col245_247) STORING (col245_250, col245_253, col245_255, col245_256, col245_257, col245_262), UNIQUE (col245_260 DESC, col245_256 DESC, col245_252 DESC, col245_258 DESC, col245_254 ASC, col245_247, col245_257 ASC, col245_246 ASC, col245_251 ASC, col245_262) STORING (col245_248, col245_250, col245_253, col245_255), INDEX (col245_253 ASC, col245_260, col245_248 ASC, col245_252 ASC, col245_259 DESC, col245_247 ASC, col245_254 DESC, col245_249, col245_246, col245_262 DESC, col245_256, col245_261, col245_255 DESC, col245_257 ASC, col245_250 DESC, col245_258, col245_251 DESC))",
          |   "CREATE SCHEMA schema265 AUTHORIZATION root"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): \"sql txn\" meta={id=dca49080 key=/Table/SystemConfigSpan/Start pri=0.00908049 epo=0 ts=1609142750.552532112,2 min=1609142750.242656693,0 seq=12} lock=true stat=ABORTED rts=1609142750.242656693,0 wto=false max=1609142750.242656693,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:05:50.360519",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER SEQUENCE schema67.seq304 RENAME TO schema67.seq305"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema67.seq304\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:05:50.592556",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA IF NOT EXISTS schema244 AUTHORIZATION root"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2546130&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@08c89a597a06520c30faf01965f9c74fe9b9854f:

          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:08:08.471838",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table138 ALTER COLUMN \"col138_167\" DROP DEFAULT"
          |  ],
          |  "expectedExecErrors": "42703",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col138_167\" does not exist (SQLSTATE 42703)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:08:08.483467",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER INDEX schema1.table145@\"index145_169\" RENAME TO \"index145_170\""
          |  ],
          |  "expectedExecErrors": "42704",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: index \"index145_169\" does not exist (SQLSTATE 42704)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:08:08.45373",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table126 RENAME TO public.table168"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=6a3acba2 key=/Table/SystemConfigSpan/Start pri=0.00010943 epo=0 ts=1609164488.845381055,1 min=1609164488.453665154,0 seq=6} lock=true stat=PENDING rts=1609164488.453665154,0 wto=false max=1609164488.453665154,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "14:08:08.725352",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA IF NOT EXISTS schema164 AUTHORIZATION root"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2547217&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@08c89a597a06520c30faf01965f9c74fe9b9854f:

          | {
          |  "workerId": 0,
          |  "clientTimestamp": "21:28:18.878883",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE TABLE schema120.table342 AS SELECT public.table1.col1_4, public.table1.col1_84, public.table1.col1_3, public.table1.col1_4, public.table1.col1_3, public.table1.col1_84, public.table1.col1_5 FROM public.table1, public.table1"
          |  ],
          |  "expectedExecErrors": "42701,42712",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: source name \"table1\" specified more than once (missing AS clause) (SQLSTATE 42712)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "21:28:18.889199",
          |  "ops": [
          |   "BEGIN",
          |   "DROP TABLE IF EXISTS public.table1 CASCADE"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: updating mutation job for dropped table: TransactionRetryWithProtoRefreshError: WriteTooOldError: write at timestamp 1609190898.889065319,0 too old; wrote at 1609190899.033941394,1: \"sql txn\" meta={id=40ec0d3f pri=0.00320240 epo=0 ts=1609190899.033941394,1 min=1609190898.889065319,0 seq=0} lock=true stat=PENDING rts=1609190898.889065319,0 wto=false max=1609190898.889065319,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "21:28:18.883176",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table1 ALTER COLUMN \"col1_84\" DROP DEFAULT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=6d62ffac key=/Table/SystemConfigSpan/Start pri=0.03878251 epo=0 ts=1609190899.201162822,1 min=1609190898.883076864,0 seq=6} lock=true stat=PENDING rts=1609190898.883076864,0 wto=false max=1609190898.883076864,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "21:28:18.901046",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table1 ALTER COLUMN \"col1_84\" SET NOT NULL"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "23514",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: constraint in the middle of being added (SQLSTATE 55000)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2547724&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@365c5504b75c9a9260365a628a5110c48312178b:

          |   "BEGIN",
          |   "ALTER SEQUENCE public.seq451 RENAME TO public.seq577",
          |   "ALTER TABLE public.table498 RENAME COLUMN \"col498_582\" TO \"col498_583\""
          |  ],
          |  "expectedExecErrors": "42703",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col498_582\" does not exist (SQLSTATE 42703)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:25.212412",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table498 DROP COLUMN \"col462_476\"",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=7bc64044 key=/Table/SystemConfigSpan/Start pri=0.00535646 epo=0 ts=1609228766.599776447,1 min=1609228765.212358364,0 seq=12} lock=true stat=PENDING rts=1609228765.212358364,0 wto=false max=1609228765.712358364,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:25.772093",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE INDEX index498_561 ON public.table498 (col462_474, col462_468 ASC, col462_471 DESC, col462_477 DESC, col462_475, col462_464 DESC, col462_466 DESC, col462_469, col462_465, col462_463 DESC, col462_476 DESC) STORING (col462_467, col462_470, col462_473, col462_479, col462_472, col462_478)",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=58b73fb4 key=/Table/SystemConfigSpan/Start pri=0.02427092 epo=0 ts=1609228766.696109920,1 min=1609228765.772048013,0 seq=11} lock=true stat=PENDING rts=1609228765.772048013,0 wto=false max=1609228766.272048013,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:25.856053",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table498 ADD COLUMN col498_557 INET",
          |   "INSERT INTO public.table498 (col462_463,col462_464,col462_465,col462_466,col462_467,col462_468,col462_469,col462_470,col462_471,col462_472,col462_473,col462_474,col462_475,col462_476,col462_477,col462_478,col462_479) VALUES ('1989-10-04 10:18:37.000952+00:00':::TIMESTAMPTZ,3998596948:::OID,NULL,665612105:::OID,2309018599:::OID,'-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ,'1975-12-15':::DATE,B'11010110110010001011100',e'X%\\\\.]bd':::STRING,0:::OID,(-128):::INT8,'2010-07-19 10:48:09.000766+00:00':::TIMESTAMPTZ,(-1):::INT8,8977284540559029844:::INT8,(-0.8359984230889594):::FLOAT8,(-0.6004924178123474):::FLOAT8,NULL),('2031-04-16 03:33:17.000632+00:00':::TIMESTAMPTZ,4235590612:::OID,NULL,3482020415:::OID,1197537398:::OID,'1970-02-05 01:30:45.000254+00:00':::TIMESTAMPTZ,'infinity':::DATE,B'11110001100101000000010',e'\\x00':::STRING,2702423792:::OID,(-2446522845392937103):::INT8,'2006-03-12 18:39:47.000809+00:00':::TIMESTAMPTZ,1380620500081808684:::INT8,(-1690329091054563930):::INT8,0.8649639015955946:::FLOAT8,1.401298464324817e-45:::FLOAT8,471728919:::INT8),('1998-05-06 14:46:36.000271+00:00':::TIMESTAMPTZ,4229393587:::OID,2420745322:::OID,1395648869:::OID,3639020959:::OID,'2030-11-25 03:25:05.000238+00:00':::TIMESTAMPTZ,'1980-10-04':::DATE,B'10101111111010010101100',e'y6kGf\\x07Cd':::STRING,2978103345:::OID,4175738562372083653:::INT8,'2019-12-20 11:30:53.000296+00:00':::TIMESTAMPTZ,(-128):::INT8,(-5923934630710996906):::INT8,0.2750996132218033:::FLOAT8,(-0.282414972782135):::FLOAT8,(-1755518902):::INT8)"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: failed to satisfy CHECK constraint (col462_479 IS NOT NULL) (SQLSTATE 23514)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2548186&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@6d49a323b52966becfe8a2c38a1a8ccdf8ee58a1:

          |   "SELECT 'validating all objects'",
          |   "ALTER TABLE schema282.table283 ALTER COLUMN IrrelevantColumnName SET DATA TYPE IrrelevantDataType"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema282.table283\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:02.084392",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects'",
          |   "CREATE SEQUENCE public.seq184"
          |  ],
          |  "expectedExecErrors": "42P07",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"seq184\" already exists (SQLSTATE 42P07)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:01.398427",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA IF NOT EXISTS schema158 AUTHORIZATION root",
          |   "SELECT 'validating all objects'",
          |   "CREATE SEQUENCE IF NOT EXISTS public.seq184",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "",
          |  "txStatus": "TxStatusCommitSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:02.101304",
          |  "ops": [
          |   "BEGIN",
          |   "INSERT INTO schema51.table285 (IrrelevantColumnName) VALUES (\"IrrelevantValue\")"
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: descriptor is being dropped (SQLSTATE XXUUU)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2550095&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@b93fd531b93cb010729cb73fe679cdff9388cf27:

          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=6a376ab3 key=/Table/59/1/620398892638109697/0 pri=0.03281441 epo=0 ts=1609401116.169108297,2 min=1609401115.888815354,0 seq=4} lock=true stat=PENDING rts=1609401115.888815354,0 wto=false max=1609401116.388815354,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:51:55.977317",
          |  "ops": [
          |   "BEGIN",
          |   "DROP TABLE IF EXISTS public.table32 RESTRICT",
          |   "ALTER TABLE public.table99 RENAME COLUMN \"IrrelevantColumnName\" TO \"OtherIrrelevantName\""
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table99\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:51:55.908641",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table32 RENAME COLUMN \"col32_34\" TO \"col32_91\"",
          |   "ALTER TABLE public.table32 ALTER COLUMN \"col32_51\" DROP DEFAULT",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=efcea190 key=/Table/SystemConfigSpan/Start pri=0.02182615 epo=0 ts=1609401116.169108297,1 min=1609401115.908587334,0 seq=19} lock=true stat=PENDING rts=1609401115.908587334,0 wto=false max=1609401116.408587334,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:51:55.9956",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects'",
          |   "ALTER TABLE public.table32 ALTER COLUMN \"col32_49\" DROP DEFAULT",
          |   "INSERT INTO public.table32 (col32_49,col32_34,col32_51) VALUES ((-1882):::INT8,4139866189:::OID,'1996-07-02 03:11:53.000317+00:00':::TIMESTAMPTZ),(11539:::INT8,841370213:::OID,'1971-09-23 12:12:24.000174+00:00':::TIMESTAMPTZ)",
          |   "ALTER TABLE public.table32 ALTER COLUMN \"col32_49\" DROP NOT NULL"
          |  ],
          |  "expectedExecErrors": "0A000",
          |  "expectedCommitErrors": "",
          |  "message": "***FAIL; Failed to receive an execution error when errors were expected",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2552340&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@d7bbe0060531063b9bee29f69bc4d23d41b84e3d:

          |   "DROP SCHEMA \"schema2\" CASCADE"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=a3643fd5 key=/Table/SystemConfigSpan/Start pri=0.00909055 epo=0 ts=1609487566.121417747,1 min=1609487565.705046223,0 seq=12} lock=true stat=PENDING rts=1609487565.705046223,0 wto=false max=1609487565.705046223,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:52:46.028239",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects'",
          |   "CREATE SCHEMA schema146 AUTHORIZATION root",
          |   "ALTER SEQUENCE public.seq112 RENAME TO public.seq162",
          |   "DROP SEQUENCE public.seq162",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schemachange 52 2} {seq112 81 0}], old versions still in use (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:52:46.121493",
          |  "ops": [
          |   "BEGIN",
          |   "DROP SCHEMA \"schema117\" CASCADE",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schemachange 52 2} {schema117 82 0}], old versions still in use (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:52:46.272617",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SCHEMA schema161 AUTHORIZATION root"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot rollback to savepoint after a transaction restart (SQLSTATE 40001)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2553797&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@6de6fca6a320c33ce0f9033b527138526d02127b:

          |  "ops": [
          |   "BEGIN",
          |   "DROP SCHEMA \"schema91\" CASCADE"
          |  ],
          |  "expectedExecErrors": "3F000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: unknown schema \"schema91\" (SQLSTATE 3F000)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:06:29.587795",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table93 RENAME COLUMN \"IrrelevantColumnName\" TO \"OtherIrrelevantName\""
          |  ],
          |  "expectedExecErrors": "42P01",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table93\" does not exist (SQLSTATE 42P01)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:06:29.357976",
          |  "ops": [
          |   "BEGIN",
          |   "CREATE SEQUENCE IF NOT EXISTS public.seq87",
          |   "ALTER TABLE public.table76 DROP CONSTRAINT \"table76_col76_77_key3\""
          |  ],
          |  "expectedExecErrors": "0A000",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: unimplemented: cannot drop UNIQUE constraint \"table76_col76_77_key3\" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "08:06:29.486993",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE public.table76 ADD COLUMN col76_79 REGPROCEDURE",
          |   "CREATE TYPE public.enum90 AS ENUM ('s', 'dfaada', 'fsafd', 'dfsa', 'assfs', 'ad')",
          |   "ALTER TABLE public.table76 RENAME TO public.table94"
          |  ],
          |  "expectedExecErrors": "2BP01",
          |  "expectedCommitErrors": "",
          |  "message": "***FAIL; Failed to receive an execution error when errors were expected",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2558315&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 3 years ago

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@cee475331ca3629b503cd2e7c7919b72c98a5ca5:

          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | fully got expected execution error: ERROR: column \"col120_122\" is not a computed column (SQLSTATE 42611)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:29.102491",
          |  "ops": [
          |   "BEGIN",
          |   "SELECT 'validating all objects'",
          |   "CREATE TABLE IF NOT EXISTS schema92.table203 (col203_204 NAME NOT NULL, col203_205 INET NOT NULL, col203_206 INTERVAL, col203_207 BIT(8), col203_208 INTERVAL NOT NULL, col203_209 BIT(36) NOT NULL, col203_210 TIMESTAMP NOT NULL, col203_211 NAME NOT NULL, col203_212 \"char\" NOT NULL, PRIMARY KEY (col203_205 ASC, col203_212, col203_204, col203_209 ASC, col203_210), UNIQUE (col203_206 ASC) STORING (col203_208), UNIQUE (col203_211 ASC, col203_207 DESC, col203_205) STORING (col203_208), INDEX (col203_210 DESC, col203_209 ASC, col203_207 ASC, col203_211 ASC, col203_208 DESC, col203_204 DESC), INDEX (col203_209, col203_206 DESC, col203_208, col203_204 ASC, col203_211 DESC, col203_212 DESC, col203_210, col203_207), UNIQUE (col203_210), INDEX (col203_208 ASC, col203_206 DESC, col203_207 ASC, col203_211), FAMILY (col203_210, col203_209), FAMILY (col203_212), FAMILY (col203_208), FAMILY (col203_205), FAMILY (col203_204, col203_206, col203_211), FAMILY (col203_207))",
          |   "ALTER TABLE schema92.table203 ADD CONSTRAINT table203_col203_210_unique UNIQUE (col203_210)",
          |   "COMMIT"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=9733edea key=/Table/SystemConfigSpan/Start pri=0.01971242 epo=0 ts=1609919969.226823095,1 min=1609919969.102416407,0 seq=18} lock=true stat=PENDING rts=1609919969.102416407,0 wto=false max=1609919969.602416407,0 (SQLSTATE 40001)",
          |  "txStatus": "TxStatusCommitFailure"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:28.997953",
          |  "ops": [
          |   "BEGIN",
          |   "INSERT INTO schema92.table120 (col120_121,col120_122,col120_123,col120_124,col120_125,col120_126,col120_127,col120_128) VALUES ('8':::STRING,3746366395:::OID,'1995-01-13 13:49:22.000578':::TIMESTAMP,'BOX(-0.47932045977756543 -1.2774556217035022,0.977002681064577 0.9406558188284304)':::BOX2D,1053858095:::OID,'\\x':::BYTES,'227.16.252.138/18':::INET,'BOX(-0.8195947183191074 -0.42920262049070673,0.5289934228508908 -0.292160066809891)':::BOX2D),('K':::STRING,3475297357:::OID,'1971-02-04 07:06:53.000489':::TIMESTAMP,'BOX(-0.21020999228947335 -0.67993238359354,0.14707168410478744 -0.5872397218685912)':::BOX2D,295876617:::OID,'\\xfe4908cc602a6d':::BYTES,'de1f:426a:b854:9696:e08b:77b1:21a7:22cd/91':::INET,'BOX(-0.48932634526945706 -0.9121483592212682,0.8716083211888099 -0.7001534626151269)':::BOX2D),(' ':::STRING,954674372:::OID,'1990-06-12 09:00:16.000037':::TIMESTAMP,'BOX(-0.38133139996998205 0.36451384724354163,-0.30003963874955863 0.7745487458657498)':::BOX2D,0:::OID,'\\xc1':::BYTES,'144.53.160.127/30':::INET,'BOX(-1.4395781341891367 0.7637335544978013,0.6722168417120721 0.9004798589308582)':::BOX2D)",
          |   "ALTER TABLE schema92.table120 ALTER COLUMN \"col120_124\" DROP NOT NULL"
          |  ],
          |  "expectedExecErrors": "0A000,42P16",
          |  "expectedCommitErrors": "",
          |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: unimplemented: the first schema change statement in a transaction must precede any writes (SQLSTATE 0A000)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
          | {
          |  "workerId": 0,
          |  "clientTimestamp": "07:59:28.791208",
          |  "ops": [
          |   "BEGIN",
          |   "ALTER TABLE schema92.table120 ADD CONSTRAINT table120_col120_124_unique UNIQUE (col120_124)",
          |   "ALTER TABLE schema92.table120 ADD CONSTRAINT table120_col120_124_table120_col120_124_fk FOREIGN KEY (col120_124) REFERENCES schema92.table120 (col120_124) ON DELETE CASCADE ON UPDATE CASCADE"
          |  ],
          |  "expectedExecErrors": "",
          |  "expectedCommitErrors": "",
          |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: there is no unique constraint matching given keys for referenced table table120 (SQLSTATE 23503)",
          |  "txStatus": "TxStatusRollbackSuccess"
          | }
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2561359&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)