roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.15k stars 3.81k forks source link

roachtest: scbench/randomload/nodes=3/ops=10000/conc=20 failed #56230

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 4 years ago

(jayshrivastava) resolved by https://github.com/cockroachdb/cockroach/pull/56379, but there still some different errors left to solve in the comments

(roachtest).scbench/randomload/nodes=3/ops=10000/conc=20 failed on master@e25ee7ad4bcdeda49f40eb52c24d2d1b1aa67a86:

          | I201103 08:13:21.524996 1 workload/cli/run.go:356  creating load generator...
          | I201103 08:13:21.597243 1 workload/cli/run.go:387  creating load generator... done (took 72.246217ms)
          | Error: ***UNEXPECTED ERROR: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=24fa1aae key=/Table/SystemConfigSpan/Start pri=0.05040495 epo=0 ts=1604391209.856504541,1 min=1604391209.226930126,0 seq=6} lock=true stat=PENDING rts=1604391209.226930126,0 wto=false max=1604391209.226930126,0 (SQLSTATE 40001)
          | Error: COMMAND_PROBLEM: exit status 1
          | (1) COMMAND_PROBLEM
          | Wraps: (2) Node 1. Command with error:
          |   | ```
          |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 10000 --concurrency 20
          |   | ```
          | Wraps: (3) exit status 1
          | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
          |
          | stdout:
          | <... some data truncated by circular buffer; go to artifacts for details ...>
          | opOk
          |     8.0s        0            3.0            6.5    771.8    872.4    872.4    872.4 txnCmtErr
          |     8.0s        0            1.0            3.1     16.3     16.3     16.3     16.3 txnOk
          |     8.0s        0           25.0           20.1      2.4    268.4    469.8    469.8 txnRbk
          | BEGIN
          |   ALTER TABLE public.table309 DROP COLUMN "col309_368";
          | ROLLBACK; ERROR: column "col309_368" does not exist (SQLSTATE 42703)
          | BEGIN
          |   ALTER TABLE public.table309 ALTER COLUMN "col303_1" SET DATA TYPE public.enum370;
          | ROLLBACK; ERROR: type "public.enum370" does not exist (SQLSTATE 42704)
          | BEGIN
          |   ALTER TABLE public.table309 DROP CONSTRAINT "table303_col303_15_col303_17_col303_8_col303_13_col303_18_col303_10_col303_14_key";
          | ROLLBACK; ERROR: unimplemented: cannot drop UNIQUE constraint "table303_col303_15_col303_17_col303_8_col303_13_col303_18_col303_10_col303_14_key" using ALTER TABLE DROP CONSTRAINT, use DROP INDEX CASCADE instead (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table309 ALTER COLUMN "col303_3" SET DATA TYPE varchar;
          | ROLLBACK; ERROR: ALTER COLUMN TYPE from float4 to varchar is only supported experimentally (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table309 ALTER COLUMN "col303_16" SET DATA TYPE "char";
          | ROLLBACK; ERROR: ALTER COLUMN TYPE from timestamp to "char" is only supported experimentally (SQLSTATE 0A000)
          | BEGIN
          |   ALTER TABLE public.table309 DROP COLUMN "col303_16";
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=3fa1992f key=/Table/SystemConfigSpan/Start pri=0.01312331 epo=0 ts=1604391209.856504541,1 min=1604391209.458920462,0 seq=7} lock=true stat=PENDING rts=1604391209.458920462,0 wto=false max=1604391209.458920462,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table309 RENAME TO public.table369;
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=9dc0ce92 key=/Table/SystemConfigSpan/Start pri=0.00709184 epo=0 ts=1604391209.856504541,1 min=1604391209.302654671,0 seq=6} lock=true stat=PENDING rts=1604391209.302654671,0 wto=false max=1604391209.302654671,0 (SQLSTATE 40001)
          | BEGIN
          |   DROP TABLE public.table309;
          | ROLLBACK; ERROR: restart transaction: failed to mark job 603982425192792065 as as successful: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=9be48adb key=/Table/SystemConfigSpan/Start pri=0.01119124 epo=0 ts=1604391209.863562534,1 min=1604391209.523850704,0 seq=5} lock=true stat=PENDING rts=1604391209.523850704,0 wto=false max=1604391209.523850704,0 (SQLSTATE 40001)
          | BEGIN
          |   CREATE INDEX IF NOT EXISTS index309_372 ON public.table309 (col303_9 DESC, col303_14 ASC, col303_17, col303_3 DESC, col303_2 DESC, col303_10, col303_12 ASC, col303_5, col303_13 ASC, col303_18, col303_6, col303_0 ASC, col303_7 DESC, col303_1 DESC, col303_11 DESC, col303_16, col303_15 DESC, col303_8 ASC);
          | ROLLBACK; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=26768ce1 key=/Table/SystemConfigSpan/Start pri=0.02456629 epo=0 ts=1604391209.856504541,1 min=1604391209.422275976,0 seq=6} lock=true stat=PENDING rts=1604391209.422275976,0 wto=false max=1604391209.422275976,0 (SQLSTATE 40001)
          | BEGIN
          |   ALTER TABLE public.table309 ADD COLUMN col309_371 TIME NOT NULL;
          | ***FAIL; Expected no errors, but got ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): "sql txn" meta={id=24fa1aae key=/Table/SystemConfigSpan/Start pri=0.05040495 epo=0 ts=1604391209.856504541,1 min=1604391209.226930126,0 seq=6} lock=true stat=PENDING rts=1604391209.226930126,0 wto=false max=1604391209.226930126,0 (SQLSTATE 40001)
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

Artifacts: [/scbench/randomload/nodes=3/ops=10000/conc=20](https://teamcity.cockroachdb.com/viewLog.html?buildId=2412951&tab=artifacts#/scbench/randomload/nodes=3/ops=10000/conc=20) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Ascbench%2Frandomload%2Fnodes%3D3%2Fops%3D10000%2Fconc%3D20.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

Jira issue: CRDB-2965