search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.06k stars 3.8k forks source link

Sentry: hlc.go:421: log.Fatal: wall time 1709553902972723299 is not allowed to be greater than upper bound of 1709553902970633980. (1) attached stack trace -- stack trace: | github.com/cockroachdb... #119859

Closed cockroach-sentry closed 7 months ago

cockroach-sentry commented 7 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5035239988/?referrer=webhooks_plugin

Panic Message:

hlc.go:421: log.Fatal: wall time 1709553902972723299 is not allowed to be greater than upper bound of 1709553902970633980.
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).enforceWallTimeWithinBoundLocked
  |     github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:421
  | github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).NowAsClockTimestamp
  |     github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:412
  | github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).Now
  |     github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/hlc.go:389
  | github.com/cockroachdb/cockroach/pkg/sql.(*ExecutorConfig).GenerateID
  |     github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:1490
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:239
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:169
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1020
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIteratorEx
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:728
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIterator
  |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:714
  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
  |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:195
  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).scanTenantsForRunnableServices
  |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:106
  | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start.func1
  |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:60
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) log.Fatal: wall time 1709553902972723299 is not allowed to be greater than upper bound of 1709553902970633980.
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: wall time 1709553902972723299 is not allowed to be greater than upper bound of 1709553902970633980.
hlc.go:421: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/server/server_controller_orchestration.go#L59-L61](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/server/server_controller_orchestration.go#L59-L61) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/server/server_controller_orchestration.go#L105-L107](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/server/server_controller_orchestration.go#L105-L107) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/server/server_controller_orchestration.go#L194-L196](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/server/server_controller_orchestration.go#L194-L196) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L713-L715](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L713-L715) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L727-L729](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L727-L729) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L1019-L1021](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L1019-L1021) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L168-L170](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L168-L170) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L238-L240](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/internal.go#L238-L240) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/exec_util.go#L1489-L1491](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/sql/exec_util.go#L1489-L1491) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/hlc/pkg/util/hlc/hlc.go#L388-L390](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/hlc/pkg/util/hlc/hlc.go#L388-L390) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/hlc/pkg/util/hlc/hlc.go#L411-L413](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/hlc/pkg/util/hlc/hlc.go#L411-L413) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/hlc/pkg/util/hlc/hlc.go#L420-L422](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/hlc/pkg/util/hlc/hlc.go#L420-L422) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/server/server_controller_orchestration.go in pkg/server.(*serverController).start.func1 at line 60
pkg/server/server_controller_orchestration.go in pkg/server.(*serverController).scanTenantsForRunnableServices at line 106
pkg/server/server_controller_orchestration.go in pkg/server.(*serverController).getExpectedRunningTenants at line 195
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).QueryIterator at line 714
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).QueryIteratorEx at line 728
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).execInternal at line 1020
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).runWithEx at line 169
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).initConnEx at line 239
pkg/sql/exec_util.go in pkg/sql.(*ExecutorConfig).GenerateID at line 1490
pkg/util/hlc/pkg/util/hlc/hlc.go in pkg/util/hlc.(*Clock).Now at line 389
pkg/util/hlc/pkg/util/hlc/hlc.go in pkg/util/hlc.(*Clock).NowAsClockTimestamp at line 412
pkg/util/hlc/pkg/util/hlc/hlc.go in pkg/util/hlc.(*Clock).enforceWallTimeWithinBoundLocked at line 421

Tags

Tag Value
Command server
Environment v23.1.11
Go Version go1.19.10
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.11
Cockroach SHA 62ad175cbc0e9041d993ea1cc911e07d279671f5
# of CPUs 10
# of Goroutines 569

Jira issue: CRDB-36356

yuzefovich commented 7 months ago

unactionable