pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.28k stars 5.84k forks source link

TiDB "panic in the recoverable goroutine" after flashback: runtime error: invalid memory address or nil pointer dereference #55285

Closed fubinzh closed 3 months ago

fubinzh commented 3 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. 2 TiDB clusters, upstream 3 TiKV; Downstream: 4 TiKV, downstreasm TiKV cofiguration:
  2. Enable log backup for upstream cluster
  3. For upstream, run workload to prepare 350000 rows of data, rowwidth: ~100k
  4. Perform full backup for upstream
  5. Get current timestamp (TS1) of upstream, and wait log bakcup checkpoint advance to TS1
  6. For upstream, run workload for 1 hour
  7. Flash backup upstream to TS1
  8. Run several queries to verify flash backup correctness

2. What did you expect to see? (Required)

TiDB should not panic

3. What did you see instead (Required)

TiDB panic seen

{"level":"ERROR","container":"tidb","namespace":"brie-consistence-pitr-flashback-tps-7617650-1-320","pod":"upstream-tidb-0","log":"[misc.go:114] [\"panic in the recoverable goroutine\"] [label=domain] [funcInfo=updateStatsWorker] [r=\"runtime error: invalid memory address or nil pointer dereference\"] [stack=\"github.com/pingcap/tidb/pkg/util.Recover\\n\\t/workspace/source/tidb/pkg/util/misc.go:118\\nruntime.gopanic\\n\\t/usr/local/go/src/runtime/panic.go:914\\nruntime.panicmem\\n\\t/usr/local/go/src/runtime/panic.go:261\\nruntime.sigpanic\\n\\t/usr/local/go/src/runtime/signal_unix.go:861\\ngithub.com/pingcap/tidb/pkg/statistics/handle/util.(*DDLEvent).IsMemOrSysDB\\n\\t/workspace/source/tidb/pkg/statistics/handle/util/ddl_event.go:55\\ngithub.com/pingcap/tidb/pkg/statistics/handle/ddl.(*ddlHandlerImpl).HandleDDLEvent\\n\\t/workspace/source/tidb/pkg/statistics/handle/ddl/ddl.go:58\\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).updateStatsWorker\\n\\t/workspace/source/tidb/pkg/domain/domain.go:2432\\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).UpdateTableStatsLoop.func3\\n\\t/workspace/source/tidb/pkg/domain/domain.go:2236\\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupEnhancedWrapper).Run.func1\\n\\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:99\"]"}

image

4. What is your TiDB version? (Required)

["Welcome to TiDB."] ["Release Version"=v8.1.1] [Edition=Community] ["Git Commit Hash"=370bd5a5331b3c6fe51f7246f7e5e7c7efadaa58] ["Git Branch"=HEAD] ["UTC Build Time"="2024-08-07 02:23:48"]

fubinzh commented 3 months ago

/label affects-8.1

Defined2014 commented 3 months ago

dup with https://github.com/pingcap/tidb/issues/52898, close this issue.