Open yinyuncan opened 1 year ago
手动杀死Prometheus和ng后,ng重启失败 (Description ng fails to restart after prometheus and ng are manually killed)
tidb版本:6.5 ng版本: Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832
错误日志位于:/tidb-deploy/prometheus-9090/log/tsdb.log
2023-03-07T06:36:53.570Z info /go/pkg/mod/github.com/!victoria!metrics/!victoria!metrics@v1.65.0/app/vmselect/promql/rollup_result_cache.go:106 loading rollupResult cache from "/tidb-data/prometheus-9090/tsdb/cache/rollupResult"... 2023-03-07T06:36:53.571Z info /go/pkg/mod/github.com/!victoria!metrics/!victoria!metrics@v1.65.0/app/vmselect/promql/rollup_result_cache.go:132 loaded rollupResult cache from "/tidb-data/prometheus-9090/tsdb/cache/rollupResult" in 0.001 seconds; entriesCount: 0, sizeBytes: 0 2023-03-07T06:36:53.571Z info /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/timeseries/vm.go:45 started VictoriaMetrics in 0.094 seconds 2023/03/07 14:36:53 Arena too small, toWrite:126 newTotal:21810362 limit:21810342 github.com/dgraph-io/badger/v3/y.AssertTruef /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/y/error.go:62 github.com/dgraph-io/badger/v3/skl.(*Arena).putVal /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/arena.go:83 github.com/dgraph-io/badger/v3/skl.newNode /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/skl.go:114 github.com/dgraph-io/badger/v3/skl.(*Skiplist).Put /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/skl/skl.go:306 github.com/dgraph-io/badger/v3.(*memTable).replayFunction.func1 /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:258 github.com/dgraph-io/badger/v3.(*logFile).iterate /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:521 github.com/dgraph-io/badger/v3.(*memTable).UpdateSkipList /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:219 github.com/dgraph-io/badger/v3.(*DB).openMemTable /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:146 github.com/dgraph-io/badger/v3.(*DB).openMemTables /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/memtable.go:88 github.com/dgraph-io/badger/v3.Open /go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.1/db.go:333 github.com/genjidb/genji/engine/badgerengine.NewEngine /go/pkg/mod/github.com/crazycs520/genji/engine/badgerengine@v0.12.1-0.20220328082424-727a2d089bde/engine.go:25 github.com/pingcap/ng-monitoring/database/document.Init /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/document/document.go:34 github.com/pingcap/ng-monitoring/database.Init /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/database/database.go:14 main.main /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/ng-monitoring/main.go:68 runtime.main /usr/local/go/src/runtime/proc.go:250 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1594
ng.log 无报错 最后的日志如下
[2023/03/07 14:41:44.272 +08:00] [INFO] [printer.go:25] ["Welcome to ng-monitoring."] ["Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832] ["Git Branch"=heads/refs/tags/v6.5.0] ["UTC Build Time"="2022-12-16 08:18:47"] [GoVersion=go1.19.3] [2023/03/07 14:41:44.273 +08:00] [INFO] [main.go:64] [config] [config="{\"address\":\"0.0.0.0:12020\",\"advertise_address\":\"10.0.0.80:12020\",\"pd\":{\"endpoints\":[\"10.0.0.80:2379\",\"10.0.0.76:2379\",\"10.0.0.74:2379\"]},\"log\":{\"path\":\"/tidb-deploy/prometheus-9090/log\",\"level\":\"INFO\"},\"storage\":{\"path\":\"/tidb-data/prometheus-9090\"},\"continuous_profiling\":{\"enable\":false,\"profile_seconds\":10,\"interval_seconds\":60,\"timeout_seconds\":120,\"data_retention_seconds\":259200},\"security\":{\"ca_path\":\"\",\"cert_path\":\"\",\"key_path\":\"\"}}"]
实在是不知道怎么解决,于是问了一下ChatGpt
根据ChatGpt分析的结果,需要调整一下Badger数据库的arena 内存,呃 也不知道是否正确,大佬帮帮忙
手动杀死Prometheus和ng后,ng重启失败 (Description ng fails to restart after prometheus and ng are manually killed)
tidb版本:6.5 ng版本: Git Commit Hash"=f1c05e221155c2c95d391957971defbcbbf56832
错误日志位于:/tidb-deploy/prometheus-9090/log/tsdb.log
ng.log 无报错 最后的日志如下