search

pingcap / tiup

A component manager for TiDB
https://tiup.io
Apache License 2.0
417 stars 310 forks source link

`tiup cluster replay` keeps using cached checkpoint after stopping service #2267

Closed pcqz closed 11 months ago

pcqz commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?

step1: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster deploy test2743 v5.3.0 test.yaml

test.yaml includes the following config, but the directory /data1 doesn't exist.

server_configs:
 tidb:
   oom-use-tmp-storage: false
   tmp-storage-path: '/data1'

step2: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster start test2743 --init /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster upgrade test2743 v6.5.3

The upgrade process failed when restarting tidb instance because of the error "mkdir /data1: permission denied" in the message log.

step3: Execute 'mkdir /data1 & chown tidb:tidb /data1', then replay the upgrade process: /home/tidb/.tiup/components/cluster/v1.12.3/tiup-cluster replay gcpHVb4kGTC

  1. What did you expect to see?

The upgrade process should be successful.

  1. What did you see instead?

The upgrade process failed with the error "failed to stop: tidb-node node_exporter-9677.service, please check the instance's log() for more detail.: timed out waiting for port 9677 to be stopped after 2m0s"

The tiup debug and message logs showed that the above node exporter service had stopped, but tiup was still using the checkpoint of upgrade to check service port.

message log:

Aug 28 18:11:47 systemd: Stopping node_exporter service...
Aug 28 18:11:47 systemd: Stopped node_exporter service.

tiup debug log:

2023-08-28T18:11:47.028+0800    INFO    Stopping component node_exporter
2023-08-28T18:11:47.028+0800    INFO            Stopping instance tidb-node
2023-08-28T18:11:47.222+0800    INFO    SSHCommand      {"host": "tidb-node", "port": "22", "cmd": "export LANG=C; PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin /usr/bin/sudo -H bash -c \"systemctl daemon-reload && systemctl stop node_exporter-9677.service\"", "stdout": "", "stderr": ""}
2023-08-28T18:11:47.222+0800    INFO    CheckPoint      {"host": "tidb-node", "port": 22, "user": "tidb", "sudo": true, "cmd": "systemctl daemon-reload && systemctl stop node_exporter-9677.service", "stdout": "", "stderr": "", "__hash__": "b8381cfd8180bdf5a3b7924d0249538ac15dee81", "__func__": "github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute", "hit": false}
2023-08-28T18:11:47.222+0800    INFO    CheckPoint      {"host": "10.2.8.3", "port": 22, "user": "tidb", "sudo": false, "cmd": "ss -ltn", "stdout": "State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              \nLISTEN     0      128          *:30183                    *:*                  \nLISTEN     0      64           *:9000                     *:*                  \nLISTEN     0      128          *:26185                    *:*                  \nLISTEN     0      128          *:8234                     *:*                  \nLISTEN     0      128          *:26186                    *:*                  \nLISTEN     0      128          *:26187                    *:*                  \nLISTEN     0      32768  127.0.0.1:12303                    *:*                  \nLISTEN     0      128          *:111                      *:*                  \nLISTEN     0      128          *:27185                    *:*                  \nLISTEN     0      128          *:27186                    *:*                  \nLISTEN     0      128          *:27187                    *:*                  \nLISTEN     0      128          *:20180                    *:*                  \nLISTEN     0      128          *:20181                    *:*                  \nLISTEN     0      32768  127.0.0.1:17365                    *:*                  \nLISTEN     0      128          *:22                       *:*                  \nLISTEN     0      128          *:20182                    *:*                  \nLISTEN     0      100    127.0.0.1:25                       *:*                  \nLISTEN     0      64           *:8123                     *:*                  \nLISTEN     0      32768  127.0.0.1:28859                    *:*                  \nLISTEN     0      32768  127.0.0.1:12830                    *:*                  \nLISTEN     0      32768  127.0.0.1:28510                    *:*                  \nLISTEN     0      32768  127.0.0.1:22719                    *:*                  \nLISTEN     0      32768  127.0.0.1:28544                    *:*                  \nLISTEN     0      32768  127.0.0.1:10273                    *:*                  \nLISTEN     0      32768  127.0.0.1:17315                    *:*                  \nLISTEN     0      128          *:20292                    *:*                  \nLISTEN     0      128          *:30180                    *:*                  \nLISTEN     0      128          *:26116                    *:*                  \nLISTEN     0      128          *:30181                    *:*                  \nLISTEN     0      32768  10.2.8.3:9094                     *:*                  \nLISTEN     0      1024   127.0.0.1:33062                    *:*                  \nLISTEN     0      32768       :::9671                    :::*                  \nLISTEN     0      32768       :::10088                   :::*                  \nLISTEN     0      32768       :::4713                    :::*                  \nLISTEN     0      80          :::3306                    :::*                  \nLISTEN     0      32768       :::20170                   :::*                  \nLISTEN     0      32768       :::20170                   :::*                  \nLISTEN     0      32768       :::20170                   :::*                  \nLISTEN     0      32768       :::20170                   :::*                  \nLISTEN     0      32768       :::20170                   :::*                  \nLISTEN     0      32768       :::4779                    :::*                  \nLISTEN     0      32768       :::2379                    :::*                  \nLISTEN     0      32768       :::22379                   :::*                  \nLISTEN     0      32768       :::39115                   :::*                  \nLISTEN     0      32768       :::4780                    :::*                  \nLISTEN     0      32768       :::9100                    :::*                  \nLISTEN     0      32768       :::2380                    :::*                  \nLISTEN     0      32768       :::5004                    :::*                  \nLISTEN     0      32768       :::8300                    :::*                  \nLISTEN     0      32768       :::22380                   :::*                  \nLISTEN     0      32768       :::9677                    :::*                  \nLISTEN     0      1024        :::3309                    :::*                  \nLISTEN     0      32768       :::10798                   :::*                  \nLISTEN     0      32768       :::19183                   :::*                  \nLISTEN     0      32768       :::1679                    :::*                  \nLISTEN     0      128         :::111                     :::*                  \nLISTEN     0      32768       :::20080                   :::*                  \nLISTEN     0      32768       :::21680                   :::*                  \nLISTEN     0      32768       :::30160                   :::*                  \nLISTEN     0      32768       :::30160                   :::*                  \nLISTEN     0      32768       :::30160                   :::*                  \nLISTEN     0      32768       :::30160                   :::*                  \nLISTEN     0      32768       :::30160                   :::*                  \nLISTEN     0      32768       :::4913                    :::*                  \nLISTEN     0      32768       :::30161                   :::*                  \nLISTEN     0      32768       :::30161                   :::*                  \nLISTEN     0      32768       :::30161                   :::*                  \nLISTEN     0      32768       :::30161                   :::*                  \nLISTEN     0      32768       :::30161                   :::*                  \nLISTEN     0      32768       :::19090                   :::*                  \nLISTEN     0      32768       :::4723                    :::*                  \nLISTEN     0      32768       :::9651                    :::*                  \nLISTEN     0      32768       :::30163                   :::*                  \nLISTEN     0      32768       :::30163                   :::*                  \nLISTEN     0      32768       :::30163                   :::*                  \nLISTEN     0      32768       :::30163                   :::*                  \nLISTEN     0      32768       :::30163                   :::*                  \nLISTEN     0      32768       :::32020                   :::*                  \nLISTEN     0      32768       :::26164                   :::*                  \nLISTEN     0      32768       :::26164                   :::*                  \nLISTEN     0      32768       :::26164                   :::*                  \nLISTEN     0      32768       :::26164                   :::*                  \nLISTEN     0      32768       :::26164                   :::*                  \nLISTEN     0      32768       :::26165                   :::*                  \nLISTEN     0      32768       :::26165                   :::*                  \nLISTEN     0      32768       :::26165                   :::*                  \nLISTEN     0      32768       :::26165                   :::*                  \nLISTEN     0      32768       :::26165                   :::*                  \nLISTEN     0      128         :::22                      :::*                  \nLISTEN     0      32768       :::26166                   :::*                  \nLISTEN     0      32768       :::26166                   :::*                  \nLISTEN     0      32768       :::26166                   :::*                  \nLISTEN     0      32768       :::26166                   :::*                  \nLISTEN     0      32768       :::26166                   :::*                  \nLISTEN     0      32768       :::12727                   :::*                  \nLISTEN     0      32768       :::2679                    :::*                  \nLISTEN     0      32768       :::3000                    :::*                  \nLISTEN     0      32768       :::2680                    :::*                  \nLISTEN     0      32768       :::9657                    :::*                  \nLISTEN     0      100        ::1:25                      :::*                  \nLISTEN     0      32768       :::26106                   :::*                  \nLISTEN     0      32768       :::26106                   :::*                  \nLISTEN     0      32768       :::26106                   :::*                  \nLISTEN     0      32768       :::26106                   :::*                  \nLISTEN     0      32768       :::26106                   :::*                  \nLISTEN     0      32768  ::ffff:10.2.8.3:3930                    :::*                  \nLISTEN     0      32768       :::2779                    :::*                  \nLISTEN     0      32768       :::9115                    :::*                  \nLISTEN     0      32768       :::4923                    :::*                  \nLISTEN     3      32768       :::2780                    :::*                  \nLISTEN     0      32768       :::39100                   :::*                  \nLISTEN     0      32768       :::27164                   :::*                  \nLISTEN     0      32768       :::27164                   :::*                  \nLISTEN     0      32768       :::27164                   :::*                  \nLISTEN     0      32768       :::27164                   :::*                  \nLISTEN     0      32768       :::27164                   :::*                  \nLISTEN     0      32768       :::27165                   :::*                  \nLISTEN     0      32768       :::27165                   :::*                  \nLISTEN     0      32768       :::27165                   :::*                  \nLISTEN     0      32768       :::27165                   :::*                  \nLISTEN     0      32768       :::27165                   :::*                  \nLISTEN     0      32768       :::27166                   :::*                  \nLISTEN     0      32768       :::27166                   :::*                  \nLISTEN     0      32768       :::27166                   :::*                  \nLISTEN     0      32768       :::27166                   :::*                  \nLISTEN     0      32768       :::27166                   :::*                  \nLISTEN     0      32768       :::17183                   :::*                  \nLISTEN     0      32768       :::9791                    :::*                  \nLISTEN     0      32768       :::17088                   :::*                  \nLISTEN     0      32768       :::10080                   :::*                  \nLISTEN     0      32768       :::4000                    :::*                  \nLISTEN     0      32768       :::24000                   :::*                  \nLISTEN     0      32768       :::9792                    :::*                  \nLISTEN     0      32768       :::20160                   :::*                  \nLISTEN     0      32768       :::20160                   :::*                  \nLISTEN     0      32768       :::20160                   :::*                  \nLISTEN     0      32768       :::20160                   :::*                  \nLISTEN     0      32768       :::20160                   :::*                  \nLISTEN     0      32768       :::10081                   :::*                  \nLISTEN     0      32768       :::4001                    :::*                  \nLISTEN     0      32768       :::20161                   :::*                  \nLISTEN     0      32768       :::20161                   :::*                  \nLISTEN     0      32768       :::20161                   :::*                  \nLISTEN     0      32768       :::20161                   :::*                  \nLISTEN     0      32768       :::20161                   :::*                  \nLISTEN     0      32768       :::17346                   :::*                  \nLISTEN     0      32768       :::17090                   :::*                  \nLISTEN     0      32768       :::9090                    :::*                  \nLISTEN     0      32768       :::13346                   :::*                  \nLISTEN     0      32768       :::20162                   :::*                  \nLISTEN     0      32768       :::20162                   :::*                  \nLISTEN     0      32768       :::20162                   :::*                  \nLISTEN     0      32768       :::20162                   :::*                  \nLISTEN     0      32768       :::20162                   :::*                  \nLISTEN     0      32768       :::3779                    :::*                  \nLISTEN     0      32768       :::3780                    :::*                  \nLISTEN     0      70          :::33060                   :::*                  \nLISTEN     0      32768       :::9093                    :::*                  \n", "stderr": "", "__hash__": "b8381cfd8180bdf5a3b7924d0249538ac15dee81", "__func__": "github.com/pingcap/tiup/pkg/cluster/executor.(*CheckPointExecutor).Execute", "hit": true}
  1. What version of TiUP are you using (tiup --version)? v1.12.3
seiya-annie commented 1 year ago

/found gs