search

pingcap / tiup

A component manager for TiDB
https://tiup.io
Apache License 2.0
418 stars 309 forks source link

Fail to run `tiup playground` #92

Closed Reminiscent closed 4 years ago

Reminiscent commented 4 years ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?

    Run tiup playground

  2. What did you expect to see? Set up a cluster

  3. What did you see instead? tidb.log

    [2020/04/10 16:56:04.980 +08:00] [INFO] [main.go:280] ["disable Prometheus push client"]
[2020/04/10 16:56:04.980 +08:00] [INFO] [store.go:68] ["new store"] [path=tikv://127.0.0.1:2379]
[2020/04/10 16:56:04.980 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[127.0.0.1:2379]"]
[2020/04/10 16:56:04.981 +08:00] [INFO] [systime_mon.go:25] ["start system time monitor"]
[2020/04/10 16:56:04.981 +08:00] [WARN] [base_client.go:170] ["[pd] failed to get cluster id"] [url=http://127.0.0.1:2379] [error="error:rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\" target:127.0.0.1:2379 status:TRANSIENT_FAILURE"]
[2020/04/10 16:56:05.984 +08:00] [WARN] [base_client.go:170] ["[pd] failed to get cluster id"] [url=http://127.0.0.1:2379] [error="error:rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\" target:127.0.0.1:2379 status:TRANSIENT_FAILURE"]
[2020/04/10 16:56:07.585 +08:00] [WARN] [base_client.go:170] ["[pd] failed to get cluster id"] [url=http://127.0.0.1:2379] [error="error:rpc error: code = Unknown desc = server not started target:127.0.0.1:2379 status:READY"]
[2020/04/10 16:56:08.598 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=]
[2020/04/10 16:56:08.599 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6814004129680144671]
[2020/04/10 16:56:08.602 +08:00] [INFO] [store.go:74] ["new store with retry success"]
[2020/04/10 16:56:29.089 +08:00] [WARN] [backoff.go:309] ["pdRPC backoffer.maxSleep 20000ms is exceeded, errors:\nregion not found for key \"mBootstra\\xffpKey\\x00\\x00\\x00\\x00\\xfb\\x00\\x00\\x00\\x00\\x00\\x00\\x00s\" at 2020-04-10T16:56:25.168466511+08:00\nregion not found for key \"mBootstra\\xffpKey\\x00\\x00\\x00\\x00\\xfb\\x00\\x00\\x00\\x00\\x00\\x00\\x00s\" at 2020-04-10T16:56:27.23407044+08:00\nregion not found for key \"mBootstra\\xffpKey\\x00\\x00\\x00\\x00\\xfb\\x00\\x00\\x00\\x00\\x00\\x00\\x00s\" at 2020-04-10T16:56:29.089404542+08:00"]
[2020/04/10 16:56:29.089 +08:00] [FATAL] [session.go:1850] ["check bootstrapped failed"] [error="[tikv:9001]PD server timeout"] [stack="github.com/pingcap/tidb/session.getStoreBootstrapVersion\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/session/session.go:1850\ngithub.com/pingcap/tidb/session.BootstrapSession\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/session/session.go:1650\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/tidb-server/main.go:234\nmain.main\n\t/home/jenkins/agent/workspace/tidb_v4.0.0-rc/go/src/github.com/pingcap/tidb/tidb-server/main.go:171\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
~

pd.log

[2020/04/10 16:56:05.055 +08:00] [INFO] [util.go:51] ["Welcome to Placement Driver (PD)"]
[2020/04/10 16:56:05.055 +08:00] [INFO] [util.go:52] [PD] [release-version=v4.0.0-rc]
[2020/04/10 16:56:05.055 +08:00] [INFO] [util.go:53] [PD] [git-hash=6f06805f3b0070107fcb4af68b2fc224dee0714d]
[2020/04/10 16:56:05.055 +08:00] [INFO] [util.go:54] [PD] [git-branch=heads/refs/tags/v4.0.0-rc]
[2020/04/10 16:56:05.055 +08:00] [INFO] [util.go:55] [PD] [utc-build-time="2020-04-08 07:49:10"]
[2020/04/10 16:56:05.055 +08:00] [INFO] [metricutil.go:81] ["disable Prometheus push client"]
[2020/04/10 16:56:05.055 +08:00] [INFO] [server.go:220] ["PD Config"] [config="{\"client-urls\":\"http://127.0.0.1:2379\",\"peer-urls\":\"http://127.0.0.1:2380\",\"advertise-client-urls\":\"http://127.0.0.1:2379\",\"advertise-peer-urls\":\"http://127.0.0.1:2380\",\"name\":\"pd-0\",\"data-dir\":\"/home/jackyan/.tiup/data/tt/pd-0/data\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"pd-0=http://127.0.0.1:2380\",\"initial-cluster-state\":\"new\",\"join\":\"\",\"lease\":3,\"log\":{\"level\":\"\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"/home/jackyan/.tiup/data/tt/pd-0/pd.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null},\"log-file\":\"\",\"log-level\":\"\",\"tso-save-interval\":\"3s\",\"metric\":{\"job\":\"pd-0\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":3,\"max-pending-peer-count\":16,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"false\",\"patrol-region-interval\":\"100ms\",\"max-store-down-time\":\"30m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-balance-rate\":15,\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"label\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"store-limit-mode\":\"manual\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"false\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-reset-ts-gap\":86400000000000,\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\"},\"cluster-version\":\"0.0.0\",\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"label-property\":null,\"WarningMsgs\":null,\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"@@@

tikv.log

[2020/04/10 16:56:05.161 +08:00] [INFO] [lib.rs:90] ["Welcome to TiKV"]
[2020/04/10 16:56:05.161 +08:00] [INFO] [lib.rs:92] []
[2020/04/10 16:56:05.161 +08:00] [INFO] [lib.rs:92] ["Release Version:   4.0.0-rc"]
[2020/04/10 16:56:05.161 +08:00] [INFO] [lib.rs:92] ["Git Commit Hash:   f45d0c963df3ee4b1011caf5eb146cacd1fbbad8"]
[2020/04/10 16:56:05.161 +08:00] [INFO] [lib.rs:92] ["Git Commit Branch: heads/refs/tags/v4.0.0-rc"]
[2020/04/10 16:56:05.164 +08:00] [INFO] [lib.rs:92] ["UTC Build Time:    2020-04-08 07:25:16"]
[2020/04/10 16:56:05.164 +08:00] [INFO] [lib.rs:92] ["Rust Version:      rustc 1.42.0-nightly (0de96d37f 2019-12-19)"]
[2020/04/10 16:56:05.164 +08:00] [INFO] [lib.rs:92] ["Enable Features:   jemalloc portable sse protobuf-codec"]
[2020/04/10 16:56:05.164 +08:00] [INFO] [lib.rs:92] ["Profile:           dist_release"]
[2020/04/10 16:56:05.164 +08:00] [WARN] [lib.rs:527] ["environment variable `TZ` is missing, using `/etc/localtime`"]
[2020/04/10 16:56:05.164 +08:00] [INFO] [lib.rs:539] ["environment variable is present"] [http_proxy=http://192.168.43.183:1080/]
[2020/04/10 16:56:05.164 +08:00] [INFO] [lib.rs:539] ["environment variable is present"] [https_proxy=http://192.168.43.183:1080/]
[2020/04/10 16:56:05.164 +08:00] [WARN] [server.rs:813] ["check: kernel"] [err="kernel parameters net.core.somaxconn got 128, expect 32768"]
[2020/04/10 16:56:05.164 +08:00] [WARN] [server.rs:813] ["check: kernel"] [err="kernel parameters net.ipv4.tcp_syncookies got 1, expect 0"]
[2020/04/10 16:56:05.164 +08:00] [WARN] [server.rs:813] ["check: kernel"] [err="kernel parameters vm.swappiness got 60, expect 0"]
[2020/04/10 16:56:05.165 +08:00] [INFO] [util.rs:427] ["connecting to PD endpoint"] [endpoints=http://127.0.0.1:2379]
[2020/04/10 16:56:05.165 +08:00] [INFO] [<unknown>] ["Connecting to server 127.0.0.1:2379 via HTTP proxy ipv4:192.168.43.183:1080"]
[2020/04/10 16:56:05.168 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7fe20e064150 for subchannel 0x7fe21293b1c0"]
[2020/04/10 16:56:05.512 +08:00] [INFO] [util.rs:387] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 14-UNAVAILABLE, details: Some(\"Socket closed\") }))"] [endpoints=http://127.0.0.1:2379]
[2020/04/10 16:56:05.512 +08:00] [WARN] [client.rs:58] ["validate PD endpoints failed"] [err="Other(\"[components/pd_client/src/util.rs:418]: PD cluster failed to respond\")"]
[2020/04/10 16:56:05.813 +08:00] [INFO] [util.rs:427] ["connecting to PD endpoint"] [endpoints=http://127.0.0.1:2379]
[2020/04/10 16:56:05.816 +08:00] [INFO] [<unknown>] ["Connecting to server 127.0.0.1:2379 via HTTP proxy ipv4:192.168.43.183:1080"]
[2020/04/10 16:56:05.823 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7fe210e89dd0 for subchannel 0x7fe21293b380"]
[2020/04/10 16:56:06.123 +08:00] [INFO] [util.rs:387] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { status: 14-UNAVAILABLE, details: Some(\"Socket closed\") }))"] [endpoints=http://127.0.0.1:2379]
[2020/04/10 16:56:06.424 +08:00] [INFO] [util.rs:427] ["connecting to PD endpoint"] [endpoints=http://127.0.0.1:2379]
jackyan@jackyan-virtual-machine:~$ tiup playground
Starting /home/jackyan/.tiup/components/playground/v0.0.6/playground  
Playground Bootstrapping...
Starting /home/jackyan/.tiup/components/pd/v4.0.0-rc/pd-server --name=pd-0 --data-dir=/home/jackyan/.tiup/data/RvkAAqA/pd-0/data --peer-urls=http://127.0.0.1:2380 --advertise-peer-urls=http://127.0.0.1:2380 --client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --log-file=/home/jackyan/.tiup/data/RvkAAqA/pd-0/pd.log --initial-cluster=pd-0=http://127.0.0.1:2380 
Starting /home/jackyan/.tiup/components/tikv/v4.0.0-rc/tikv-server --addr=127.0.0.1:20160 --status-addr=127.0.0.1:20180 --pd=http://127.0.0.1:2379 --config=/home/jackyan/.tiup/data/RvkAAqA/tikv-0/tikv.toml --data-dir=/home/jackyan/.tiup/data/RvkAAqA/tikv-0/data --log-file=/home/jackyan/.tiup/data/RvkAAqA/tikv-0/tikv.log 
Starting /home/jackyan/.tiup/components/tidb/v4.0.0-rc/tidb-server -P 4000 --store=tikv --host=127.0.0.1 --status=10080 --path=127.0.0.1:2379 --log-file=/home/jackyan/.tiup/data/RvkAAqA/tidb-0/tidb.log 
........................Error: start `/home/jackyan/.tiup/components/tidb/v4.0.0-rc/tidb-server` (wd:/home/jackyan/.tiup/data/RvkAAqA/tidb-0) failed: exit status 1
....................................To view the dashboard: http://127.0.0.1:2379/dashboard
  1. What version of TiUP are you using (tiup --version)? tiup version v0.0.2+Unknown(Unknown)
jackyan@jackyan-virtual-machine:~$ tiup list
Available components (Last Modified: 2020-02-27T15:20:35+08:00):
Name               Installed                     Platforms                 Description
----               ---------                     ---------                 -----------
tidb               YES(v4.0.0-beta.2,v4.0.0-rc)  darwin/amd64,linux/amd64  TiDB is an open source distributed HTAP database compatible with the MySQL protocol
tikv               YES(v4.0.0-beta.2,v4.0.0-rc)  darwin/amd64,linux/amd64  Distributed transactional key-value database, originally created to complement TiDB
pd                 YES(v4.0.0-beta.2,v4.0.0-rc)  darwin/amd64,linux/amd64  PD is the abbreviation for Placement Driver. It is used to manage and schedule the TiKV cluster
playground         YES(v0.0.5,v0.0.6)            darwin/amd64,linux/amd64  Bootstrap a local TiDB cluster
client             YES(v0.0.5,v0.0.6)            darwin/amd64,linux/amd64  A simple mysql client to connect TiDB
prometheus                                       darwin/amd64,linux/amd64  The Prometheus monitoring system and time series database.
tpc                                              darwin/amd64,linux/amd64  A toolbox to benchmark workloads in TPC
package                                          darwin/amd64,linux/amd64  A toolbox to package tiup component
tiops                                            darwin/amd64,linux/amd64  Bootstrap a remote TiDB cluster
grafana                                          linux/amd64,darwin/amd64  Grafana is the open source analytics & monitoring solution for every database
alertmanager                                     darwin/amd64,linux/amd64  Prometheus Alertmanager
blackbox_exporter                                darwin/amd64,linux/amd64  Blackbox prober exporter
node_exporter                                    darwin/amd64,linux/amd64  Blackbox prober exporter
pushgateway                                      darwin/amd64,linux/amd64  Blackbox prober exporter
tiflash                                          linux/amd64               
drainer                                          linux/amd64               The drainer componet of TiDB binlog service
pump                                             linux/amd64               The pump componet of TiDB binlog service
cluster                                          linux/amd64,darwin/amd64  Deploy a TiDB cluster for production
Reminiscent commented 4 years ago

PROBLEM SOLVED! The reason is that I set the HTTP proxy and it causes tidb and tikv failed to connect the pd. Close the ISSUE. Thanks to @lonng