Closed victorggsimida closed 5 years ago
And
how to find which table this region belongs to
opertor log in pd.log
2018/09/28 15:00:57.971 cluster.go:449: [info] [region 0xca8f30] operator timeout: makeUpOfflineReplica (kind:region,replica, region:88042(4304,260), createAt:2018-09-28 14:50:17.276408872 +0800 CST m=+4420956.914345118, currentStep:0, steps:[add peer 48665230 on store 48166008 remove peer on store 45276315]) timeout
Could you please tell us which versions of PD and TiKV do you use? @victorggsimida
@rleungx pd
Release Version: v2.0.5
Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
Git Branch: release-2.0
UTC Build Time: 2018-07-06 10:27:51
tikv
TiKV
Release Version: 2.1.0-beta
Git Commit Hash: 96022a982b6b34d9cd8b690cf6d4b0b85ffae247
Git Commit Branch: master
UTC Build Time: 2018-08-06 11:53:06
Rust Version: rustc 1.29.0-nightly (4f3c7a472 2018-07-17)
Do you only have two replicas and one of them is down? If so, it seems that when we have two replicas and one of them is down, it cannot elect a new leader to make up the replica. But PD will still think there are two replicas so that it will continue to do it and it will fail definitely.
how to find which table this region belongs to
You can use tidb-ctl to get it.
how to resolve this?
You can remove the down peer by using tikv-ctl unsafe-recover
. Here is a documentation about how to use it.
@rleungx
tikv-ctl --path ( which store path shou i input )
45276315 this store is lost. path is not existed
@victorggsimida You need to login to store which still exists to use tikv-ctl unsafe-recover remove-fail-stores
with specifying some parameters:
--db
according to your situation (e.g. /path/to/tikv/db)--stores
the store which is lost--regions
the broken region@rleungx
./tidb-ctl -H*.*.*.* -P10083 region -i88042
{
"region_id": 88042,
"start_key": "dIAAAAAAAALRX3KAAAAAAFo3rw==",
"end_key": "dIAAAAAAAALRX3KAAAAAAF/IYQ==",
"frames": null
}
is there something wrong with my test?
./tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores --stores 45276315 --regions 88042 error: Found argument '--stores' which wasn't expected, or isn't valid in this context
$ tikv-ctl --db /path/to/tikv/db unsafe-recover remove-fail-stores --stores 3 --regions 1001,1002
USAGE:
tikv-ctl unsafe-recover remove-fail-stores
another command
./tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores 45276315 88042
thread 'main' panicked at 'called Result::unwrap()
on an Err
value: "IO error: While lock file: /ssd2/tidb/tikv/deploy2/data/db/LOCK: Resource temporarily unavailable"', libcore/result.rs:945:5
note: Run with RUST_BACKTRACE=1
for a backtrace.
i download the latest version of tikv-ctl, and it has no stores option
./tikv-ctl --help TiKV Ctl PingCAP Distributed transactional key value database powered by Rust and Raft
USAGE: tikv-ctl [OPTIONS] [SUBCOMMAND]
FLAGS: -h, --help Prints help information -V, --version Prints version information
OPTIONS:
--ca-path
@victorggsimida Sorry for that. There are something wrong with this documentation. You can replace --stores
and --regions
with -s
and -r
respectively.
Have you ever dropped the table? It seems that the table which this region belongs to might be dropped.
@rleungx so i want to find the table which this region belongs to .
database name and table name
@ rleungx it still did not work. is there something wrong with my command? thank you
./tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores -s 45276315 -r 88042
error: Found argument '-s' which wasn't expected, or isn't valid in this context
USAGE:
tikv-ctl unsafe-recover remove-fail-stores <stores>...
For more information try --help
@victorggsimida Could you just try to use the latest tikv-ctl
? It seems that this one is outdated.
And also you can use curl http://{TiDBIP}:10080/schema?table_id={tableID}
to find the table name. The ID of this table is 721
.
If there is no result, you can try to use admin show ddl jobs;
to find the table name.
@rleungx
i try the latest one again.
md5sum tikv-ctl
c78a5189c3fdf3bfa515897fa541aeb6 tikv-ctl
./tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores -s 45276315 -r 88042
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "IO error: While lock file: /ssd2/tidb/tikv/deploy2/data/db/LOCK: Resource temporarily unavailable"', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
./tikv-ctl RUST_BACKTRACE=1 --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores -s 45276315 -r 88042
error: Found argument 'RUST_BACKTRACE=1' which wasn't expected, or isn't valid in this context
You should stop the TiKV which you run this command on. @victorggsimida
@rleungx
my wrong. i did not tell u that i have try with shutdown the tikv.
and it seems that data/db/LOCK will not release itself.
can i just delete LOCK file?
Here is the recovery process:
./tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores -s 45276315 -r 88042
according to your casecan i just delete LOCK file?
You can use lsof
to check whether this file is occupied by some process. If it is not occupied, you can delete it.
@rleungx sry to disturb u again
/tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores 45276315
removing stores [45276315] from configrations...
success
./tikv-ctl --db /ssd2/tidb/tikv/deploy2/data/db unsafe-recover remove-fail-stores -s 45276315
error: Found argument '-s' which wasn't expected, or isn't valid in this context
USAGE:
tikv-ctl unsafe-recover remove-fail-stores
2. after run the command belove :sucessful
and i still see region info include
禄 region 88042 { "id": 88042, "start_key": "t\x80\x00\x00\x00\x00\x00\x02\xff\xd1_r\x80\x00\x00\x00\x00\xffZ7\xaf\x00\x00\x00\x00\x00\xfa", "end_key": "t\x80\x00\x00\x00\x00\x00\x02\xff\xd1r\x80\x00\x00\x00\x00\xff\xc8a\x00\x00\x00\x00\x00\xfa", "epoch": { "conf_ver": 260, "version": 4304 }, "peers": [ { "id": 45412102, "store_id": 45276315 }, { "id": 46840252, "store_id": 38566112 } ], "leader": { "id": 46840252, "store_id": 38566112 }, "approximate_size": 25 }
@rleungx
thank u for your help. the region has already remove from the down store.
if i did not handle it manualy, it will occupy the schedul process with "makeUpOfflineReplica" and maybe it will affect the region balance and hot region balance.
thank u again.
Question
Before asking a question, make sure you have:
one of the region info : 禄 region 88042 { "id": 88042, "start_key": "t\x80\x00\x00\x00\x00\x00\x02\xff\xd1_r\x80\x00\x00\x00\x00\xffZ7\xaf\x00\x00\x00\x00\x00\xfa", "end_key": "t\x80\x00\x00\x00\x00\x00\x02\xff\xd1r\x80\x00\x00\x00\x00\xff\xc8a\x00\x00\x00\x00\x00\xfa", "epoch": { "conf_ver": 260, "version": 4304 }, "peers": [ { "id": 45412102, "store_id": 45276315 }, { "id": 46840252, "store_id": 38566112 } ], "leader": { "id": 46840252, "store_id": 38566112 }, "approximate_size": 25 }
this region has replication in 45276315 and 38566112. But 45276315 is down, and data is lost.
And 38566112 is fine.
operator show
"makeUpOfflineReplica (kind:region,replica, region:88042(4304,260), createAt:2018-09-28 14:38:03.552894614 +0800 CST m=+4420223.190830856, currentStep:0, steps:[add peer 48665174 on store 48166008 remove peer on store 45276315]) ",
but this makeup is not ok. And this happens all the time from Sep,08th.
And in 38566112 tikv .log
how to resolve this?
this two command has already run.
but it still the same