tikv / importer

tikv-importer is a front-end to help ingesting large number of KV pairs into a TiKV cluster
Apache License 2.0
20 stars 14 forks source link

tikv-importer fails with no error message when using domain name as listening address #16

Open tennix opened 5 years ago

tennix commented 5 years ago

Bug Report

I was running tidb-lightning in Kubernetes with domain name as the listening address, but the tikv-importer failed with no logs. After some debugging, I found that tikv-importer can start using Pod IP as listening address.

To reproduce this, simply add tikv-importer binary to pingcap/alpine-glibc Docker image, and run the following commands in the new image. To help ease the reproduce, I've built an image uhub.ucloud.cn/pingcap/tidb-lightning:latest.

$ docker run -it --rm uhub.ucloud.cn/pingcap/tidb-lightning:latest
/ # cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2      318b01eaf0c1
/ # /tikv-importer -h
TiKV Importer 3.0.2
The TiKV Authors
The importer server for TiKV

USAGE:
    tikv-importer [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -A, --addr <IP:PORT>       Set the listening address
    -C, --config <FILE>        Set the configuration
        --import-dir <PATH>    Set the directory to store importing kv data
        --log-file <FILE>      Set the log file
        --log-level <LEVEL>    Set the log level [possible values: trace, debug, info, warn, error, off]
/ # /tikv-importer --addr=318b01eaf0c1:20170
/ # echo $?
1
/ # /tikv-importer --addr=172.17.0.2:20170
^C[2019/08/11 05:48:33.494 +00:00] [INFO] [signal_handler.rs:21] ["receive signal 2, stopping server..."]
/#

To run tidb-lightning in Kubernetes, tidb-lightning needs to support domain name which is required for stateful applications.

kennytm commented 5 years ago

This isn't a problem specific to Importer. TiKV has the same problem since SocketAddr::from_str won't (and shouldn't) do DNS resolution.

https://github.com/tikv/tikv/blob/f5215d631d3145d824c0d0940cd490ff3f8bbf80/src/server/server.rs#L104

The TiDB Operator works fine because the TiKV server is listening on 0.0.0.0. The DNS name is exposed through --advertise-addr.

/tikv-server --pd=c-pd:2379 --advertise-addr=pod_name-0.pod_name-peer.k8s_ns.svc:20160 --addr=0.0.0.0:20160 --data-dir=/var/lib/tikv --capacity=3667GB --config=/etc/tikv/tikv.toml

The --advertise-addr setting is meaningless for Importer since we did not register Importer on PD.

Lightning supports DNS names just fine.

bin/tidb-lightning --importer 318b01eaf0c1:20170 ...

The only issue here is "no error message" (TiKV will emit a FATAL log like below). I'm gonna transfer this issue to the Importer repo.

[2019/08/11 14:24:45.576 +08:00] [FATAL] [server.rs:263] ["failed to create server: AddrParseError(())"]