search

alibaba / kt-connect

A toolkit for Integrating with your kubernetes dev environment more efficiently
https://alibaba.github.io/kt-connect/#/
GNU General Public License v3.0
1.52k stars 230 forks source link

0.3.3 使用 ktctl --debug connect 连接时报错（Failed to start: connect to port-forward failed） #302

Closed pbh3644 closed 2 years ago

pbh3644 commented 2 years ago

6:27PM INF KtConnect 0.3.3-beta1 start at 16604 (windows amd64) 6:27PM DBG Rectify pod kt-rectifier-mkyzi created 6:27PM INF Fetching cluster time ... 6:27PM DBG Execute command [date +%s] in kt-rectifier-mkyzi:standalone 6:27PM INF No time difference 6:27PM DBG Find 2 kt pods 6:27PM DBG Pod kt-connect-shadow-elkux does no have heart beat annotation 6:27PM DBG Pod kt-rectifier-mkyzi does no have heart beat annotation 6:27PM DBG Private Key generated 6:27PM DBG Public key generated 6:27PM INF Successful create config map kt-connect-shadow-cfkjd 6:27PM INF Deploying shadow pod kt-connect-shadow-cfkjd in namespace default 6:27PM INF Waiting for pod kt-connect-shadow-cfkjd ... 6:27PM INF Pod kt-connect-shadow-cfkjd is ready 6:27PM DBG Using port 47437 6:27PM DBG Request port forward pod:22 -> local:47437 via https://192.168.0.126:6443 Forwarding from 127.0.0.1:47437 -> 22 Forwarding from [::1]:47437 -> 22 Handling connection for 47437 6:27PM INF Port forward connection established Handling connection for 47437 2022/04/18 18:27:18 Using existing driver 0.14 2022/04/18 18:27:18 Creating adapter time="2022-04-18T18:27:19+08:00" level=info msg="[STACK] tun://KtConnectTunnel <-> socks5://127.0.0.1:2223" 6:27PM INF Tun device KtConnectTunnel is ready 2022/04/18 18:27:19 Removed orphaned adapter "KtConnectTunnel 1" 6:27PM DBG Cluster CIDR are: [10.233.0.0/16 192.168.0.126/32] 6:27PM DBG Using cluster ip 192.168.0.126 6:27PM INF Adding route to 10.233.0.0/16 6:27PM DBG Task name = netsh.exe, cmd.Args = [netsh interface ip set address KtConnectTunnel static 10.233.0.0 255.255.0.0] 6:27PM DBG Start netsh.exe at pid: 17364 6:27PM DBG Task name = netsh.exe, cmd.Args = [netsh interface ipv4 add route 10.233.0.0/16 KtConnectTunnel 10.233.0.0] 6:27PM DBG Start netsh.exe at pid: 15748 6:27PM INF Adding route to 192.168.0.126/32 6:27PM DBG Task name = netsh.exe, cmd.Args = [netsh interface ip add address KtConnectTunnel 192.168.0.126 255.255.255.255] 6:27PM DBG Start netsh.exe at pid: 5136 6:27PM DBG Task name = netsh.exe, cmd.Args = [netsh interface ipv4 add route 192.168.0.126/32 KtConnectTunnel 192.168.0.126] 6:27PM DBG Start netsh.exe at pid: 10528 6:27PM INF Route to tun device completed 6:27PM INF Setting up dns in local mode 6:27PM DBG Service found: kubernetes.default 10.233.0.1 6:27PM DBG Service found: tomcat.default 10.233.35.69 6:27PM DBG Dump hosts successful 6:27PM DBG Service tomcat added 6:27PM DBG Service kubernetes added 6:27PM DBG Using port 56017 6:27PM DBG Pod kt-rectifier-mkyzi deleted 6:27PM DBG Request port forward pod:53 -> local:56017 via https://192.168.0.126:6443 6:27PM DBG Pod kt-connect-shadow-elkux deleted 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 1 6:28PM ERR Port forward to 56017 -> 53 pod kt-connect-shadow-cfkjd interrupted error="error upgrading connection: error sending request: Post \"https://192.168.0.126:6443/api/v1/namespaces/default/pods/kt-connect-shadow-cfkjd/portforward\": dial tcp 192.168.0.126:6443: connectex: No connection could be made because the target machine actively refused it." 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 2 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 3 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 4 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 5 Handling connection for 47437 6:28PM DBG Heartbeat port forward 47437 ticked at 2022-04-18 18:28:15 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 6 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 7 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 8 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 9 6:28PM DBG Waiting for port forward (dial tcp :56017: connectex: No connection could be made because the target machine actively refused it.), retry: 10 Error: connect to port-forward failed Usage: ktctl connect [command options]

Flags: --mode string Connect mode 'tun2socks' or 'sshuttle' (default "tun2socks") --dnsMode string Specify how to resolve service domains, can be 'localDNS', 'podDNS', 'hosts' or 'hosts:', for multiple namespaces use ',' separation (default "localDNS") --shareShadow Use shared shadow pod --clusterDomain string The cluster domain provided to kubernetes api-server (default "cluster.local") --disablePodIp Disable access to pod IP address --skipCleanup Do not auto cleanup residual resources in cluster --includeIps string Specify extra IP ranges which should be route to cluster, e.g. '172.2.0.0/16', use ',' separated --excludeIps string Do not route specified IPs to cluster, e.g. '192.168.64.2' or '192.168.64.0/24', use ',' separated --disableTunDevice (tun2socks mode only) Create socks5 proxy without tun device --disableTunRoute (tun2socks mode only) Do not auto setup tun device route --proxyPort int (tun2socks mode only) Specify the local port which socks5 proxy should use (default 2223) --dnsCacheTtl int (local dns mode only) DNS cache refresh interval in seconds (default 60) -h, --help help for connect

Global Flags: -n, --namespace string Specify target namespace (otherwise follow kubeconfig current context) -c, --kubeconfig string Specify path of KubeConfig file -i, --image string Customize shadow image (default "registry.cn-hangzhou.aliyuncs.com/rdc-incubator/kt-connect-shadow:v0.3.3-beta1") --imagePullSecret string Custom image pull secret --serviceAccount string Specify ServiceAccount name for shadow pod (default "default") --nodeSelector string Specify location of shadow and route pod by node label, e.g. 'disk=ssd,region=hangzhou' -d, --debug Print debug log -l, --withLabel string Extra labels on shadow pod e.g. 'label1=val1,label2=val2' --withAnnotation string Extra annotation on shadow pod e.g. 'annotation1=val1,annotation2=val2' --portForwardTimeout int Seconds to wait before port-forward connection timeout (default 10) --podCreationTimeout int Seconds to wait before shadow or router pod creation timeout (default 60) --useShadowDeployment Deploy shadow container as deployment --useLocalTime Use local time for resource heartbeat timestamp -f, --forceUpdate Always update shadow image --context string Specify current context of kubeconfig --podQuota string Specify resource limit for shadow and router pod, e.g. '0.5c,512m'

6:28PM ERR Failed to start: connect to port-forward failed 6:28PM DBG Cleaning workspace 6:28PM INF Removed pid file C:\Users\ASUS/.ktctl/connect-16604.pid 6:28PM DBG Received event "C:\Users\ASUS\.ktctl\connect-16604.pid": REMOVE 6:28PM INF Pid file was removed 6:28PM INF Removed key file C:\Users\ASUS/.ktctl/pk/kt-connect-shadow-cfkjd_id_rsa 6:28PM DBG Dropping hosts records ... 6:28PM INF Drop hosts successful 6:28PM INF Cleaning configmap kt-connect-shadow-cfkjd 6:28PM INF Cleaning shadow pod kt-connect-shadow-cfkjd

linfan commented 2 years ago

问题收到，从日志来看，0.3.3-beta1的修复依然没有处理好API Server地址被加入到路由代理列表的问题，我们再继续排查一下原因。

可以先通过加 --excludeIps 参数绕过解决： sudo connect --excludeIps 192.168.0.126/32

linfan commented 2 years ago

已查明问题，请升级到 0.3.3-beta2 版本 https://github.com/alibaba/kt-connect/releases/tag/v0.3.3-beta2

linfan commented 2 years ago

0.3.3 正式版已发布

HuangDayu commented 2 years ago

0.3.3 正式版已发布

0.3.3 还是有这个问题，如果kt-connect-shadow-qstmj部署在default命名空间下，然后用serviceName请求了非default下的服务，则会出现这个问题。文档上已经更新到 0.3.4 了，然后并没有。 @linfan

little-hang commented 2 years ago

0.3.6版本：ktctl --debug --image=iregistry..com/zhiyun/kt-connect-shadow:stable --kubeconfig=//.kube/config connect 11:33AM INF Using cluster context kubernetes-admin@cluster.local (cluster.local) 11:33AM INF KtConnect 0.3.6 start at 23967 (linux amd64) 11:33AM INF Fetching cluster time ... 11:33AM INF Fetching cluster time ... 11:33AM INF Fetching cluster time ... 11:33AM INF Fetching cluster time ... 11:33AM INF Using tun2socks mode 11:33AM INF Successful create config map kt-connect-shadow-qtary 11:33AM INF Deploying shadow pod kt-connect-shadow-qtary in namespace default 11:33AM INF Waiting for pod kt-connect-shadow-qtary ... 11:33AM INF Pod kt-connect-shadow-qtary is ready 11:33AM INF Port forward local:39746 -> pod kt-connect-shadow-qtary:22 established 11:33AM INF Socks proxy established 11:33AM INF Tun device kt0 is ready 11:33AM INF Adding route to 10.233.0.0/16 11:33AM INF Adding route to 10.233.0.0/16 11:33AM WRN Failed to set route 10.233.0.0/16 to tun device 11:33AM INF Adding route to 10.232.21.0/24 11:33AM INF Adding route to 10.31.79.14/32 11:33AM WRN Some route rule is not setup properly 11:33AM WRN Skipped route to [10.31.79.14/32] 11:33AM INF Route to tun device completed 11:33AM INF Setting up dns in local mode 11:34AM ERR Exit: connect to port-forward failed 11:34AM INF Removed pid file /root/.kt/pid/connect-23967.pid 11:34AM INF Removed key file /root/.kt/key/kt-connect-shadow-qtary.key 11:34AM INF Pid file was removed 11:34AM INF Drop hosts successful 11:34AM INF Cleaning configmap kt-connect-shadow-qtary 11:34AM ERR Delete configmap kt-connect-shadow-qtary failed error="Delete \"https://10.232.21.**:6443/api/v1/namespaces/default/configmaps/kt-connect-shadow-qtary\": net/http: TLS handshake timeout" 11:34AM INF Cleaning shadow pod kt-connect-shadow-qtary 11:34AM ERR Failed to setup port forward local:5800 -> pod kt-connect-shadow-qtary:53 error="error upgrading connection: error sending request: Post \"https://10.232.21.**:6443/api/v1/namespaces/default/pods/kt-connect-shadow-qtary/portforward\": read tcp 10.31.77.15:36064->10.232.21.:6443: read: connection reset by peer" 11:34AM ERR Delete shadow pod kt-connect-shadow-qtary failed error="Delete \"https://10.232.21.:6443/api/v1/namespaces/default/pods/kt-connect-shadow-qtary\": read tcp 10.31.77.15:41352->10.232.21.**:6443: read: connection reset by peer"

@linfan

Andilay commented 2 years ago

v0.3.6 Ubuntu 20.04：sudo ktctl connect -d

3:35PM DBG Background task log to /tmp/kt-2153406166 3:35PM INF Using cluster context kubernetes-admin@kubernetes (kubernetes) 3:35PM INF KtConnect 0.3.6 start at 6394 (linux amd64) 3:35PM DBG Rectify pod kt-rectifier-tojbv created 3:35PM INF Fetching cluster time ... 3:35PM DBG Execute command [date +%s] in kt-rectifier-tojbv:standalone 3:35PM DBG Time difference is -53 3:35PM INF Using tun2socks mode 3:35PM DBG Find 1 kt pods 3:35PM DBG Private Key generated 3:35PM DBG Public key generated 3:35PM INF Successful create config map kt-connect-shadow-gxvcx 3:35PM INF Deploying shadow pod kt-connect-shadow-gxvcx in namespace default 3:35PM INF Waiting for pod kt-connect-shadow-gxvcx ... 3:35PM INF Pod kt-connect-shadow-gxvcx is ready 3:35PM DBG Using port 56036 3:35PM DBG Request port forward pod:22 -> local:56036 via https://192.168.12.109:6443 3:35PM INF Port forward local:56036 -> pod kt-connect-shadow-gxvcx:22 established 3:35PM INF Socks proxy established 3:35PM INF Tun device kt0 is ready 3:35PM DBG Service CIDR are: [10.105.0.0/16 10.109.0.0/16 10.107.0.0/16 10.103.0.0/16 10.108.0.0/16 10.104.0.0/16 10.101.0.0/16 10.96.0.0/16 10.100.0.0/16 10.98.0.0/16 10.97.0.0/16 10.111.0.0/16 10.106.0.0/16 10.110.0.0/16 10.99.0.0/16 10.102.202.25/32] 3:35PM DBG Pod CIDR are: [100.64.0.0/16 192.168.12.109/32] 3:35PM DBG Using cluster IP 192.168.12.109 3:35PM DBG Cluster CIDR are: [10.105.0.0/16 10.109.0.0/16 10.107.0.0/16 10.103.0.0/16 10.108.0.0/16 10.104.0.0/16 10.101.0.0/16 10.96.0.0/16 10.100.0.0/16 10.98.0.0/16 10.97.0.0/16 10.111.0.0/16 10.106.0.0/16 10.110.0.0/16 10.99.0.0/16 10.102.202.25/32 100.64.0.0/16 192.168.12.109/32] 3:35PM DBG Task /usr/sbin/ip with args [ip link set dev kt0 up] 3:35PM INF Adding route to 10.105.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.105.0.0/16 dev kt0] 3:35PM INF Adding route to 10.109.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.109.0.0/16 dev kt0] 3:35PM INF Adding route to 10.107.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.107.0.0/16 dev kt0] 3:35PM INF Adding route to 10.103.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.103.0.0/16 dev kt0] 3:35PM INF Adding route to 10.108.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.108.0.0/16 dev kt0] 3:35PM INF Adding route to 10.104.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.104.0.0/16 dev kt0] 3:35PM INF Adding route to 10.101.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.101.0.0/16 dev kt0] 3:35PM INF Adding route to 10.96.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.96.0.0/16 dev kt0] 3:35PM INF Adding route to 10.100.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.100.0.0/16 dev kt0] 3:35PM INF Adding route to 10.98.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.98.0.0/16 dev kt0] 3:35PM INF Adding route to 10.97.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.97.0.0/16 dev kt0] 3:35PM INF Adding route to 10.111.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.111.0.0/16 dev kt0] 3:35PM INF Adding route to 10.106.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.106.0.0/16 dev kt0] 3:35PM INF Adding route to 10.110.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.110.0.0/16 dev kt0] 3:35PM INF Adding route to 10.99.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.99.0.0/16 dev kt0] 3:35PM INF Adding route to 10.102.202.25/32 3:35PM DBG Task /usr/sbin/ip with args [ip route add 10.102.202.25/32 dev kt0] 3:35PM INF Adding route to 100.64.0.0/16 3:35PM DBG Task /usr/sbin/ip with args [ip route add 100.64.0.0/16 dev kt0] 3:35PM INF Adding route to 192.168.12.109/32 3:35PM DBG Task /usr/sbin/ip with args [ip route add 192.168.12.109/32 dev kt0] 3:35PM DBG Task /usr/sbin/ip with args [ip route show] 3:35PM WRN Skipped route to [10.102.202.25/32 192.168.12.109/32] 3:35PM INF Route to tun device completed 3:35PM INF Setting up dns in local mode 3:35PM DBG Dump hosts successful 3:35PM DBG Using port 57061 3:35PM DBG Request port forward pod:53 -> local:57061 via https://192.168.12.109:6443 3:35PM DBG Port forward reconnecting ... 3:35PM DBG Request port forward pod:22 -> local:56036 via https://192.168.12.109:6443 3:35PM ERR Exit: connect to port-forward failed 3:35PM DBG Cleaning workspace 3:35PM INF Removed pid file /root/.kt/pid/connect-6394.pid 3:35PM INF Removed key file /root/.kt/key/kt-connect-shadow-gxvcx.key 3:35PM DBG Dropping hosts records ... 3:35PM DBG Received event "/root/.kt/pid/connect-6394.pid": REMOVE 3:35PM INF Pid file was removed 3:35PM INF Drop hosts successful 3:35PM INF Cleaning configmap kt-connect-shadow-gxvcx 3:36PM ERR Delete configmap kt-connect-shadow-gxvcx failed error="Delete \"https://192.168.12.109:6443/api/v1/namespaces/default/configmaps/kt-connect-shadow-gxvcx\": net/http: TLS handshake timeout" 3:36PM INF Cleaning shadow pod kt-connect-shadow-gxvcx 3:36PM ERR Failed to setup port forward local:57061 -> pod kt-connect-shadow-gxvcx:53 error="error upgrading connection: error sending request: Post \"https://192.168.12.109:6443/api/v1/namespaces/default/pods/kt-connect-shadow-gxvcx/portforward\": read tcp 192.168.11.63:34238->192.168.12.109:6443: read: connection reset by peer" 3:36PM DBG Port forward local:56036 -> pod kt-connect-shadow-gxvcx:22 interrupted error="error upgrading connection: error sending request: Post \"https://192.168.12.109:6443/api/v1/namespaces/default/pods/kt-connect-shadow-gxvcx/portforward\": read tcp 192.168.11.63:34240->192.168.12.109:6443: read: connection reset by peer" 3:36PM ERR Delete shadow pod kt-connect-shadow-gxvcx failed error="Delete \"https://192.168.12.109:6443/api/v1/namespaces/default/pods/kt-connect-shadow-gxvcx\": read tcp 192.168.11.63:34248->192.168.12.109:6443: read: connection reset by peer"

@linfan