Closed oomichi closed 5 years ago
エラーメッセージ
<*errors.errorString | 0xc421252080>: {
s: "failed running \"sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' \": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)",
}
failed running "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' ": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)
エラーになったテストコード
179 // Timeout in seconds is available as the fifth column from
180 // /proc/net/nf_conntrack.
181 result, err := framework.IssueSSHCommandWithResult(
182 fmt.Sprintf(
183 "sudo cat /proc/net/nf_conntrack "+
184 "| grep 'CLOSE_WAIT.*dst=%v.*dport=%v' "+
185 "| tail -n 1"+
186 "| awk '{print $5}' ",
187 serverNodeInfo.nodeIp,
188 testDaemonTcpPort),
189 framework.TestContext.Provider,
190 clientNodeInfo.node)
191 framework.ExpectNoError(err)
SSHログイン先アドレスのとり方
3311 func IssueSSHCommandWithResult(cmd, provider string, node *v1.Node) (*SSHResult, error) {
3312 Logf("Getting external IP address for %s", node.Name)
3313 host := ""
3314 for _, a := range node.Status.Addresses {
3315 if a.Type == v1.NodeExternalIP {
3316 host = net.JoinHostPort(a.Address, sshPort)
3317 break
3318 }
3319 }
対象のログ。e2e テストランナーから master にパスワード無しで SSH ログインできる必要あり
STEP: Checking /proc/net/nf_conntrack for the timeout
Sep 15 02:04:40.526: INFO: Getting external IP address for k8s-master
Sep 15 02:04:40.526: INFO: SSH "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'" on k8s-master(192.168.1.108:22)
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: command: sudo cat /proc/net/nf_conntrack | grep 'dport=11302'
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stdout: ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stderr: ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: exit code: 0
Sep 15 02:04:40.527: INFO: Getting external IP address for k8s-master
Sep 15 02:04:40.527: INFO: SSH "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' " on k8s-master(192.168.1.108:22)
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: command: sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}'
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stdout: ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stderr: ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: exit code: 0
Sep 15 02:04:40.527: INFO: Unexpected error occurred: failed running "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' ": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)
ssh-login できるようにしたが、別の問題あり
STEP: Checking /proc/net/nf_conntrack for the timeout
Sep 17 23:29:38.946: INFO: Getting external IP address for k8s-master
Sep 17 23:29:38.946: INFO: SSH "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'" on k8s-master(192.168.1.108:22)
Sep 17 23:29:39.258: INFO: ssh ubuntu@192.168.1.108:22: command: sudo cat /proc/net/nf_conntrack | grep 'dport=11302'
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stdout: ""
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stderr: "cat: /proc/net/nf_conntrack: No such file or directory\n"
Sep 17 23:29:39.260: INFO: ssh ubuntu@192.168.1.108:22: exit code: 1
Sep 17 23:29:39.260: INFO: Unexpected error occurred: failed running "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'": <nil> (exit code 1)
[AfterEach] [sig-network] Network
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
...
Sep 17 23:29:39.363: INFO:
Logging pods the kubelet thinks is on node k8s-node01
Sep 17 23:29:39.375: INFO: kube-flannel-ds-tllws started at 2018-08-17 09:12:53 +0000 UTC (1+1 container statuses recorded)
Sep 17 23:29:39.375: INFO: Init container install-cni ready: true, restart count 1
Sep 17 23:29:39.375: INFO: Container kube-flannel ready: true, restart count 1
Sep 17 23:29:39.375: INFO: kube-proxy-hxp7z started at 2018-07-31 23:08:51 +0000 UTC (0+1 container statuses recorded)
Sep 17 23:29:39.375: INFO: Container kube-proxy ready: true, restart count 3
Sep 17 23:29:39.375: INFO: e2e-net-server started at 2018-09-17 23:29:27 +0000 UTC (0+1 container statuses recorded)
Sep 17 23:29:39.375: INFO: Container e2e-net-server ready: true, restart count 0
Sep 17 23:29:39.440: INFO:
Latency metrics for node k8s-node01
STEP: Dumping a list of prepulled images on each node...
Sep 17 23:29:39.446: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-network-mb5s9" for this suite.
Sep 17 23:29:45.481: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered
Sep 17 23:29:45.693: INFO: namespace: e2e-tests-network-mb5s9, resource: bindings, ignored listing per whitelist
Sep 17 23:29:45.725: INFO: namespace e2e-tests-network-mb5s9 deletion completed in 6.273917752s
~ Failure [18.318 seconds]
[sig-network] Network
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22
should set TCP CLOSE_WAIT timeout [It]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:50
Expected error:
<*errors.errorString | 0xc420b0e430>: {
s: "failed running \"sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\": <nil> (exit code 1)",
}
failed running "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'": <nil> (exit code 1)
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:178
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSep 17 23:29:45.727: INFO: Running AfterSuite actions on all node
Sep 17 23:29:45.727: INFO: Running AfterSuite actions on node 1
Summarizing 1 Failure:
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:178
Ran 1 of 999 Specs in 92.542 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 998 Skipped --- FAIL: TestE2E (92.57s)
問題のログを抽出
Sep 17 23:29:39.258: INFO: ssh ubuntu@192.168.1.108:22: command: sudo cat /proc/net/nf_conntrack | grep 'dport=11302'
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stdout: ""
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stderr: "cat: /proc/net/nf_conntrack: No such file or directory\n"
/proc/net/nf_conntrack は netfilter 機能を構成するファイルの一つ Netfilterは、いわゆるファイアウォールやルータとしての役割を果たし、iptables コマンドで操作する。 nf_conntrack がロードされていることがわかる。
# lsmod | grep nf_conntrack
nf_conntrack_netlink 40960 0
nfnetlink 16384 1 nf_conntrack_netlink
nf_conntrack_ipv4 20480 2
nf_defrag_ipv4 16384 1 nf_conntrack_ipv4
nf_conntrack 106496 6 nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
で、/proc/sys/net/netfilter/ 配下にそれらしいファイルが大量にある。
# ls /proc/sys/net/netfilter/
nf_conntrack_acct nf_conntrack_icmp_timeout nf_conntrack_tcp_timeout_established nf_conntrack_timestamp
nf_conntrack_buckets nf_conntrack_log_invalid nf_conntrack_tcp_timeout_fin_wait nf_conntrack_udp_timeout
nf_conntrack_checksum nf_conntrack_max nf_conntrack_tcp_timeout_last_ack nf_conntrack_udp_timeout_stream
nf_conntrack_count nf_conntrack_tcp_be_liberal nf_conntrack_tcp_timeout_max_retrans nf_log
nf_conntrack_events nf_conntrack_tcp_loose nf_conntrack_tcp_timeout_syn_recv nf_log_all_netns
nf_conntrack_expect_max nf_conntrack_tcp_max_retrans nf_conntrack_tcp_timeout_syn_sent
nf_conntrack_generic_timeout nf_conntrack_tcp_timeout_close nf_conntrack_tcp_timeout_time_wait
nf_conntrack_helper nf_conntrack_tcp_timeout_close_wait nf_conntrack_tcp_timeout_unacknowledged
しかし、問題の /proc/net/nf_conntrack は存在しない。
CentOS 7でどうなっているのか、試してみる。
$ wget https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
$ openstack image create --container-format bare --disk-format qcow2 \
--file CentOS-7-x86_64-GenericCloud.qcow2 CentOS-7-x86_64
$ openstack image list
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 60408625-0466-4a31-9246-31dcac191cc9 | CentOS-7-x86_64 | active |
| 73f70800-1d0c-4569-a3c5-29c70775c334 | Ubuntu-16.04-x86_64 | active |
+--------------------------------------+---------------------+--------+
$ nova boot --key-name mykey --flavor m1.medium --image 60408625-0466-4a31-9246-31dcac191cc9 centos7
$ ssh centos@192.168.1.112
$ uname -a
Linux centos7.novalocal 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ sudo ls /proc/net/nf_conntrack
ls: cannot access /proc/net/nf_conntrack: No such file or directory
$ ls /proc/sys/net/netfilter/
nf_log nf_log_all_netns
CentOS 7でも /proc/net/nf_conntrack は存在しない。
linux kernelのコード net/netfilter/nf_conntrack_standalone.c
38 #ifdef CONFIG_NF_CONNTRACK_PROCFS
...
392 static int nf_conntrack_standalone_init_proc(struct net *net)
393 {
394 struct proc_dir_entry *pde;
395
396 pde = proc_create("nf_conntrack", 0440, net->proc_net, &ct_file_ops);
397 if (!pde)
398 goto out_nf_conntrack;
399
400 pde = proc_create("nf_conntrack", S_IRUGO, net->proc_net_stat,
401 &ct_cpu_seq_fops);
402 if (!pde)
403 goto out_stat_nf_conntrack;
404 return 0;
405
406 out_stat_nf_conntrack:
407 remove_proc_entry("nf_conntrack", net->proc_net);
408 out_nf_conntrack:
409 return -ENOMEM;
410 }
で、kernel config では無効になっている。 /boot/config-4.4.0-134-generic
# CONFIG_NF_CONNTRACK_PROCFS is not set
Ubuntu だと絶対に通らないな・・
そもそも timeout 値を取りたかったみたいだけど、代わりに取れそうなファイルはあるか?
$ cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait
3600
がそれっぽいけど。 とりあえず、configを有効にしてkernel ビルドしてみてどんな値が取れるのか、試してみる。
$ sudo apt-get install libssl-dev bc
$ cp /boot/config-4.4.0-134-generic .config
$ vi .config
- # CONFIG_NF_CONNTRACK_PROCFS is not set
+ CONFIG_NF_CONNTRACK_PROCFS=y
$ make oldconfig
$ make
...
$ sudo make modules_install
$ sudo make install
$ sudo vi /etc/default/grub
- GRUB_DEFAULT=0
+ GRUB_DEFAULT="gnulinux-advanced-c4c7809b-0c20-440a-88ea-a7da158f24b8>gnulinux-4.4.0-advanced-c4c7809b-0c20-440a-88ea-a7da158f24b8"
$ sudo update-grub
$ sudo sync
$ sudo reboot
ソースから確認するか・・
net/netfilter/nf_conntrack_proto_tcp.c
71 static unsigned int tcp_timeouts[TCP_CONNTRACK_TIMEOUT_MAX] __read_mostly = {
72 [TCP_CONNTRACK_SYN_SENT] = 2 MINS,
73 [TCP_CONNTRACK_SYN_RECV] = 60 SECS,
74 [TCP_CONNTRACK_ESTABLISHED] = 5 DAYS,
75 [TCP_CONNTRACK_FIN_WAIT] = 2 MINS,
76 [TCP_CONNTRACK_CLOSE_WAIT] = 60 SECS,
...
1585 pn->ctl_table[0].data = &tn->timeouts[TCP_CONNTRACK_SYN_SENT];
1586 pn->ctl_table[1].data = &tn->timeouts[TCP_CONNTRACK_SYN_RECV];
1587 pn->ctl_table[2].data = &tn->timeouts[TCP_CONNTRACK_ESTABLISHED];
1588 pn->ctl_table[3].data = &tn->timeouts[TCP_CONNTRACK_FIN_WAIT];
1589 pn->ctl_table[4].data = &tn->timeouts[TCP_CONNTRACK_CLOSE_WAIT];
...
1405 static struct ctl_table tcp_sysctl_table[] = {
1406 {
1407 .procname = "nf_conntrack_tcp_timeout_syn_sent",
..
1411 },
1412 {
1413 .procname = "nf_conntrack_tcp_timeout_syn_recv",
..
1417 },
1418 {
1419 .procname = "nf_conntrack_tcp_timeout_established",
..
1423 },
1424 {
1425 .procname = "nf_conntrack_tcp_timeout_fin_wait",
..
1429 },
1430 {
1431 .procname = "nf_conntrack_tcp_timeout_close_wait",
1432 .maxlen = sizeof(unsigned int),
1433 .mode = 0644,
1434 .proc_handler = proc_dointvec_jiffies,
1435 },
これを見ると nf_conntrack_tcp_timeout_close_wait は 60 SECS、SECS は
#define SECS * HZ
と定義されており、
$ cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait
3600
であることから HZ が 60?
テストコード
183 "sudo cat /proc/net/nf_conntrack "+
184 "| grep 'CLOSE_WAIT.*dst=%v.*dport=%v' "+
185 "| tail -n 1"+
186 "| awk '{print $5}' ",
から /proc/net/nf_conntrack ファイルに CLOSE_WAIT が含まれることを期待している。 しかし、これは特定の通信(ノードとPod間)の CLOSE_WAIT をとろうとしている。 /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait では、それらしい値が取れない。
kernel buildが終わったら、どういう内容の proc ファイルができるか試してみる。 → 存在しない。別の方法で有効化にする?
$ uname -a
Linux build 4.4.0 #1 SMP Wed Sep 19 23:13:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ sudo grep CONFIG_NF_CONNTRACK_PROCFS /boot/config-4.4.0
CONFIG_NF_CONNTRACK_PROCFS=y
$ sudo find . -name nf_conntrack
$ sudo ls /proc/net/nf_conntrack
ls: cannot access '/proc/net/nf_conntrack': No such file or directory
更にカーネルモジュールのロードが必要だった。他の環境ではロード済みのモジュール
$ sudo modprobe nf_conntrack_ipv4
$ sudo cat /proc/net/nf_conntrack
ipv4 2 tcp 6 431999 ESTABLISHED src=192.168.1.112 dst=192.168.1.1 sport=22 dport=42010 src=192.168.1.1 dst=192.168.1.112 sport=42010 dport=22 [ASSURED] mark=0 zone=0 use=2
$
k8s-master, k8s-node01 にインストールしてテストが通るか試してみる。 dockerデーモンが立ち上がらず、その結果 kubelet が立ち上がらず、kubernetes が動かない。
Sep 21 22:09:48 localhost docker[1004]: time="2018-09-21T22:09:48.909768646Z" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not
supported"
Sep 21 22:09:48 localhost docker[1004]: time="2018-09-21T22:09:48.910103255Z" level=fatal msg="Error starting daemon: error initializing graphdriver: driver
not supported"
aufs ストレージドライバがない? 元の Ubuntu カーネル
$ lsmod | grep aufs
aufs 217088 43
ビルドしたカーネルでは aufs がロードできない。
自動スキップ PR がマージされたので close とする。
まとめ
~ Failure [15.835 seconds] [sig-network] Network /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22 should set TCP CLOSE_WAIT timeout [It] /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:50
Expected error: <errors.errorString | 0xc421252080>: { s: "failed running \"sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.dst=192.168.1.109.dport=11302' | tail -n 1| awk '{print $5}' \": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)", } failed running "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' ": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0) not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:191
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSep 15 01:16:26.843: INFO: Running AfterSuite actions on all node Sep 15 01:16:26.843: INFO: Running AfterSuite actions on node 1
Summarizing 1 Failure:
[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:191
Ran 1 of 999 Specs in 15.932 seconds FAIL! -- 0 Passed | 1 Failed | 0 Pending | 998 Skipped --- FAIL: TestE2E (15.96s) FAIL
Ginkgo ran 1 suite in 16.174939555s Test Suite Failed !!! Error in ./hack/ginkgo-e2e.sh:143 Error in ./hack/ginkgo-e2e.sh:143. '"${ginkgo}" "${ginkgo_args[@]:+${ginkgo_args[@]}}" "${e2e_test}" -- "${auth_config[@]:+${auth_config[@]}}" --ginkgo.flakeAttempts="${FLAKE_ATTEMPTS}" --host="${KUBE_MASTER_URL}" --provider="${KUBERNETES_PROVIDER}" --gce-project="${PROJECT:-}" --gce-zone="${ZONE:-}" --gce-region="${REGION:-}" --gce-multizone="${MULTIZONE:-false}" --gke-cluster="${CLUSTER_NAME:-}" --kube-master="${KUBE_MASTER:-}" --cluster-tag="${CLUSTER_ID:-}" --cloud-config-file="${CLOUD_CONFIG:-}" --repo-root="${KUBE_ROOT}" --node-instance-group="${NODE_INSTANCE_GROUP:-}" --prefix="${KUBE_GCE_INSTANCE_PREFIX:-e2e}" --network="${KUBE_GCE_NETWORK:-${KUBE_GKE_NETWORK:-e2e}}" --node-tag="${NODE_TAG:-}" --master-tag="${MASTER_TAG:-}" --cluster-monitoring-mode="${KUBE_ENABLE_CLUSTER_MONITORING:-standalone}" --prometheus-monitoring="${KUBE_ENABLE_PROMETHEUS_MONITORING:-false}" ${KUBE_CONTAINER_RUNTIME:+"--container-runtime=${KUBE_CONTAINER_RUNTIME}"} ${MASTER_OS_DISTRIBUTION:+"--master-os-distro=${MASTER_OS_DISTRIBUTION}"} ${NODE_OS_DISTRIBUTION:+"--node-os-distro=${NODE_OS_DISTRIBUTION}"} ${NUM_NODES:+"--num-nodes=${NUM_NODES}"} ${E2E_REPORT_DIR:+"--report-dir=${E2E_REPORT_DIR}"} ${E2E_REPORT_PREFIX:+"--report-prefix=${E2E_REPORT_PREFIX}"} "${@:-}"' exited with status 1 Call stack: 1: ./hack/ginkgo-e2e.sh:143 main(...) Exiting with status 1 2018/09/15 01:16:26 process.go:155: Step './hack/ginkgo-e2e.sh --ginkgo.focus=should\sset\sTCP\sCLOSE_WAIT\stimeout' finished in 16.21553464s 2018/09/15 01:16:26 main.go:307: Something went wrong: encountered 1 errors: [error during ./hack/ginkgo-e2e.sh --ginkgo.focus=should\sset\sTCP\sCLOSE_WAIT\stimeout: exit status 1] 2018/09/15 01:16:26 e2e.go:81: err: exit status 1 exit status 1