oomichi / try-kubernetes

12 stars 5 forks source link

[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout #48

Closed oomichi closed 5 years ago

oomichi commented 5 years ago

まとめ

~ Failure [15.835 seconds] [sig-network] Network /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22 should set TCP CLOSE_WAIT timeout [It] /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:50

Expected error: <errors.errorString | 0xc421252080>: { s: "failed running \"sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.dst=192.168.1.109.dport=11302' | tail -n 1| awk '{print $5}' \": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)", } failed running "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' ": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0) not to have occurred

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:191

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSep 15 01:16:26.843: INFO: Running AfterSuite actions on all node Sep 15 01:16:26.843: INFO: Running AfterSuite actions on node 1

Summarizing 1 Failure:

[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:191

Ran 1 of 999 Specs in 15.932 seconds FAIL! -- 0 Passed | 1 Failed | 0 Pending | 998 Skipped --- FAIL: TestE2E (15.96s) FAIL

Ginkgo ran 1 suite in 16.174939555s Test Suite Failed !!! Error in ./hack/ginkgo-e2e.sh:143 Error in ./hack/ginkgo-e2e.sh:143. '"${ginkgo}" "${ginkgo_args[@]:+${ginkgo_args[@]}}" "${e2e_test}" -- "${auth_config[@]:+${auth_config[@]}}" --ginkgo.flakeAttempts="${FLAKE_ATTEMPTS}" --host="${KUBE_MASTER_URL}" --provider="${KUBERNETES_PROVIDER}" --gce-project="${PROJECT:-}" --gce-zone="${ZONE:-}" --gce-region="${REGION:-}" --gce-multizone="${MULTIZONE:-false}" --gke-cluster="${CLUSTER_NAME:-}" --kube-master="${KUBE_MASTER:-}" --cluster-tag="${CLUSTER_ID:-}" --cloud-config-file="${CLOUD_CONFIG:-}" --repo-root="${KUBE_ROOT}" --node-instance-group="${NODE_INSTANCE_GROUP:-}" --prefix="${KUBE_GCE_INSTANCE_PREFIX:-e2e}" --network="${KUBE_GCE_NETWORK:-${KUBE_GKE_NETWORK:-e2e}}" --node-tag="${NODE_TAG:-}" --master-tag="${MASTER_TAG:-}" --cluster-monitoring-mode="${KUBE_ENABLE_CLUSTER_MONITORING:-standalone}" --prometheus-monitoring="${KUBE_ENABLE_PROMETHEUS_MONITORING:-false}" ${KUBE_CONTAINER_RUNTIME:+"--container-runtime=${KUBE_CONTAINER_RUNTIME}"} ${MASTER_OS_DISTRIBUTION:+"--master-os-distro=${MASTER_OS_DISTRIBUTION}"} ${NODE_OS_DISTRIBUTION:+"--node-os-distro=${NODE_OS_DISTRIBUTION}"} ${NUM_NODES:+"--num-nodes=${NUM_NODES}"} ${E2E_REPORT_DIR:+"--report-dir=${E2E_REPORT_DIR}"} ${E2E_REPORT_PREFIX:+"--report-prefix=${E2E_REPORT_PREFIX}"} "${@:-}"' exited with status 1 Call stack: 1: ./hack/ginkgo-e2e.sh:143 main(...) Exiting with status 1 2018/09/15 01:16:26 process.go:155: Step './hack/ginkgo-e2e.sh --ginkgo.focus=should\sset\sTCP\sCLOSE_WAIT\stimeout' finished in 16.21553464s 2018/09/15 01:16:26 main.go:307: Something went wrong: encountered 1 errors: [error during ./hack/ginkgo-e2e.sh --ginkgo.focus=should\sset\sTCP\sCLOSE_WAIT\stimeout: exit status 1] 2018/09/15 01:16:26 e2e.go:81: err: exit status 1 exit status 1

oomichi commented 5 years ago

エラーメッセージ

      <*errors.errorString | 0xc421252080>: {
          s: "failed running \"sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' \": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)",
      }
      failed running "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' ": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)

エラーになったテストコード

179                 // Timeout in seconds is available as the fifth column from
180                 // /proc/net/nf_conntrack.
181                 result, err := framework.IssueSSHCommandWithResult(
182                         fmt.Sprintf(
183                                 "sudo cat /proc/net/nf_conntrack "+
184                                         "| grep 'CLOSE_WAIT.*dst=%v.*dport=%v' "+
185                                         "| tail -n 1"+
186                                         "| awk '{print $5}' ",
187                                 serverNodeInfo.nodeIp,
188                                 testDaemonTcpPort),
189                         framework.TestContext.Provider,
190                         clientNodeInfo.node)
191                 framework.ExpectNoError(err)

SSHログイン先アドレスのとり方

3311 func IssueSSHCommandWithResult(cmd, provider string, node *v1.Node) (*SSHResult, error) {
3312         Logf("Getting external IP address for %s", node.Name)
3313         host := ""
3314         for _, a := range node.Status.Addresses {
3315                 if a.Type == v1.NodeExternalIP {
3316                         host = net.JoinHostPort(a.Address, sshPort)
3317                         break
3318                 }
3319         }
oomichi commented 5 years ago

対象のログ。e2e テストランナーから master にパスワード無しで SSH ログインできる必要あり

STEP: Checking /proc/net/nf_conntrack for the timeout
Sep 15 02:04:40.526: INFO: Getting external IP address for k8s-master
Sep 15 02:04:40.526: INFO: SSH "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'" on k8s-master(192.168.1.108:22)
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: command:   sudo cat /proc/net/nf_conntrack | grep 'dport=11302'
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stdout:    ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stderr:    ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: exit code: 0
Sep 15 02:04:40.527: INFO: Getting external IP address for k8s-master
Sep 15 02:04:40.527: INFO: SSH "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' " on k8s-master(192.168.1.108:22)
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: command:   sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}'
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stdout:    ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: stderr:    ""
Sep 15 02:04:40.527: INFO: ssh @192.168.1.108:22: exit code: 0
Sep 15 02:04:40.527: INFO: Unexpected error occurred: failed running "sudo cat /proc/net/nf_conntrack | grep 'CLOSE_WAIT.*dst=192.168.1.109.*dport=11302' | tail -n 1| awk '{print $5}' ": error getting signer for provider skeleton: 'error reading SSH key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory'' (exit code 0)
oomichi commented 5 years ago

ssh-login できるようにしたが、別の問題あり

STEP: Checking /proc/net/nf_conntrack for the timeout
Sep 17 23:29:38.946: INFO: Getting external IP address for k8s-master
Sep 17 23:29:38.946: INFO: SSH "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'" on k8s-master(192.168.1.108:22)
Sep 17 23:29:39.258: INFO: ssh ubuntu@192.168.1.108:22: command:   sudo cat /proc/net/nf_conntrack | grep 'dport=11302'
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stdout:    ""
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stderr:    "cat: /proc/net/nf_conntrack: No such file or directory\n"
Sep 17 23:29:39.260: INFO: ssh ubuntu@192.168.1.108:22: exit code: 1
Sep 17 23:29:39.260: INFO: Unexpected error occurred: failed running "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'": <nil> (exit code 1)
[AfterEach] [sig-network] Network
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
...
Sep 17 23:29:39.363: INFO:
Logging pods the kubelet thinks is on node k8s-node01
Sep 17 23:29:39.375: INFO: kube-flannel-ds-tllws started at 2018-08-17 09:12:53 +0000 UTC (1+1 container statuses recorded)
Sep 17 23:29:39.375: INFO:      Init container install-cni ready: true, restart count 1
Sep 17 23:29:39.375: INFO:      Container kube-flannel ready: true, restart count 1
Sep 17 23:29:39.375: INFO: kube-proxy-hxp7z started at 2018-07-31 23:08:51 +0000 UTC (0+1 container statuses recorded)
Sep 17 23:29:39.375: INFO:      Container kube-proxy ready: true, restart count 3
Sep 17 23:29:39.375: INFO: e2e-net-server started at 2018-09-17 23:29:27 +0000 UTC (0+1 container statuses recorded)
Sep 17 23:29:39.375: INFO:      Container e2e-net-server ready: true, restart count 0
Sep 17 23:29:39.440: INFO:
Latency metrics for node k8s-node01
STEP: Dumping a list of prepulled images on each node...
Sep 17 23:29:39.446: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-network-mb5s9" for this suite.
Sep 17 23:29:45.481: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered
Sep 17 23:29:45.693: INFO: namespace: e2e-tests-network-mb5s9, resource: bindings, ignored listing per whitelist
Sep 17 23:29:45.725: INFO: namespace e2e-tests-network-mb5s9 deletion completed in 6.273917752s

~ Failure [18.318 seconds]
[sig-network] Network
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22
  should set TCP CLOSE_WAIT timeout [It]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:50

  Expected error:
      <*errors.errorString | 0xc420b0e430>: {
          s: "failed running \"sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\": <nil> (exit code 1)",
      }
      failed running "sudo cat /proc/net/nf_conntrack | grep 'dport=11302'": <nil> (exit code 1)
  not to have occurred

  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:178
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSep 17 23:29:45.727: INFO: Running AfterSuite actions on all node
Sep 17 23:29:45.727: INFO: Running AfterSuite actions on node 1

Summarizing 1 Failure:

[Fail] [sig-network] Network [It] should set TCP CLOSE_WAIT timeout
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:178

Ran 1 of 999 Specs in 92.542 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 998 Skipped --- FAIL: TestE2E (92.57s)
oomichi commented 5 years ago

問題のログを抽出

Sep 17 23:29:39.258: INFO: ssh ubuntu@192.168.1.108:22: command:   sudo cat /proc/net/nf_conntrack | grep 'dport=11302'
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stdout:    ""
Sep 17 23:29:39.259: INFO: ssh ubuntu@192.168.1.108:22: stderr:    "cat: /proc/net/nf_conntrack: No such file or directory\n"

/proc/net/nf_conntrack は netfilter 機能を構成するファイルの一つ Netfilterは、いわゆるファイアウォールやルータとしての役割を果たし、iptables コマンドで操作する。 nf_conntrack がロードされていることがわかる。

# lsmod  | grep nf_conntrack
nf_conntrack_netlink    40960  0
nfnetlink              16384  1 nf_conntrack_netlink
nf_conntrack_ipv4      20480  2
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_conntrack          106496  6 nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4

で、/proc/sys/net/netfilter/ 配下にそれらしいファイルが大量にある。

# ls /proc/sys/net/netfilter/
nf_conntrack_acct             nf_conntrack_icmp_timeout            nf_conntrack_tcp_timeout_established     nf_conntrack_timestamp
nf_conntrack_buckets          nf_conntrack_log_invalid             nf_conntrack_tcp_timeout_fin_wait        nf_conntrack_udp_timeout
nf_conntrack_checksum         nf_conntrack_max                     nf_conntrack_tcp_timeout_last_ack        nf_conntrack_udp_timeout_stream
nf_conntrack_count            nf_conntrack_tcp_be_liberal          nf_conntrack_tcp_timeout_max_retrans     nf_log
nf_conntrack_events           nf_conntrack_tcp_loose               nf_conntrack_tcp_timeout_syn_recv        nf_log_all_netns
nf_conntrack_expect_max       nf_conntrack_tcp_max_retrans         nf_conntrack_tcp_timeout_syn_sent
nf_conntrack_generic_timeout  nf_conntrack_tcp_timeout_close       nf_conntrack_tcp_timeout_time_wait
nf_conntrack_helper           nf_conntrack_tcp_timeout_close_wait  nf_conntrack_tcp_timeout_unacknowledged

しかし、問題の /proc/net/nf_conntrack は存在しない。

oomichi commented 5 years ago

CentOS 7でどうなっているのか、試してみる。

$ wget https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
$ openstack image create --container-format bare --disk-format qcow2 \
 --file CentOS-7-x86_64-GenericCloud.qcow2 CentOS-7-x86_64
$ openstack image list
+--------------------------------------+---------------------+--------+
| ID                                   | Name                | Status |
+--------------------------------------+---------------------+--------+
| 60408625-0466-4a31-9246-31dcac191cc9 | CentOS-7-x86_64     | active |
| 73f70800-1d0c-4569-a3c5-29c70775c334 | Ubuntu-16.04-x86_64 | active |
+--------------------------------------+---------------------+--------+
$ nova boot --key-name mykey --flavor m1.medium --image 60408625-0466-4a31-9246-31dcac191cc9 centos7
$ ssh centos@192.168.1.112
$ uname -a
Linux centos7.novalocal 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ sudo ls /proc/net/nf_conntrack
ls: cannot access /proc/net/nf_conntrack: No such file or directory
$ ls /proc/sys/net/netfilter/
nf_log  nf_log_all_netns

CentOS 7でも /proc/net/nf_conntrack は存在しない。

oomichi commented 5 years ago

linux kernelのコード net/netfilter/nf_conntrack_standalone.c

 38 #ifdef CONFIG_NF_CONNTRACK_PROCFS
...
392 static int nf_conntrack_standalone_init_proc(struct net *net)
393 {
394         struct proc_dir_entry *pde;
395
396         pde = proc_create("nf_conntrack", 0440, net->proc_net, &ct_file_ops);
397         if (!pde)
398                 goto out_nf_conntrack;
399
400         pde = proc_create("nf_conntrack", S_IRUGO, net->proc_net_stat,
401                           &ct_cpu_seq_fops);
402         if (!pde)
403                 goto out_stat_nf_conntrack;
404         return 0;
405
406 out_stat_nf_conntrack:
407         remove_proc_entry("nf_conntrack", net->proc_net);
408 out_nf_conntrack:
409         return -ENOMEM;
410 }

で、kernel config では無効になっている。 /boot/config-4.4.0-134-generic

# CONFIG_NF_CONNTRACK_PROCFS is not set
oomichi commented 5 years ago

Ubuntu だと絶対に通らないな・・

oomichi commented 5 years ago

そもそも timeout 値を取りたかったみたいだけど、代わりに取れそうなファイルはあるか?

$ cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait
3600

がそれっぽいけど。 とりあえず、configを有効にしてkernel ビルドしてみてどんな値が取れるのか、試してみる。

$ sudo apt-get install libssl-dev bc
$ cp /boot/config-4.4.0-134-generic .config
$ vi .config
- # CONFIG_NF_CONNTRACK_PROCFS is not set
+ CONFIG_NF_CONNTRACK_PROCFS=y
$ make oldconfig
$ make
...
$ sudo make modules_install
$ sudo make install
$ sudo vi /etc/default/grub
- GRUB_DEFAULT=0
+ GRUB_DEFAULT="gnulinux-advanced-c4c7809b-0c20-440a-88ea-a7da158f24b8>gnulinux-4.4.0-advanced-c4c7809b-0c20-440a-88ea-a7da158f24b8"
$ sudo update-grub
$ sudo sync
$ sudo reboot
oomichi commented 5 years ago

ソースから確認するか・・

oomichi commented 5 years ago

net/netfilter/nf_conntrack_proto_tcp.c

  71 static unsigned int tcp_timeouts[TCP_CONNTRACK_TIMEOUT_MAX] __read_mostly = {
  72         [TCP_CONNTRACK_SYN_SENT]        = 2 MINS,
  73         [TCP_CONNTRACK_SYN_RECV]        = 60 SECS,
  74         [TCP_CONNTRACK_ESTABLISHED]     = 5 DAYS,
  75         [TCP_CONNTRACK_FIN_WAIT]        = 2 MINS,
  76         [TCP_CONNTRACK_CLOSE_WAIT]      = 60 SECS,
...
1585         pn->ctl_table[0].data = &tn->timeouts[TCP_CONNTRACK_SYN_SENT];
1586         pn->ctl_table[1].data = &tn->timeouts[TCP_CONNTRACK_SYN_RECV];
1587         pn->ctl_table[2].data = &tn->timeouts[TCP_CONNTRACK_ESTABLISHED];
1588         pn->ctl_table[3].data = &tn->timeouts[TCP_CONNTRACK_FIN_WAIT];
1589         pn->ctl_table[4].data = &tn->timeouts[TCP_CONNTRACK_CLOSE_WAIT];
...
1405 static struct ctl_table tcp_sysctl_table[] = {
1406         {
1407                 .procname       = "nf_conntrack_tcp_timeout_syn_sent",
 ..
1411         },
1412         {
1413                 .procname       = "nf_conntrack_tcp_timeout_syn_recv",
 ..
1417         },
1418         {
1419                 .procname       = "nf_conntrack_tcp_timeout_established",
 ..
1423         },
1424         {
1425                 .procname       = "nf_conntrack_tcp_timeout_fin_wait",
 ..
1429         },
1430         {
1431                 .procname       = "nf_conntrack_tcp_timeout_close_wait",
1432                 .maxlen         = sizeof(unsigned int),
1433                 .mode           = 0644,
1434                 .proc_handler   = proc_dointvec_jiffies,
1435         },

これを見ると nf_conntrack_tcp_timeout_close_wait は 60 SECS、SECS は

#define SECS * HZ

と定義されており、

$ cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait
3600

であることから HZ が 60?

oomichi commented 5 years ago

テストコード

183                                 "sudo cat /proc/net/nf_conntrack "+
184                                         "| grep 'CLOSE_WAIT.*dst=%v.*dport=%v' "+
185                                         "| tail -n 1"+
186                                         "| awk '{print $5}' ",

から /proc/net/nf_conntrack ファイルに CLOSE_WAIT が含まれることを期待している。 しかし、これは特定の通信(ノードとPod間)の CLOSE_WAIT をとろうとしている。 /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait では、それらしい値が取れない。

oomichi commented 5 years ago

kernel buildが終わったら、どういう内容の proc ファイルができるか試してみる。 → 存在しない。別の方法で有効化にする?

$ uname -a
Linux build 4.4.0 #1 SMP Wed Sep 19 23:13:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ sudo grep CONFIG_NF_CONNTRACK_PROCFS /boot/config-4.4.0
CONFIG_NF_CONNTRACK_PROCFS=y
$ sudo find . -name nf_conntrack
$ sudo ls /proc/net/nf_conntrack
ls: cannot access '/proc/net/nf_conntrack': No such file or directory

更にカーネルモジュールのロードが必要だった。他の環境ではロード済みのモジュール

$ sudo modprobe nf_conntrack_ipv4
$ sudo cat /proc/net/nf_conntrack
ipv4     2 tcp      6 431999 ESTABLISHED src=192.168.1.112 dst=192.168.1.1 sport=22 dport=42010 src=192.168.1.1 dst=192.168.1.112 sport=42010 dport=22 [ASSURED] mark=0 zone=0 use=2
$
oomichi commented 5 years ago

k8s-master, k8s-node01 にインストールしてテストが通るか試してみる。 dockerデーモンが立ち上がらず、その結果 kubelet が立ち上がらず、kubernetes が動かない。

Sep 21 22:09:48 localhost docker[1004]: time="2018-09-21T22:09:48.909768646Z" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not
 supported"
Sep 21 22:09:48 localhost docker[1004]: time="2018-09-21T22:09:48.910103255Z" level=fatal msg="Error starting daemon: error initializing graphdriver: driver
not supported"

aufs ストレージドライバがない? 元の Ubuntu カーネル

$ lsmod | grep aufs
aufs                  217088  43

ビルドしたカーネルでは aufs がロードできない。

oomichi commented 5 years ago

自動スキップ PR がマージされたので close とする。