search

rancher / rke

Rancher Kubernetes Engine (RKE), an extremely simple, lightning fast Kubernetes distribution that runs entirely within containers.
Apache License 2.0
3.22k stars 582 forks source link

Some containers still running after rke remove #832

Closed mateuszkwiatkowski closed 6 years ago

mateuszkwiatkowski commented 6 years ago

RKE version: rke version v0.1.8

Docker version:

# docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Wed Jun 20 21:43:51 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Wed Jun 20 21:42:00 2018
  OS/Arch:      linux/amd64
  Experimental: false

# docker info
Containers: 13
 Running: 8
 Paused: 0
 Stopped: 5
Images: 7
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-22-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 64
Total Memory: 188.9GiB
Name: sled10348
ID: UPIN:G4LO:DNAL:A22B:763O:4WXX:TOIE:W3Z5:ROGC:OPQ6:YC73:3XQS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Operating system and kernel:

# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
#  uname -a
Linux sled10288 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Type/provider of hosts: Bare-metal

cluster.yml file:

nodes:
   - address: master-1
     role: ["controlplane", "etcd"]
   - address: master-2
     role: ["controlplane", "etcd"]
   - address: master-3
     role: ["controlplane", "etcd"]
   - address: node-1
     role: ["worker"]
   - address: node-2
     role: ["worker"]
   - address: node-3
     role: ["worker"]

ssh_key_path:
ssh_agent_auth: true
ignore_docker_version: true
cluster_name: prod-bm

services:
  kube-api:
    service_cluster_ip_range: 10.255.28.0/23
  kube-controller:
    cluster_cidr: 10.255.30.0/23
    service_cluster_ip_range: 10.255.28.0/23
  kubelet:
    cluster_dns_server: 10.255.30.10
    cluster_domain: [SNIP]
network:
  plugin: canal

authentication:
  stretegy: x509
  sans:
  - "*.prodn-bm.[SNIP]"

authorization:
  mode: rbac

Steps to Reproduce:

Remove cluster with ./rke remove --config cluster.yaml.

Results:

There are containers running on hosts:

$ for host in master-1 master-2 master-3 node-1 node-2 node-3; do echo $host; ssh $host docker ps; done
master-1
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS               NAMES
dcfea57a353a        rancher/coreos-flannel    "/opt/bin/flanneld -…"   3 hours ago         Up 3 hours                              k8s_kube-flannel_canal-4wxd2_kube-system_7844aade-956a-11e8-80e3-008cfa0b1aec_0
3af181e2262c        rancher/calico-cni        "/install-cni.sh"        3 hours ago         Up 3 hours                              k8s_install-cni_canal-4wxd2_kube-system_7844aade-956a-11e8-80e3-008cfa0b1aec_0
739d0b052f4b        rancher/calico-node       "start_runit"            3 hours ago         Up 3 hours                              k8s_calico-node_canal-4wxd2_kube-system_7844aade-956a-11e8-80e3-008cfa0b1aec_0
21e9d57ae330        rancher/pause-amd64:3.1   "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_canal-4wxd2_kube-system_7844aade-956a-11e8-80e3-008cfa0b1aec_0
master-2
CONTAINER ID        IMAGE                     COMMAND             CREATED             STATUS              PORTS               NAMES
b692d178e3b2        rancher/calico-cni        "/install-cni.sh"   3 hours ago         Up 3 hours                              k8s_install-cni_canal-6vjvj_kube-system_7844a092-956a-11e8-80e3-008cfa0b1aec_0
1ff7ce51d031        rancher/calico-node       "start_runit"       3 hours ago         Up 3 hours                              k8s_calico-node_canal-6vjvj_kube-system_7844a092-956a-11e8-80e3-008cfa0b1aec_0
3e4a01f22871        rancher/pause-amd64:3.1   "/pause"            3 hours ago         Up 3 hours                              k8s_POD_canal-6vjvj_kube-system_7844a092-956a-11e8-80e3-008cfa0b1aec_0
master-3
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS               NAMES
31a4a276292c        rancher/coreos-flannel    "/opt/bin/flanneld -…"   3 hours ago         Up 3 hours                              k8s_kube-flannel_canal-dxx9q_kube-system_7844ab95-956a-11e8-80e3-008cfa0b1aec_0
83741e0fc5bd        rancher/calico-cni        "/install-cni.sh"        3 hours ago         Up 3 hours                              k8s_install-cni_canal-dxx9q_kube-system_7844ab95-956a-11e8-80e3-008cfa0b1aec_0
5032ca8223ca        rancher/calico-node       "start_runit"            3 hours ago         Up 3 hours                              k8s_calico-node_canal-dxx9q_kube-system_7844ab95-956a-11e8-80e3-008cfa0b1aec_0
135d7075cf55        rancher/pause-amd64:3.1   "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_canal-dxx9q_kube-system_7844ab95-956a-11e8-80e3-008cfa0b1aec_0
node-1
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
73bb1a732749        rancher/pause-amd64:3.1            "/pause"                 10 minutes ago      Up 9 minutes                            k8s_POD_default-http-backend-564b9b6c5b-jwdj4_ingress-nginx_9175f9b0-956a-11e8-80e3-008cfa0b1aec_1944
58b5f715f6e8        rancher/nginx-ingress-controller   "/usr/bin/dumb-init …"   3 hours ago         Up 3 hours                              k8s_nginx-ingress-controller_nginx-ingress-controller-xvznk_ingress-nginx_91760c58-956a-11e8-80e3-008cfa0b1aec_0
81f24ef5ffe4        rancher/pause-amd64:3.1            "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_nginx-ingress-controller-xvznk_ingress-nginx_91760c58-956a-11e8-80e3-008cfa0b1aec_0
1aaa428f0b0e        rancher/calico-cni                 "/install-cni.sh"        3 hours ago         Up 3 hours                              k8s_install-cni_canal-jxh7t_kube-system_7842d85a-956a-11e8-80e3-008cfa0b1aec_0
7282a62bd4ae        rancher/calico-node                "start_runit"            3 hours ago         Up About an hour                        k8s_calico-node_canal-jxh7t_kube-system_7842d85a-956a-11e8-80e3-008cfa0b1aec_0
73e24e4cdc9f        rancher/pause-amd64:3.1            "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_canal-jxh7t_kube-system_7842d85a-956a-11e8-80e3-008cfa0b1aec_0
node-2
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
79c1661b1bac        rancher/nginx-ingress-controller   "/usr/bin/dumb-init …"   3 hours ago         Up 3 hours                              k8s_nginx-ingress-controller_nginx-ingress-controller-5xgnn_ingress-nginx_91734346-956a-11e8-80e3-008cfa0b1aec_0
4b7ae9b7d550        rancher/pause-amd64:3.1            "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_nginx-ingress-controller-5xgnn_ingress-nginx_91734346-956a-11e8-80e3-008cfa0b1aec_0
1a045b063c5b        rancher/calico-cni                 "/install-cni.sh"        3 hours ago         Up 3 hours                              k8s_install-cni_canal-svq7v_kube-system_78414c63-956a-11e8-80e3-008cfa0b1aec_0
f86b11b3df0a        rancher/calico-node                "start_runit"            3 hours ago         Up 3 hours                              k8s_calico-node_canal-svq7v_kube-system_78414c63-956a-11e8-80e3-008cfa0b1aec_0
891f025b8d52        rancher/pause-amd64:3.1            "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_canal-svq7v_kube-system_78414c63-956a-11e8-80e3-008cfa0b1aec_0
node-3
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
3a0c9f5debc0        rancher/nginx-ingress-controller   "/usr/bin/dumb-init …"   3 hours ago         Up 3 hours                              k8s_nginx-ingress-controller_nginx-ingress-controller-dcxcr_ingress-nginx_91761098-956a-11e8-80e3-008cfa0b1aec_0
3a5a51a23b10        rancher/pause-amd64:3.1            "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_nginx-ingress-controller-dcxcr_ingress-nginx_91761098-956a-11e8-80e3-008cfa0b1aec_0
b90032c10cac        rancher/calico-cni                 "/install-cni.sh"        3 hours ago         Up 3 hours                              k8s_install-cni_canal-67pxg_kube-system_7842d6b5-956a-11e8-80e3-008cfa0b1aec_0
a3ae9766fb49        rancher/calico-node                "start_runit"            3 hours ago         Up 3 hours                              k8s_calico-node_canal-67pxg_kube-system_7842d6b5-956a-11e8-80e3-008cfa0b1aec_0
41ec890ed5fa        rancher/pause-amd64:3.1            "/pause"                 3 hours ago         Up 3 hours                              k8s_POD_canal-67pxg_kube-system_7842d6b5-956a-11e8-80e3-008cfa0b1aec_0
superseb commented 6 years ago

This is by design, rke remove does not remove pods: https://github.com/rancher/rke/issues/319

mateuszkwiatkowski commented 6 years ago

@superseb thank you for explanation. I think this is documentation problem then because there's no mention that this is expected: https://rancher.com/docs/rke/v0.1.x/en/installation/managing-clusters/#removing-kubernetes-clusters-from-nodes Could you please reopen?

superseb commented 6 years ago

https://github.com/rancher/docs/pull/564

mateuszkwiatkowski commented 6 years ago

@superseb thank you very much!

iahmad-khan commented 5 years ago

Do we need to remove them manually each time?