Adding extra_binds when updating a cluster does not work on new nodes

cbluth commented 4 years ago

RKE version:

$ rke -version
rke version v1.0.4

Docker version: (docker version,docker info preferred)

$ docker version ; docker info
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.39 (downgraded from 1.40)
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:42:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false
Client:
 Debug Mode: false

Server:
 Containers: 43
  Running: 35
  Paused: 0
  Stopped: 8
 Images: 21
 Server Version: 18.09.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-76-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 27.48GiB
 Name: node-c1
 ID: HZVZ:ELMZ:KV54:YNP2:NV4Z:4OT2:N476:4DZA:32MV:4USE:HN2V:XJA5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No swap limit support

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

$ cat /etc/os-release ; uname -r
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
4.15.0-76-generic

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) KVM Virtual Machines on top of Baremetals

cluster.yml file:

nodes:
# master
- address: master-c1.example.org
  user: root
  role:
  - controlplane
  - etcd
- address: master-c2.example.org
  user: root
  role:
  - controlplane
  - etcd
- address: master-c3.example.org
  user: root
  role:
  - controlplane
  - etcd
# nodes
- address: node-c1.example.org
  user: root
  role:
  - worker
- address: node-c1b.example.org
  user: root
  role:
  - worker
- address: node-c1c.example.org
  user: root
  role:
  - worker
- address: node-c2.example.org
  user: root
  role:
  - worker
- address: node-c3.example.org
  user: root
  role:
  - worker
- address: node-c4.example.org
  user: root
  role:
  - worker
- address: node-c5.example.org
  user: root
  role:
  - worker
- address: node-c6.example.org
  user: root
  role:
  - worker
# cfg
services:
  etcd:
  kube-api:
    service_cluster_ip_range: 10.235.128.0/19
  kube-controller:
    cluster_cidr: 10.235.160.0/19
    service_cluster_ip_range: 10.235.128.0/19
  scheduler:
  kubelet:
    cluster_domain: example.org
    cluster_dns_server: 10.235.128.10
    extra_binds: 
     - "/var/volume:/var/volume:rshared"
  kubeproxy:
cluster_name: "cluster"
dns:
  provider: coredns
network:
  plugin: calico
ingress:
  provider: none
kubernetes_version: "v1.17.2-rancher1-2"
authentication:
  strategy: x509
  sans:
  - k8s1.example.org
  - 10.235.128.1
  - master-c1.example.org
  - master-c2.example.org
  - master-c3.example.org
  webhook: null
authorization:
  mode: rbac
  options: {}
ssh_agent_auth: true
ignore_docker_version: false

Steps to Reproduce:

try to use a PersistentVolume on a local path, i am using this:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /var/volume/grafana
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node.example.org

Results:

then, if you look on the path of my-node, you will see that the directory doesnt exist, nor does data persist on the host. i am trying to use the extra_binds configuration, as indicated by other issues on github, but it doesnt seem to work.

Is there a way to use RKE and local path (hostpath) volumes?

cbluth commented 4 years ago

i am discovering that this issue happens only when using `rke --update-only`

i tested this on a freshly made cluster, and the `extra_binds` options is working.

adding this option to a cluster-config.yaml and then updating does not work. it also does not work on new nodes added to the cluster.

stale[bot] commented 4 years ago

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

rancher / rke