search

rancher / rke

Rancher Kubernetes Engine (RKE), an extremely simple, lightning fast Kubernetes distribution that runs entirely within containers.
Apache License 2.0
3.21k stars 582 forks source link

"rke up" fails with "Cannot connect to the Docker daemon" but docker returns a response on the socket #872

Closed PerlStalker closed 5 years ago

PerlStalker commented 6 years ago

RKE version:

rke version v0.1.9

Docker version: (docker version,docker info preferred)

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.03.2-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-133-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.796 GiB
Name: kube01
ID: 4TWF:6W57:ODOR:GVPE:4SO6:734Z:434F:SLED:5ZWZ:2RPQ:BDNJ:SRV6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
4.4.0-133-generic

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Linux KVM

cluster.yml file:

# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: kube01
  port: "22"
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: "rke"
  docker_socket: /var/run/docker.sock
  ssh_key_path: ~/.ssh/id_rsa
  labels:
    app: ingress
# - address: kube02
#   port: "22"
#   internal_address: ""
#   role:
#   - controlplane
#   - worker
#   - etcd
#   hostname_override: ""
#   user: rke
#   docker_socket: /var/run/docker.sock
#   ssh_key: ""
#   ssh_key_path: ~/.ssh/id_rsa
#   labels:
#     app: ingress
# - address: kube03
#   port: "22"
#   internal_address: ""
#   role:
#   - controlplane
#   - worker
#   - etcd
#   hostname_override: ""
#   user: rke
#   docker_socket: /var/run/docker.sock
#   ssh_key: ""
#   ssh_key_path: ~/.ssh/id_rsa
#   labels:
#     app: ingress
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: true
    retention: "24h"
    creation: "5m0s"
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.100.14.0/23
    service_node_port_range: ""
    pod_security_policy: true
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.101.0.0/16
    service_cluster_ip_range: 10.100.14.0/23
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.100.14.10
    fail_swap_on: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: calico
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
addons: ""
addons_include: []
system_images:
  etcd: rancher/coreos-etcd:v3.2.18
  alpine: rancher/rke-tools:v0.1.13
  nginx_proxy: rancher/rke-tools:v0.1.13
  cert_downloader: rancher/rke-tools:v0.1.13
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.13
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.11.1-rancher1
  flannel: rancher/coreos-flannel:v0.9.1
  flannel_cni: rancher/coreos-flannel-cni:v0.2.0
  calico_node: rancher/calico-node:v3.1.1
  calico_cni: rancher/calico-cni:v3.1.1
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.1
  canal_cni: rancher/calico-cni:v3.1.1
  canal_flannel: rancher/coreos-flannel:v0.9.1
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/id_rsa
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries:
  - url: registry.redacted
    user: redacted
    password: redacted
ingress:
  provider: "nginx"
  options: {}
  node_selector:
    app: ingress
  extra_args: {}
cluster_name: "prod"
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
monitoring:
  provider: ""
  options: {}

Steps to Reproduce:

rke -d up --config cluster-test.yml

Results:

Attempting to bring up a new cluster fails. The warning claims that it failed to setup ssh tunneling and that it can't connect to the Docker socket. The message is below.

INFO[0000] Building Kubernetes cluster                  
DEBU[0000] Host: kube01 has role: controlplane          
DEBU[0000] Host: kube01 has role: worker                
DEBU[0000] Host: kube01 has role: etcd                  
INFO[0000] [dialer] Setup tunnel for host [kube01]      
DEBU[0000] Connecting to Docker API for host [kube01] 
WARN[0050] Failed to set up SSH tunneling for host [kube01]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? 
WARN[0050] Removing host [kube01] from node lists       
FATA[0050] Cluster must have at least one etcd plane host

The troubleshooting guide suggests that it's a problem with the address or port. However, I have verified that rke is successfully logging in with ssh and connecting to the docker using socat to sniff the API socket.

mv docker.sock docker-real.sock; socat -t100 -x -v UNIX-LISTEN:/var/run/docker.sock,mode=777,reuseaddr,fork UNIX-CONNECT:docker-real.sock

The output is below.

> 2018/08/20 15:31:18.838222  length=97 from=0 to=96                                                                                                                                                 
 47 45 54 20 2f 76 31 2e 32 34 2f 69 6e 66 6f 20  GET /v1.24/info 
 48 54 54 50 2f 31 2e 31 0d 0a                    HTTP/1.1..      
 48 6f 73 74 3a 20 64 6f 63 6b 65 72 0d 0a        Host: docker..  
 55 73 65 72 2d 41 67 65 6e 74 3a 20 47 6f 2d 68  User-Agent: Go-h
 74 74 70 2d 63 6c 69 65 6e 74 2f 31 2e 31 0d 0a  ttp-client/1.1..
 41 63 63 65 70 74 2d 45 6e 63 6f 64 69 6e 67 3a  Accept-Encoding:
 20 67 7a 69 70 0d 0a                              gzip..         
 0d 0a                                            ..              
--                                                                
< 2018/08/20 15:31:18.856200  length=2511 from=0 to=2510          
 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d  HTTP/1.1 200 OK.
 0a                                               .               
 41 70 69 2d 56 65 72 73 69 6f 6e 3a 20 31 2e 32  Api-Version: 1.2
 37 0d 0a                                         7..             
 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70  Content-Type: ap
 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a  plication/json..
 44 6f 63 6b 65 72 2d 45 78 70 65 72 69 6d 65 6e  Docker-Experimen
 74 61 6c 3a 20 66 61 6c 73 65 0d 0a              tal: false..    
 53 65 72 76 65 72 3a 20 44 6f 63 6b 65 72 2f 31  Server: Docker/1
 37 2e 30 33 2e 32 2d 63 65 20 28 6c 69 6e 75 78  7.03.2-ce (linux
 29 0d 0a                                         )..             
 44 61 74 65 3a 20 4d 6f 6e 2c 20 32 30 20 41 75  Date: Mon, 20 Au
 67 20 32 30 31 38 20 32 31 3a 33 31 3a 31 38 20  g 2018 21:31:18 
 47 4d 54 0d 0a                                   GMT..           
 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e  Transfer-Encodin
 67 3a 20 63 68 75 6e 6b 65 64 0d 0a              g: chunked..    
 0d 0a                                            ..              
 38 66 64 0d 0a                                   8fd..           
 7b 22 49 44 22 3a 22 34 54 57 46 3a 36 57 35 37  {"ID":"4TWF:6W57
 3a 4f 44 4f 52 3a 47 56 50 45 3a 34 53 4f 36 3a  :ODOR:GVPE:4SO6:
 37 33 34 5a 3a 34 33 34 46 3a 53 4c 45 44 3a 35  734Z:434F:SLED:5
 5a 57 5a 3a 32 52 50 51 3a 42 44 4e 4a 3a 53 52  ZWZ:2RPQ:BDNJ:SR
 56 36 22 2c 22 43 6f 6e 74 61 69 6e 65 72 73 22  V6","Containers"
 3a 30 2c 22 43 6f 6e 74 61 69 6e 65 72 73 52 75  :0,"ContainersRu
 6e 6e 69 6e 67 22 3a 30 2c 22 43 6f 6e 74 61 69  nning":0,"Contai
 6e 65 72 73 50 61 75 73 65 64 22 3a 30 2c 22 43  nersPaused":0,"C
 6f 6e 74 61 69 6e 65 72 73 53 74 6f 70 70 65 64  ontainersStopped
 22 3a 30 2c 22 49 6d 61 67 65 73 22 3a 30 2c 22  ":0,"Images":0,"
 44 72 69 76 65 72 22 3a 22 61 75 66 73 22 2c 22  Driver":"aufs","
 44 72 69 76 65 72 53 74 61 74 75 73 22 3a 5b 5b  DriverStatus":[[
 22 52 6f 6f 74 20 44 69 72 22 2c 22 2f 76 61 72  "Root Dir","/var
 2f 6c 69 62 2f 64 6f 63 6b 65 72 2f 61 75 66 73  /lib/docker/aufs
 22 5d 2c 5b 22 42 61 63 6b 69 6e 67 20 46 69 6c  "],["Backing Fil
 65 73 79 73 74 65 6d 22 2c 22 65 78 74 66 73 22  esystem","extfs"
 5d 2c 5b 22 44 69 72 73 22 2c 22 30 22 5d 2c 5b  ],["Dirs","0"],[
 22 44 69 72 70 65 72 6d 31 20 53 75 70 70 6f 72  "Dirperm1 Suppor
 74 65 64 22 2c 22 74 72 75 65 22 5d 5d 2c 22 53  ted","true"]],"S
 79 73 74 65 6d 53 74 61 74 75 73 22 3a 6e 75 6c  ystemStatus":nul
 6c 2c 22 50 6c 75 67 69 6e 73 22 3a 7b 22 56 6f  l,"Plugins":{"Vo
 6c 75 6d 65 22 3a 5b 22 6c 6f 63 61 6c 22 5d 2c  lume":["local"],
 22 4e 65 74 77 6f 72 6b 22 3a 5b 22 62 72 69 64  "Network":["brid
 67 65 22 2c 22 68 6f 73 74 22 2c 22 6d 61 63 76  ge","host","macv
 6c 61 6e 22 2c 22 6e 75 6c 6c 22 2c 22 6f 76 65  lan","null","ove
 72 6c 61 79 22 5d 2c 22 41 75 74 68 6f 72 69 7a  rlay"],"Authoriz
 61 74 69 6f 6e 22 3a 6e 75 6c 6c 7d 2c 22 4d 65  ation":null},"Me
 6d 6f 72 79 4c 69 6d 69 74 22 3a 74 72 75 65 2c  moryLimit":true,
 22 53 77 61 70 4c 69 6d 69 74 22 3a 66 61 6c 73  "SwapLimit":fals
 65 2c 22 4b 65 72 6e 65 6c 4d 65 6d 6f 72 79 22  e,"KernelMemory"
 3a 74 72 75 65 2c 22 43 70 75 43 66 73 50 65 72  :true,"CpuCfsPer
 69 6f 64 22 3a 74 72 75 65 2c 22 43 70 75 43 66  iod":true,"CpuCf
 73 51 75 6f 74 61 22 3a 74 72 75 65 2c 22 43 50  sQuota":true,"CP
 55 53 68 61 72 65 73 22 3a 74 72 75 65 2c 22 43  UShares":true,"C
 50 55 53 65 74 22 3a 74 72 75 65 2c 22 49 50 76  PUSet":true,"IPv
 34 46 6f 72 77 61 72 64 69 6e 67 22 3a 74 72 75  4Forwarding":tru
 65 2c 22 42 72 69 64 67 65 4e 66 49 70 74 61 62  e,"BridgeNfIptab
 6c 65 73 22 3a 74 72 75 65 2c 22 42 72 69 64 67  les":true,"Bridg
 65 4e 66 49 70 36 74 61 62 6c 65 73 22 3a 74 72  eNfIp6tables":tr
 75 65 2c 22 44 65 62 75 67 22 3a 66 61 6c 73 65  ue,"Debug":false
 2c 22 4e 46 64 22 3a 31 34 2c 22 4f 6f 6d 4b 69  ,"NFd":14,"OomKi
 6c 6c 44 69 73 61 62 6c 65 22 3a 74 72 75 65 2c  llDisable":true,
 22 4e 47 6f 72 6f 75 74 69 6e 65 73 22 3a 32 31  "NGoroutines":21
 2c 22 53 79 73 74 65 6d 54 69 6d 65 22 3a 22 32  ,"SystemTime":"2
 30 31 38 2d 30 38 2d 32 30 54 31 35 3a 33 31 3a  018-08-20T15:31:
 31 38 2e 38 34 33 39 39 31 38 31 35 2d 30 36 3a  18.843991815-06:
 30 30 22 2c 22 4c 6f 67 67 69 6e 67 44 72 69 76  00","LoggingDriv
 65 72 22 3a 22 6a 73 6f 6e 2d 66 69 6c 65 22 2c  er":"json-file",
 22 43 67 72 6f 75 70 44 72 69 76 65 72 22 3a 22  "CgroupDriver":"
 63 67 72 6f 75 70 66 73 22 2c 22 4e 45 76 65 6e  cgroupfs","NEven
 74 73 4c 69 73 74 65 6e 65 72 22 3a 30 2c 22 4b  tsListener":0,"K
 65 72 6e 65 6c 56 65 72 73 69 6f 6e 22 3a 22 34  ernelVersion":"4
 2e 34 2e 30 2d 31 33 33 2d 67 65 6e 65 72 69 63  .4.0-133-generic
 22 2c 22 4f 70 65 72 61 74 69 6e 67 53 79 73 74  ","OperatingSyst
 65 6d 22 3a 22 55 62 75 6e 74 75 20 31 36 2e 30  em":"Ubuntu 16.0
 34 2e 35 20 4c 54 53 22 2c 22 4f 53 54 79 70 65  4.5 LTS","OSType
 22 3a 22 6c 69 6e 75 78 22 2c 22 41 72 63 68 69  ":"linux","Archi
 74 65 63 74 75 72 65 22 3a 22 78 38 36 5f 36 34  tecture":"x86_64
 22 2c 22 49 6e 64 65 78 53 65 72 76 65 72 41 64  ","IndexServerAd
 64 72 65 73 73 22 3a 22 68 74 74 70 73 3a 2f 2f  dress":"https://
 69 6e 64 65 78 2e 64 6f 63 6b 65 72 2e 69 6f 2f  index.docker.io/
 76 31 2f 22 2c 22 52 65 67 69 73 74 72 79 43 6f  v1/","RegistryCo
 6e 66 69 67 22 3a 7b 22 49 6e 73 65 63 75 72 65  nfig":{"Insecure
 52 65 67 69 73 74 72 79 43 49 44 52 73 22 3a 5b  RegistryCIDRs":[
 22 31 32 37 2e 30 2e 30 2e 30 2f 38 22 5d 2c 22  "127.0.0.0/8"],"
 49 6e 64 65 78 43 6f 6e 66 69 67 73 22 3a 7b 22  IndexConfigs":{"
 64 6f 63 6b 65 72 2e 69 6f 22 3a 7b 22 4e 61 6d  docker.io":{"Nam
 65 22 3a 22 64 6f 63 6b 65 72 2e 69 6f 22 2c 22  e":"docker.io","
 4d 69 72 72 6f 72 73 22 3a 6e 75 6c 6c 2c 22 53  Mirrors":null,"S
 65 63 75 72 65 22 3a 74 72 75 65 2c 22 4f 66 66  ecure":true,"Off
 69 63 69 61 6c 22 3a 74 72 75 65 7d 7d 2c 22 4d  icial":true}},"M
 69 72 72 6f 72 73 22 3a 5b 5d 7d 2c 22 4e 43 50  irrors":[]},"NCP
 55 22 3a 34 2c 22 4d 65 6d 54 6f 74 61 6c 22 3a  U":4,"MemTotal":
 38 33 37 30 39 36 36 35 32 38 2c 22 44 6f 63 6b  8370966528,"Dock
 65 72 52 6f 6f 74 44 69 72 22 3a 22 2f 76 61 72  erRootDir":"/var
 2f 6c 69 62 2f 64 6f 63 6b 65 72 22 2c 22 48 74  /lib/docker","Ht
 74 70 50 72 6f 78 79 22 3a 22 22 2c 22 48 74 74  tpProxy":"","Htt
 70 73 50 72 6f 78 79 22 3a 22 22 2c 22 4e 6f 50  psProxy":"","NoP
 72 6f 78 79 22 3a 22 22 2c 22 4e 61 6d 65 22 3a  roxy":"","Name":
 22 6b 75 62 65 30 31 22 2c 22 4c 61 62 65 6c 73  "kube01","Labels
 22 3a 6e 75 6c 6c 2c 22 45 78 70 65 72 69 6d 65  ":null,"Experime
 6e 74 61 6c 42 75 69 6c 64 22 3a 66 61 6c 73 65  ntalBuild":false
 2c 22 53 65 72 76 65 72 56 65 72 73 69 6f 6e 22  ,"ServerVersion"
 3a 22 31 37 2e 30 33 2e 32 2d 63 65 22 2c 22 43  :"17.03.2-ce","C
 6c 75 73 74 65 72 53 74 6f 72 65 22 3a 22 22 2c  lusterStore":"",
 22 43 6c 75 73 74 65 72 41 64 76 65 72 74 69 73  "ClusterAdvertis
 65 22 3a 22 22 2c 22 52 75 6e 74 69 6d 65 73 22  e":"","Runtimes"
 3a 7b 22 72 75 6e 63 22 3a 7b 22 70 61 74 68 22  :{"runc":{"path"
 3a 22 64 6f 63 6b 65 72 2d 72 75 6e 63 22 7d 7d  :"docker-runc"}}
 2c 22 44 65 66 61 75 6c 74 52 75 6e 74 69 6d 65  ,"DefaultRuntime
 22 3a 22 72 75 6e 63 22 2c 22 53 77 61 72 6d 22  ":"runc","Swarm"
 3a 7b 22 4e 6f 64 65 49 44 22 3a 22 22 2c 22 4e  :{"NodeID":"","N
 6f 64 65 41 64 64 72 22 3a 22 22 2c 22 4c 6f 63  odeAddr":"","Loc
 61 6c 4e 6f 64 65 53 74 61 74 65 22 3a 22 69 6e  alNodeState":"in
 61 63 74 69 76 65 22 2c 22 43 6f 6e 74 72 6f 6c  active","Control
 41 76 61 69 6c 61 62 6c 65 22 3a 66 61 6c 73 65  Available":false
 2c 22 45 72 72 6f 72 22 3a 22 22 2c 22 52 65 6d  ,"Error":"","Rem
 6f 74 65 4d 61 6e 61 67 65 72 73 22 3a 6e 75 6c  oteManagers":nul
 6c 2c 22 4e 6f 64 65 73 22 3a 30 2c 22 4d 61 6e  l,"Nodes":0,"Man
 61 67 65 72 73 22 3a 30 2c 22 43 6c 75 73 74 65  agers":0,"Cluste
 72 22 3a 7b 22 49 44 22 3a 22 22 2c 22 56 65 72  r":{"ID":"","Ver
 73 69 6f 6e 22 3a 7b 7d 2c 22 43 72 65 61 74 65  sion":{},"Create
 64 41 74 22 3a 22 30 30 30 31 2d 30 31 2d 30 31  dAt":"0001-01-01
 54 30 30 3a 30 30 3a 30 30 5a 22 2c 22 55 70 64  T00:00:00Z","Upd
 61 74 65 64 41 74 22 3a 22 30 30 30 31 2d 30 31  atedAt":"0001-01
 2d 30 31 54 30 30 3a 30 30 3a 30 30 5a 22 2c 22  -01T00:00:00Z","
 53 70 65 63 22 3a 7b 22 4f 72 63 68 65 73 74 72  Spec":{"Orchestr
 61 74 69 6f 6e 22 3a 7b 7d 2c 22 52 61 66 74 22  ation":{},"Raft"
 3a 7b 22 45 6c 65 63 74 69 6f 6e 54 69 63 6b 22  :{"ElectionTick"
 3a 30 2c 22 48 65 61 72 74 62 65 61 74 54 69 63  :0,"HeartbeatTic
 6b 22 3a 30 7d 2c 22 44 69 73 70 61 74 63 68 65  k":0},"Dispatche
 72 22 3a 7b 7d 2c 22 43 41 43 6f 6e 66 69 67 22  r":{},"CAConfig"
 3a 7b 7d 2c 22 54 61 73 6b 44 65 66 61 75 6c 74  :{},"TaskDefault
 73 22 3a 7b 7d 2c 22 45 6e 63 72 79 70 74 69 6f  s":{},"Encryptio
 6e 43 6f 6e 66 69 67 22 3a 7b 22 41 75 74 6f 4c  nConfig":{"AutoL
 6f 63 6b 4d 61 6e 61 67 65 72 73 22 3a 66 61 6c  ockManagers":fal
 73 65 7d 7d 7d 7d 2c 22 4c 69 76 65 52 65 73 74  se}}}},"LiveRest
 6f 72 65 45 6e 61 62 6c 65 64 22 3a 66 61 6c 73  oreEnabled":fals
 65 2c 22 49 73 6f 6c 61 74 69 6f 6e 22 3a 22 22  e,"Isolation":""
 2c 22 49 6e 69 74 42 69 6e 61 72 79 22 3a 22 64  ,"InitBinary":"d
 6f 63 6b 65 72 2d 69 6e 69 74 22 2c 22 43 6f 6e  ocker-init","Con
 74 61 69 6e 65 72 64 43 6f 6d 6d 69 74 22 3a 7b  tainerdCommit":{
 22 49 44 22 3a 22 34 61 62 39 39 31 37 66 65 62  "ID":"4ab9917feb
 63 61 35 34 37 39 31 63 35 66 30 37 31 61 39 64  ca54791c5f071a9d
 31 66 34 30 34 38 36 37 38 35 37 66 63 63 22 2c  1f404867857fcc",
 22 45 78 70 65 63 74 65 64 22 3a 22 34 61 62 39  "Expected":"4ab9
 39 31 37 66 65 62 63 61 35 34 37 39 31 63 35 66  917febca54791c5f
 30 37 31 61 39 64 31 66 34 30 34 38 36 37 38 35  071a9d1f40486785
 37 66 63 63 22 7d 2c 22 52 75 6e 63 43 6f 6d 6d  7fcc"},"RuncComm
 69 74 22 3a 7b 22 49 44 22 3a 22 35 34 32 39 36  it":{"ID":"54296
 63 66 34 30 61 64 38 31 34 33 62 36 32 64 62 63  cf40ad8143b62dbc
 61 61 31 64 39 30 65 35 32 30 61 32 31 33 36 64  aa1d90e520a2136d
 64 66 65 22 2c 22 45 78 70 65 63 74 65 64 22 3a  dfe","Expected":
 22 35 34 32 39 36 63 66 34 30 61 64 38 31 34 33  "54296cf40ad8143
 62 36 32 64 62 63 61 61 31 64 39 30 65 35 32 30  b62dbcaa1d90e520
 61 32 31 33 36 64 64 66 65 22 7d 2c 22 49 6e 69  a2136ddfe"},"Ini
 74 43 6f 6d 6d 69 74 22 3a 7b 22 49 44 22 3a 22  tCommit":{"ID":"
 39 34 39 65 36 66 61 22 2c 22 45 78 70 65 63 74  949e6fa","Expect
 65 64 22 3a 22 39 34 39 65 36 66 61 22 7d 2c 22  ed":"949e6fa"},"
 53 65 63 75 72 69 74 79 4f 70 74 69 6f 6e 73 22  SecurityOptions"
 3a 5b 22 61 70 70 61 72 6d 6f 72 22 2c 22 73 65  :["apparmor","se
 63 63 6f 6d 70 22 5d 2c 22 45 78 65 63 75 74 69  ccomp"],"Executi
 6f 6e 44 72 69 76 65 72 22 3a 22 3c 6e 6f 74 20  onDriver":"<not
 73 75 70 70 6f 72 74 65 64 3e 22 7d 0a           supported>"}.
 0d 0a                                            ..
 30 0d 0a                                         0..
 0d 0a                                            ..
--

As you can see, rke is connecting to Docker and running docker info. The response is being sent back but rke isn't seeing it. The result is that rke is unable to successfully start a new cluster.

carloscarnero commented 6 years ago

Can the rke user on the Ubuntu machine use Docker? What happens if you run sudo -u rke docker version there?

Also (and unrelated), maybe it is better to use the overlay2 driver instead of aufs?

PerlStalker commented 6 years ago

That was the first thing I checked. It works just fine.

$ ssh rke@kube01 docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

I doubt the storage driver has anything to do with my current problem but I'll keep it mind.

Hefeweizen commented 5 years ago

I'm pretty sure I'm seeing this as well.

$ docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

on this initial error:

INFO[0000] [dialer] Setup tunnel for host [10.12.194.32] 
WARN[0075] Failed to set up SSH tunneling for host [10.12.194.32]: Can't retrieve Docker Info: error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info: Unable to access node with address [10.12.194.32:22] using SSH. Please check if the node is up and is accepting SSH connections or check network policies and firewall rules. Error: dial tcp 10.12.194.32:22: getsockopt: operation timed out

this attempt was made with rke_darwin-amd64-v0.1.11

superseb commented 5 years ago

Please share OS information used as well.

@PerlStalker Can you also try using an IP address to rule out certain things. Also if you can post the contents of /etc/hosts, ip a s and ip route, there might be something there.

Hefeweizen commented 5 years ago

@superseb os info:

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

@PerlStalker I like the socat verification you were doing. I attempted it myself to see if I'm seeing the exact same thing. Alas, I'm not. Though the rke response is the same, I'm not seeing anything on /var/run/docker.sock while rke is running. I do see info on /var/run/docker.sock if I'm on the remote machine and just run docker ps.

Other troubleshooting: started up a vagrant guest locally and tried with rke_linux-amd64-v0.1.11; same error.

Hefeweizen commented 5 years ago

So... solved my issue. Was moving too fast and just changed 'hostname_override' when updating cluster config, ignoring 'address'. Primarily, this is user error.

That said, I'm now curious why address must be specified as ip. But so as to not derail the thread, I'll open a separate issue. follow-up: it doesn't. 'address' can be a hostname.

PerlStalker commented 5 years ago

Unfortunately, I've torn down that environment so I can no longer test. I guess you can close this.

On Tue, Oct 30, 2018 at 10:47 AM Hefeweizen notifications@github.com wrote:

@superseb https://github.com/superseb os info:

lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.5 LTS Release: 16.04 Codename: xenial

@PerlStalker https://github.com/PerlStalker I like the socat verification you were doing. I attempted it myself to see if I'm seeing the exact same thing. Alas, I'm not. Though the rke response is the same, I'm not seeing anything on /var/run/docker.sock while rke is running. I do see info on /var/run/docker.sock if I'm on the remote machine and just run docker ps.

Other troubleshooting: started up a vagrant guest locally and tried with rke_linux-amd64-v0.1.11; same error.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rancher/rke/issues/872#issuecomment-434378353, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLKrA5XhRJapumIo2g0e3Xk8PoTSEUHks5uqIKogaJpZM4WExC1 .

-- Randall Smith http://www.musicfreestatic.com/ KE0NDF

aiqs4 commented 3 years ago

same problem here! In my case, this daemon didn't run: "ssh_agent_auth: false"