search

rancher-sandbox / rancher-desktop

Container Management and Kubernetes on the Desktop
https://rancherdesktop.io
Apache License 2.0
5.92k stars 280 forks source link

Specifying `agents-memory` flag in WSL results in cluster failure #4748

Open claywd opened 1 year ago

claywd commented 1 year ago

Actual Behavior

The --agents-memory 1024m flag results in an OCI runtime error where the container literally fails to create, probably for lack of disk space when trying to map memory for the containers.

Error here 

Failed Cluster Start: Failed to add one or more agents: runtime failed to start node 'k3d-kubefirst-agent-0': docker failed to start container for node 'k3d-kubefirst-agent-0': Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/mnt/wsl/rancher-desktop/run/docker-mounts/47932a34-5fca-40ff-a362-60f6c125e970" to rootfs at "/proc/meminfo": mount /mnt/wsl/rancher-desktop/run/docker-mounts/47932a34-5fca-40ff-a362-60f6c125e970:/proc/meminfo (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Steps to Reproduce

spin up rancher desktop on windows and enable with a wsl instance then open the instance and after confirming docker is using the docker packaged in rancher desktop with which docker

run docker ps -a to make sure the connection is valid

k3d cluster create --agent-memory 1024m

and watch it fail.

Result

❯ /root/.k1/kubefirst/tools/k3d cluster create kubefirst --image rancher/k3s:v1.26.3-k3s1 --agents 1 --agents-memory 1024m --registry-create k3d-kubefirst-registry --volume /root/.k1/kubefirst/minio-storage:/var/lib/rancher/k3s/storage@all --port 443:443@loadbalancer --verbose
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:20.10.21 OSType:linux OS:Rancher Desktop WSL Distribution Arch:x86_64 CgroupVersion:1 CgroupDriver:cgroupfs Filesystem:extfs InfoName:LaptopStudio}
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  k3s-node-labels: []
  k3sargs: []
  ports:
  - 443:443@loadbalancer
  registries:
    create: k3d-kubefirst-registry
  runtime-labels: []
  volumes:
  - /root/.k1/kubefirst/minio-storage:/var/lib/rancher/k3s/storage@all
hostaliases: []
DEBU[0000] Configuration:
agents: 1
image: rancher/k3s:v1.26.3-k3s1
network: ""
options:
  k3d:
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    loadbalancer:
      configoverrides: []
    timeout: 0s
    wait: true
  kubeconfig:
    switchcurrentcontext: true
    updatedefaultkubeconfig: true
  runtime:
    agentsmemory: 1024m
    gpurequest: ""
    hostpidmode: false
    serversmemory: ""
registries:
  config: ""
  use: []
servers: 1
subnet: ""
token: ""
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:1 ExposeAPI:{Host: HostIP: HostPort:} Image:rancher/k3s:v1.26.3-k3s1 Network: Subnet: ClusterToken: Volumes:[] Ports:[] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory:1024m HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:<nil> Config:} HostAliases:[]}
==========================
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind:Simple APIVersion:k3d.io/v1alpha4} ObjectMeta:{Name:} Servers:1 Agents:1 ExposeAPI:{Host: HostIP: HostPort:37005} Image:rancher/k3s:v1.26.3-k3s1 Network: Subnet: ClusterToken: Volumes:[{Volume:/root/.k1/kubefirst/minio-storage:/var/lib/rancher/k3s/storage NodeFilters:[all]}] Ports:[{Port:443:443 NodeFilters:[loadbalancer]}] Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false NodeHookActions:[] Loadbalancer:{ConfigOverrides:[]}} K3sOptions:{ExtraArgs:[] NodeLabels:[]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest: ServersMemory: AgentsMemory:1024m HostPidMode:false Labels:[]}} Env:[] Registries:{Use:[] Create:0xc00054c000 Config:} HostAliases:[]}
==========================
INFO[0000] portmapping '443:443' targets the loadbalancer: defaulting to [servers:*:proxy agents:*:proxy]
DEBU[0000] generated loadbalancer config:
ports:
  443.tcp:
  - k3d-kubefirst-server-0
  - k3d-kubefirst-agent-0
  6443.tcp:
  - k3d-kubefirst-server-0
settings:
  workerConnections: 1024
DEBU[0000] Port Exposure Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for Port Exposure: '33333'
DEBU[0000] ===== Merged Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:kubefirst Network:{Name:k3d-kubefirst ID: External:false IPAM:{IPPrefix:zero IPPrefix IPsUsed:[] Managed:false} Members:[]} Token: Nodes:[0xc0005224e0 0xc0005229c0 0xc000522d00] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc000544080 ServerLoadBalancer:0xc000210490 ImageVolume: Volumes:[]} ClusterCreateOpts:{DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false GPURequest: ServersMemory: AgentsMemory:1024m NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] HostAliases:[] Registries:{Create:0xc0003ac0f0 Use:[] Config:<nil>}} KubeconfigOpts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-kubefirst'
INFO[0000] Created image volume k3d-kubefirst-images
INFO[0000] Creating node 'k3d-kubefirst-registry'
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Created container k3d-kubefirst-registry (ID: e0222d3a9ca3c42b3a9250f04a420364d055a9d6e3a3be8183943e4786c7ad03)
INFO[0000] Successfully created registry 'k3d-kubefirst-registry'
DEBU[0000] no netlabel present on container /k3d-kubefirst-registry
DEBU[0000] failed to get IP for container /k3d-kubefirst-registry as we couldn't find the cluster network
DEBU[0000] no netlabel present on container /k3d-kubefirst-registry
DEBU[0000] failed to get IP for container /k3d-kubefirst-registry as we couldn't find the cluster network
INFO[0000] Starting new tools node...
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] [Docker] DockerHost: '' ()
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock
DEBU[0000] Created container k3d-kubefirst-tools (ID: a89ac13a9fd6ea26161a3bd8f9508b4b40a539ef577d8d77da9922b675942cbb)
DEBU[0000] Node k3d-kubefirst-tools Start Time: 2023-05-23 22:06:19.337513418 -0500 CDT m=+0.164522975

INFO[0000] Starting Node 'k3d-kubefirst-tools'
DEBU[0000] Truncated 2023-05-24 03:06:19.69725535 +0000 UTC to 2023-05-24 03:06:19 +0000 UTC
INFO[0001] Creating node 'k3d-kubefirst-server-0'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] Created container k3d-kubefirst-server-0 (ID: bddaf9b2248dfe574362760da5483b08620d9b6cd08992d661ab6d93e07fdf72)
DEBU[0001] Created node 'k3d-kubefirst-server-0'
INFO[0001] Creating node 'k3d-kubefirst-agent-0'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] Created container k3d-kubefirst-agent-0 (ID: 35c3425b6c6c59aceaa10ca91850a31931bc125b2add3d426e5e73c4e5772757)
DEBU[0001] Created node 'k3d-kubefirst-agent-0'
INFO[0001] Creating LoadBalancer 'k3d-kubefirst-serverlb'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
DEBU[0001] Created container k3d-kubefirst-serverlb (ID: 19fda4798438733ca0707e36fc493cda2d97f5bfb5e248a597f7898107c91e87)
DEBU[0001] Created loadbalancer 'k3d-kubefirst-serverlb'
DEBU[0001] DOCKER_SOCK=/var/run/docker.sock
INFO[0001] Using the k3d-tools node to gather environment information
DEBU[0001] no netlabel present on container /k3d-kubefirst-tools
DEBU[0001] failed to get IP for container /k3d-kubefirst-tools as we couldn't find the cluster network

DEBU[0001] Deleting node k3d-kubefirst-tools ...
DEBU[0002] DOCKER_SOCK=/var/run/docker.sock
INFO[0002] HostIP: using network gateway 192.168.16.1 address
INFO[0002] Starting cluster 'kubefirst'
INFO[0002] Starting servers...
DEBU[0002] DOCKER_SOCK=/var/run/docker.sock
DEBU[0002] No fix enabled.
DEBU[0002] Node k3d-kubefirst-server-0 Start Time: 2023-05-23 22:06:21.425494938 -0500 CDT m=+2.252504496
INFO[0002] Starting Node 'k3d-kubefirst-server-0'
DEBU[0002] Truncated 2023-05-24 03:06:21.966429591 +0000 UTC to 2023-05-24 03:06:21 +0000 UTC
DEBU[0002] Waiting for node k3d-kubefirst-server-0 to get ready (Log: 'k3s is up and running')
DEBU[0006] Finished waiting for log message 'k3s is up and running' from node 'k3d-kubefirst-server-0'

INFO[0006] Starting agents...
DEBU[0006] DOCKER_SOCK=/var/run/docker.sock
DEBU[0006] No fix enabled.
DEBU[0006] Node k3d-kubefirst-agent-0 Start Time: 2023-05-23 22:06:25.201666257 -0500 CDT m=+6.028675816
INFO[0006] Starting Node 'k3d-kubefirst-agent-0'
ERRO[0006] Failed Cluster Start: Failed to add one or more agents: runtime failed to start node 'k3d-kubefirst-agent-0': docker failed to start container for node 'k3d-kubefirst-agent-0': Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/mnt/wsl/rancher-desktop/run/docker-mounts/c23eb12d-5433-4def-a2ca-a83069fbc933" to rootfs at "/proc/meminfo": mount /mnt/wsl/rancher-desktop/run/docker-mounts/c23eb12d-5433-4def-a2ca-a83069fbc933:/proc/meminfo (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERRO[0006] Failed to create cluster >>> Rolling Back
INFO[0006] Deleting cluster 'kubefirst'
DEBU[0006] no netlabel present on container /k3d-kubefirst-registry
DEBU[0006] failed to get IP for container /k3d-kubefirst-registry as we couldn't find the cluster network
DEBU[0006] Cluster Details: &{Name:kubefirst Network:{Name:k3d-kubefirst ID:d57bb6d35e9c6c653c1add53ea24abfee496c45e44045e18286bf3bc1194ec40 External:false IPAM:{IPPrefix:192.168.16.0/20 IPsUsed:[] Managed:false} Members:[]} Token:HjubuZvGMkNgzCqlqpSn Nodes:[0xc0005224e0 0xc0005229c0 0xc000522d00 0xc000523520] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:0xc000544080 ServerLoadBalancer:0xc000210490 ImageVolume:k3d-kubefirst-images Volumes:[k3d-kubefirst-images]}
DEBU[0006] Deleting node k3d-kubefirst-serverlb ...
DEBU[0006] Deleting node k3d-kubefirst-server-0 ...
DEBU[0007] Deleting node k3d-kubefirst-agent-0 ...
DEBU[0007] Cleaning fake files folder from k3d config dir for this node...
DEBU[0007] Deleting node k3d-kubefirst-registry ...
INFO[0007] Deleting cluster network 'k3d-kubefirst'
INFO[0007] Deleting 1 attached volumes...
DEBU[0007] Deleting volume k3d-kubefirst-images...
FATA[0007] Cluster creation FAILED, all changes have been rolled back!

Expected Behavior

k3d cluster comes up and offers me the command

```kubectl cluster-info````

Additional Information

No response

Rancher Desktop Version

1.8.1

Rancher Desktop K8s Version

1.26.4

Which container engine are you using?

moby (docker cli)

What operating system are you using?

Other (specify below)

Operating System / Build Version

Windows 11 22H2 BUILD 22621.1702/WSL Ubuntu Jammy 22.04 5.15.90.1-microsoft-standard-WSL2

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

I am using none of these special features.

jandubois commented 1 year ago

Just want to make sure I understand this correctly:

You have a separate WSL distro, e.g. Ubuntu, and you enable WSL integration for this distro in Rancher Desktop. Then you install k3s inside that distro and run the docker ps -a and k3d create ... commands inside that distro?

Which distro are you using for this? And which version of k3d are you using and how did you install it?

claywd commented 1 year ago

yes you got it. This usually works as long as I don't use the agent-memory flag

k3d version 5.5.1 and installed via asdf