Apple Silicon Rancher Desktop + Cisco Anyconnect Issue

fluidbios commented 1 week ago

Actual Behavior

We've got a compatibility issue with Cisco Secure Client and Rancher Desktop on MacOS (apple silicon). When using VZ it gets stuck on "Starting Virtual Machine"

Rancher Desktop Works:

With 0 VPN clients
With Zscaler

Rancher Desktop Doesn't work:

With NEW Cisco Client

Mac: 14.6.1 (23G93) Rancher: Version: 1.15.1 Cisco Secure Client: 5.1.2.42

Steps to Reproduce

Install Rancher Desktop
On initial launch after installing rancher I leave default settings and proceed to open the application (it loads into QEMU and I can switch between VZ and QEMU and initialization works as expected.
Open Cisco Secure application and connect to Client VPN (unless configured to be intercepted regular traffic is ignored)
in VZ emulation, switch to qemu (no need to wait) and switch back to VZ emulation
"Starting Virtual Machine" Note: I don't configure anything, it's just a vanilla installation.

Result

10'ish minutes later it errors out.

rancher-desktop tail -f lima.log
    "filename": "/Users/ddjohnson/Library/Application Support/rancher-desktop/lima/0/diffdisk",
    "format": "raw",
    "actual-size": 15137280000,
    "dirty-flag": false
}
2024-08-27T13:44:01.316Z: Using vzNAT networking stack
2024-08-27T13:44:01.342Z: > /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura list --json
{"name":"0","status":"Stopped","dir":"/Users/ddjohnson/Library/Application Support/rancher-desktop/lima/0","vmType":"vz","arch":"aarch64","cpuType":"","cpus":2,"memory":6442450944,"disk":107374182400,"network":[{"vzNAT":true,"macAddress":"52:55:55:60:53:c2","interface":"vznat"}],"sshLocalPort":56309,"sshConfigFile":"/Users/ddjohnson/Library/Application Support/rancher-desktop/lima/0/ssh.config","config":{"vmType":"vz","os":"Linux","arch":"aarch64","images":[{"location":"/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/alpine-lima-v0.2.39.rd3-rd-3.20.0.iso","arch":"aarch64"}],"cpus":2,"memory":"6442450944","disk":"100GiB","mounts":[{"location":"~","mountPoint":"~","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}},{"location":"/tmp/rancher-desktop","mountPoint":"/tmp/rancher-desktop","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}},{"location":"/Volumes","mountPoint":"/Volumes","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}},{"location":"/var/folders","mountPoint":"/var/folders","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}},{"location":"/private/tmp","mountPoint":"/private/tmp","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}},{"location":"/private/var/folders","mountPoint":"/private/var/folders","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}},{"location":"/Applications/Rancher Desktop.app/Contents/Resources/resources","mountPoint":"/Applications/Rancher Desktop.app/Contents/Resources/resources","writable":true,"sshfs":{"cache":true,"followSymlinks":false,"sftpDriver":""},"9p":{"securityModel":"none","protocolVersion":"9p2000.L","msize":"128KiB","cache":"mmap"},"virtiofs":{}}],"mountType":"reverse-sshfs","mountInotify":false,"ssh":{"localPort":56309,"loadDotSSHPubKeys":false,"forwardAgent":false,"forwardX11":false,"forwardX11Trusted":false},"firmware":{"legacyBIOS":false},"audio":{"device":""},"video":{"display":"none","vnc":{}},"provision":[{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nmkdir -p /bootfs\nmount --bind / /bootfs\n# /bootfs/etc is empty on first boot because it has been moved to /mnt/data/etc by lima\nif [ -f /bootfs/etc/os-release ] && ! diff -q /etc/os-release /bootfs/etc/os-release; then\n  cp /etc/machine-id /bootfs/etc\n  cp /etc/ssh/ssh_host* /bootfs/etc/ssh/\n  mkdir -p /etc/docker /etc/rancher\n  cp -pr /etc/docker /bootfs/etc\n  cp -pr /etc/rancher /bootfs/etc\n\n  rm -rf /mnt/data/etc.prev\n  mkdir /mnt/data/etc.prev\n  mv /etc/* /mnt/data/etc.prev\n  mv /bootfs/etc/* /etc\n\n  # install updated files from /usr/local, e.g. nerdctl, buildkit, cni plugins\n  cp -pr /bootfs/usr/local /usr\n\n  # lima has applied changes while the \"old\" /etc was in place; restart to apply them to the updated one.\n  reboot\nfi\numount /bootfs\nrmdir /bootfs\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nRC_CGROUP_MODE=unified\nif ! grep -q -E \"^#?rc_cgroup_mode=\\\"$RC_CGROUP_MODE\\\"\" /etc/rc.conf; then\n  sed -i -E \"s/^#?rc_cgroup_mode=\\\".*\\\"/rc_cgroup_mode=\\\"$RC_CGROUP_MODE\\\"/\" /etc/rc.conf\n  # avoid reboot loop if sed failed for any reason\n  if grep -q -E \"^rc_cgroup_mode=\\\"$RC_CGROUP_MODE\\\"\" /etc/rc.conf; then\n    reboot\n  fi\nfi\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nfstrim /mnt/data\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nsed -i -E 's/^#?MaxSessions +[0-9]+/MaxSessions 25/g' /etc/ssh/sshd_config\nrc-service --ifstarted sshd reload\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nif ! [ -d /mnt/data/root ]; then\n  mkdir -p /root\n  mv /root /mnt/data/root\nfi\nmkdir -p /root\nmount --bind /mnt/data/root /root\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nmkdir -p /etc/docker\n\n# Delete certs.d if it is a symlink (from previous boot).\n[ -L /etc/docker/certs.d ] && rm /etc/docker/certs.d\n\n# Create symlink if certs.d doesn't exist (user may have created a regular directory).\nif [ ! -e /etc/docker/certs.d ]; then\n  # We don't know if the host is Linux or macOS, so we take a guess based on which mountpoint exists.\n  if [ -d \"/Users/ddjohnson\" ]; then\n    ln -s \"/Users/ddjohnson/.docker/certs.d\" /etc/docker\n  elif [ -d \"/home/ddjohnson\" ]; then\n    ln -s \"/home/ddjohnson/.docker/certs.d\" /etc/docker\n  fi\nfi\n"},{"mode":"system","script":"#!/bin/sh\nhostname lima-rancher-desktop\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\n# During boot is the only safe time to delete old k3s versions.\nrm -rf /var/lib/rancher/k3s/data\n# Delete all tmp files older than 3 days.\nfind /tmp -depth -mtime +3 -delete\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit -o nounset -o xtrace\nfor dir in / /etc /tmp /var/lib; do\n  mount --make-shared \"${dir}\"\ndone\n"},{"mode":"system","script":"#!/bin/sh\n# Move logrotate to hourly, because busybox crond only handles time jumps up\n# to one hour; this ensures that if the machine is suspended over long\n# periods, things will still happen often enough.  This is idempotent.\nmv -n /etc/periodic/daily/logrotate /etc/periodic/hourly/\nrc-update add crond default\nrc-service crond start\n"},{"mode":"system","script":"set -o errexit -o nounset -o xtrace\nusermod --append --groups docker \"ddjohnson\"\n"},{"mode":"system","script":"export CAROOT=/run/mkcert\nmkdir -p $CAROOT\ncd $CAROOT\nmkcert -install\nmkcert localhost\nchown -R nobody:nobody $CAROOT\n"},{"mode":"system","script":"set -o errexit -o nounset -o xtrace\n\n# openresty is backgrounding itself (and writes its own pid file)\nsed -i 's/^command_background/#command_background/' /etc/init.d/rd-openresty\n\n# configure proxy only when allowed-images exists\nallowed_images_conf=/usr/local/openresty/nginx/conf/allowed-images.conf\n# Remove the reference to an obsolete image conf filename\nobsolete_image_allow_list_conf=/usr/local/openresty/nginx/conf/image-allow-list.conf\nsetproxy=\"[ -f $allowed_images_conf ] && supervise_daemon_args=\\\"-e HTTPS_PROXY=http://127.0.0.1:3128/ \\${supervise_daemon_args:-}\\\" || true\"\nfor svc in containerd docker; do\n  sed -i \"\\#-f $allowed_images_conf#d\" /etc/init.d/$svc\n  sed -i \"\\#-f $obsolete_image_allow_list_conf#d\" /etc/init.d/$svc\n  echo \"$setproxy\" >> /etc/init.d/$svc\ndone\n\n# Make sure openresty log directory exists\ninstall -d -m755 /var/log/openresty\n"},{"mode":"system","script":"#!/bin/sh\nset -o errexit\n\nmount bpffs -t bpf /sys/fs/bpf\nmount --make-shared /sys/fs/bpf\nmount --make-shared /sys/fs/cgroup\n"}],"upgradePackages":false,"containerd":{"system":false,"user":false,"archives":[{"location":"https://github.com/containerd/nerdctl/releases/download/v1.7.5/nerdctl-full-1.7.5-linux-amd64.tar.gz","arch":"x86_64","digest":"sha256:adb246a4ef15b8f3d7eed4c6b61173014a6cf343e43ad95eae2087b454dcae5d"},{"location":"https://github.com/containerd/nerdctl/releases/download/v1.7.5/nerdctl-full-1.7.5-linux-arm64.tar.gz","arch":"aarch64","digest":"sha256:ff38142440b4705e12782b7a71074849e712a42ccb69a11306343a8d9f81d8ab"}]},"guestInstallPrefix":"/usr/local","portForwards":[{"guestIPMustBeZero":true,"guestIP":"0.0.0.0","guestPortRange":[1,65535],"hostIP":"0.0.0.0","hostPortRange":[1,65535],"proto":"tcp"},{"guestIP":"127.0.0.1","guestPortRange":[1,65535],"guestSocket":"/var/run/docker.sock","hostIP":"127.0.0.1","hostPortRange":[1,65535],"hostSocket":"/Users/ddjohnson/.rd/docker.sock","proto":"tcp"}],"networks":[{"vzNAT":true,"macAddress":"52:55:55:60:53:c2","interface":"vznat"}],"hostResolver":{"enabled":true,"ipv6":false,"hosts":{"host.docker.internal":"host.lima.internal","host.rancher-desktop.internal":"host.lima.internal","lima-rancher-desktop":"lima-0"}},"propagateProxyEnv":true,"caCerts":{"removeDefaults":false},"rosetta":{"enabled":true,"binfmt":true},"plain":false,"timezone":"America/New_York"},"sshAddress":"127.0.0.1","protected":false,"limaVersion":"v0.21.0","HostOS":"darwin","HostArch":"aarch64","LimaHome":"/Users/ddjohnson/Library/Application Support/rancher-desktop/lima","IdentityFile":"/Users/ddjohnson/Library/Application Support/rancher-desktop/lima/_config/user"}

Failure Result: 
 'time="2024-08-27T09:49:37-04:00" level=info msg="[hostagent] Waiting for the essential requirement 1 of 4: \\"ssh\\""\n' +
    'time="2024-08-27T09:49:41-04:00" level=info msg="[hostagent] 2024/08/27 09:49:41 tcpproxy: for incoming conn 127.0.0.1:64900, error dialing \\"192.168.5.15:22\\": connect tcp 192.168.5.15:22: no route to host"\n' +
    'time="2024-08-27T09:49:51-04:00" level=info msg="[hostagent] Waiting for the essential requirement 1 of 4: \\"ssh\\""\n' +
    'time="2024-08-27T09:49:54-04:00" level=info msg="[hostagent] 2024/08/27 09:49:54 tcpproxy: for incoming conn 127.0.0.1:64904, error dialing \\"192.168.5.15:22\\": connect tcp 192.168.5.15:22: no route to host"\n' +
    'time="2024-08-27T09:50:04-04:00" level=info msg="[hostagent] Waiting for the essential requirement 1 of 4: \\"ssh\\""\n' +
    'time="2024-08-27T09:50:07-04:00" level=info msg="[hostagent] 2024/08/27 09:50:07 tcpproxy: for incoming conn 127.0.0.1:64914, error '... 5804 more characters,
  code: 1,
  [Symbol(child-process.command)]: '/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura start --tty=false 0'
}

error-rancher

Expected Behavior

Switch between emulation while connected to Cisco Client without interruption

Additional Information

No response

Rancher Desktop Version

1.15.1

Rancher Desktop K8s Version

v1.30.4 (stable)

Which container engine are you using?

containerd (nerdctl)

What operating system are you using?

macOS

Operating System / Build Version

Mac: 14.6.1 (23G93)

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

krishna-perugupalli commented 1 week ago

I had similar issue, not with this combination but with Rancher desktop only.

Try Factory reset and see how it reacts: Rancher Desktop -> Troubleshooting -> "Factory Reset"

Issue: #7467

fluidbios commented 1 week ago

I had similar issue, not with this combination but with Rancher desktop only.

Try Factory reset and see how it reacts: Rancher Desktop -> Troubleshooting -> "Factory Reset"

Issue: #7467

Yes factory resets are how it hasn't become a critical blocker for our Development team but this is what we document as a workaround for Cisco/Rancher users. It's very disruptive to their day-to-day work.

krishna-perugupalli commented 1 week ago

Also, I observed that qemu is eating up lots of memory. You guys had similar problems?

fluidbios commented 1 week ago

Also, I observed that qemu is eating up lots of memory. You guys had similar problems?

Our Developers aren't using QEMU. This was a way to replicate the problem with as little configuration/steps as possible for Cisco (and now) Rancher to be able to replicate it themselves. I could be off but think about what happens when you launch a Client VPN application. It's doing something with your local OS network configuration right?

So you launch rancher and the VM hooks into the OS networking, everything works fine, then you launch Client VPN, it configures a VPN adapter and at this point it's ready to break. I'm not sure if there's some sort of "state configuration" that can't handle this network configuration change. But I was just looking at chatgpt about some VM initialization steps within rancher desktop and would probably suspect the "VZVirtioNetworkDeviceConfiguration" first as where it's getting stuck.

Define VM Configuration:

Create instances of VZVirtualMachineConfiguration , VZMemoryConfiguration , VZProcessorConfiguration . Set Up Boot Loader:

Use VZLinuxBootLoader or VZMacOSBootLoader . Configure Disk and Network:

Attach disk images with VZDiskImageStorageDeviceAttachment . Configure network with VZVirtioNetworkDeviceConfiguration

rancher-sandbox / rancher-desktop