Rancher Desktop 1.8.1 darwin(x64) fails to startup on Apple Silicon M2 Max

[jaskirat8]

Actual Behavior

The following error in lima.log presents itself when Rancher Desktop is getting initialized.

2023-06-03T10:43:18.577Z: Lima: executing: sudo rm -f /tmp/rd-nerdctl-EfiRzz.nerdctl: Error: spawn /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura EACCES
2023-06-03T10:43:18.577Z: Error trying to start/update containerd: Error: spawn /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura EACCES:  Error: spawn /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura EACCES
    at ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -13,
  code: 'EACCES',
  syscall: 'spawn /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura',
  path: '/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura',
  spawnargs: [
    '--debug',
    'shell',
    '--workdir=.',
    '0',
    'sudo',
    'rm',
    '-f',
    '/tmp/rd-nerdctl-EfiRzz.nerdctl'
  ]
}

Steps to Reproduce

Install Rancher Desktop on M2 machine first time

Result

Expected Behavior

Rancher Desktop should be able to setup correctly

Additional Information

No response

Rancher Desktop Version

1.8.1

Rancher Desktop K8s Version

1.27.1

Which container engine are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

macOS Ventura 13.4

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

[adamkpickering]

Thanks for creating an issue! While we have had some kinks to work out with M1, M2 is effectively the same so I doubt that your problem is coming from that.

When you run

ls -l /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura

what is the result?

Also, How did you install Rancher Desktop?

Finally, do you have any device management software that might be interfering?

Often cases like this are due to the user's environment (not that I'm ruling out a Rancher Desktop issue).

[jaskirat8]

I installed by downloading package from Rancher's official website and again tried from Github Releases.

Yes, device management software is jamf.

I have all admin and sudo rights so not able to pinpoint what part is missing if its access related issue.

[adamkpickering]

Another user has a similar problem to you, except they were on an intel mac. They were able to solve the issue by dropping .ventura off of that filename. I'm curious if that works for you?

[jaskirat8]

But a file with that name already exists so not sure. Should i delete that and replace it with this ?

[jaskirat8]

Okay i tried moving this to limactl.ventura.bak in that case Rancher gave this error

So seems this file with .ventura is needed

[jandubois]

The file is needed, but it looks like Rancher Desktop cannot invoke it.

What happens when you call it manually from the shell:

$ /Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura --version
limactl version 0.16.0-22-g85ed52a

My version is different, but does the command run, or do you get an error?

[jaskirat8]

limactl version 0.15.0

[jaskirat8]

So i have noticed this pattern that Rancher is trying to paste nerdctl and its having problem in that.

Since the user i am running as have sudo permissions and that to without password so i am not able to co-relate what else is needed from permissions side.

[jaskirat8]

I even tried pasting the nerdctl manually but still it tries to paste on its own.

[ee-usgs]

I have this same issue on a x86 Mac running Ventura and using Rancher Desktop 1.9.1.

[seshaTarget]

i have the same issue with any rancher version in my mac venture core i7

i tried to run the limactl and getting following error in terminal

./limactl info FATA[0000] open /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/share/lima/examples/default.yaml: no such file or directory

[seshaTarget]

bin % ./limactl.ventura -v
limactl version 85ed52a

bin % ./limactl -v
limactl version 0.16.0

[pshutt97]

I can confirm this issue on an M1 Mac running 1.10.1.

[mook-as]

Does anything relevant show up in Console.app (or, if you prefer, /var/logs/ and /Library/Logs/) run around the time it is being invoked? Unfortunately, those logs are very spammy, so it would be difficult to narrow things down. I'm hoping it's something to do with Jamf and other device management things, but we don't have enough information yet to say it is.

If you do find something, please copy the entries (and a few surrounding ones) here. Note that Console.app supports ⌘C fine, so please do not use screenshots.

[seshaTarget]

hi @mook-as

I listed all the files under the two directories - /var/logs/ and /Library/Logs/ during the time of rancher invokation No file has been created nor any modifications to any files within the directory as well as its subdirectories

Only Information we get is during the Install CA Certificates Step ( after starting VM ) it fails as limactl.ventura doesnt have permissions to run some commands like rm , chmod, sudo ,etc despite giving full admin access to rancher desktop

[seshaTarget]

error 00:29:27.983222+0530 kernel mcxalr{310} [ERROR] msleep waiting for client reply (uid:502 seqid:19223 path:/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura) failed (4). Result: -1 error 00:29:27.983233+0530 kernel mcxalr{311} [ERROR] Unable to check with client for uid=502 path=/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura default 00:29:27.983236+0530 kernel mcxalr{312} ** Denying execute for uid=502 path=/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura

[seshaTarget]

Sandbox: logd_helper(937) deny(1) file-read-data /private/var/folders/fh/6hrhj3n91w3by20bgn10x3k80000gp/T/.io.rancherdesktop.app.jGNWJf

[seshaTarget]

Prompting policy for hardened runtime; service: kTCCServiceAppleEvents requires entitlement com.apple.security.automation.apple-events but it is missing for accessing={TCCDProcess: identifier=io.rancherdesktop.app, pid=31343, auid=502, euid=502, binary_path=/Applications/Rancher Desktop.app/Contents/MacOS/Rancher Desktop}, requesting={TCCDProcess: identifier=com.apple.appleeventsd, pid=510, auid=55, euid=55, binary_path=/System/Library/CoreServices/appleeventsd},

[seshaTarget]

hi @mook-as The above are the only logs i could capture which i believe are not spammy as well

[mook-as]

Great! Those first set of lines lead me to: https://github.com/microsoft/vscode/issues/123276#issuecomment-880971648

Also, that says man mcxalr is a thing (but Apple no longer appears to publish manpages on the web, so I can't link to it). So is mcxquery; that might include relevant information (but didn't in that bug).

That bug also indicates that it may be possible to get around this if we don't spawn things in parallel. That would make runs slightly slower, but that's probably still better than not working at all.

Found another issue that might have a clue on how we can get better logs out of it: https://github.com/pyenv/pyenv/issues/2588#issuecomment-1711952834

[seshaTarget]

@mook-as if you need any further logs, let me know . I will be back again in another 9hrs of time

[jandubois]

@seshaTarget Are you able to install CI builds from Github on your machine for testing. E.g. from https://github.com/rancher-sandbox/rancher-desktop/actions/runs/6265859917 ?

You need to be logged into Github to be able to download the assets, and they are not signed/notarized, so I don't know if your MDM profile will block them you will need to remove the quarantine bit manually that the browser attaches to all downloads:

xattr -dr com.apple.quarantine /Applications/Rancher\ Desktop.app/

This build should still exhibit the problem, so it would just be to establish a baseline.

[jandubois]

That bug also indicates that it may be possible to get around this if we don't spawn things in parallel.

@mook-as I suspect that mcxalr will cache the results, so it may be enough to just run e.g. limactl.ventura --version once synchronously, and then keeping to run the other calls in parallel. I hope that @seshaTarget can run builds from CI, then they could verify if this change would make it work.

[jandubois]

I suspect that mcxalr will cache the results

I discussed this with @mook-as and he pointed out that we already run limactl start synchronously, and then later asynchronous limactl shell commands are failing. So "priming the pump" by running limactl --version upfront is unlikely to make a difference.

[mook-as]

Noting down things that we've tried so far so we don't repeat them in the future:

• Running limactl serially: doesn't seem to help much.
• Copying limactl to a temporary file (in the same place): that's probably bad for modifying the app package.
• Copying limactl to a temporary directory: needs the guest agent binary next to it; still seems to break after it runs a few times. I assume mcx is noticing the pattern.

[jandubois]

We made a lot of changes in signing/notarizing bits and defining entitlements in the 1.12 release. As we are unable to reproduce the problem ourselves, could people please report back if they still have issue with the latest release, or if this issue is resolved? Thank you!

[cafe24-jhjeong02]

@jandubois I got same issue with 1.12.2. (image)

[jandubois]

@cafe24-jhjeong02 This does not look like the same issue at all. Note how your screenshot says KILLED instead of EACCES?

Can you upload a zip file of all the logs?

[jaskirat8]

Apologies, I missed the message. I installed the latest version and got this

[jawadshaikst]

I had this (or similar) issue with Rancher Desktop 1.12.2 & 1.12.3 on M1 Max/Silicon where it would fail on installing CA certs on startup. What got me around was to reset Kubernetes from the Troubleshooting tab and I ended up having to reinstall my containers to get Rancher Desktop working again.

[erppaul]

Hi all, we have this issue with Rancher Desktop Version: 1.13.1 on M2 where it would fail on installing CA certs on startup too. Reset Kubernetes or Factory Reset did not help. The devices are managed with Jamf Pro and the employees do not have admin rights on the devices. In Jamf Pro, we have set up a policy that restricts the opening of apps. We have defined a whitelist there. Which folders need to be enabled for Rancher Desktop?

[Nachox07]

Hi all, this might help people experiencing this issue when Rancher tries to update certificates.

Error:
/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura shell --workdir=. 0 sudo update-ca-certificates

Details:
'WARNING: Skipping duplicate certificate ...
  code: 1,
  [Symbol(child-process.command)]: '/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura shell --workdir=. 0 sudo update-ca-certificates'
}

This is how I solved it after debugging for some time, as I wanted to avoid reinstalling everything in the cluster at any cost due to time.

update-ca-certificates simply runs /usr/bin/c_rehash /etc/ssl/certs

1. Attach to the Rancher shell in a terminal rdctl shell
2. Run the following command to update certs manually and avoid issues sudo /usr/bin/c_rehash -old /etc/ssl/certs

After restarting Rancher, my cluster started to work again. I could see that c_rehash has some problems with mkcerts while trying to update them, but I was lucky, and using the old flag could deal with them. The way I knew mkcerts where the ones giving problems was by isolating them, you can run this script to check it:

#!/bin/sh

# Remember to run mkdir first
BACKUP_DIR="/etc/ssl/certs/backup_all"
CERT_DIR="/etc/ssl/certs"

rehash_and_check() {
    sudo /usr/bin/c_rehash "$CERT_DIR"
    if [ $? -ne 0 ]; then
        echo "Issue detected after moving $1 back. Moving it back to backup."
        sudo mv "$CERT_DIR/$1" "$BACKUP_DIR/"
    fi
}

for cert in $(ls "$BACKUP_DIR"); do
    echo "Moving $cert back to certs directory"
    sudo mv "$BACKUP_DIR/$cert" "$CERT_DIR/"
    rehash_and_check "$cert"
done

sudo /usr/bin/c_rehash "$CERT_DIR"