crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 239 forks source link

Can't get CRC running on M1 MAC #3423

Closed Nanyte25 closed 1 year ago

Nanyte25 commented 1 year ago

General information

CRC version

# Put `crc version
CRC version: 2.10.2+065f0741
OpenShift version: 4.2.0
Podman version: 4.2.0`

CRC status

# Put `crc status --log-level debug` output here

crc status --log-level debug
DEBU CRC version: 2.10.2+065f0741
DEBU OpenShift version: 4.2.0
DEBU Podman version: 4.2.0
DEBU Running 'crc status'
CRC VM:          Running
Podman:          4.2.0
RAM Usage:       154MB of 15.57GB
Disk Usage:      1.796GB of 32.74GB (Inside the CRC VM)
Cache Usage:     70.94GB
Cache Directory: /Users/vahalla3/.crc/cache

CRC config

# Put `crc config view` output here

crc config view
- consent-telemetry                     : no
- memory                                : 15258
- preset                                : podman

Host Operating System


# put the output of `sw_vers` in case of Mac

sw_vers
ProductName:        macOS
ProductVersion:     13.0
BuildVersion:       22A380

Steps to reproduce

crc start INFO Checking if running as non-root INFO Checking if crc-admin-helper executable is cached INFO Checking for obsolete admin-helper executable INFO Checking if running on a supported CPU architecture INFO Checking minimum RAM requirements INFO Checking if crc executable symlink exists INFO Checking if running emulated on a M1 CPU INFO Checking if vfkit is installed INFO Checking if old launchd config for tray and/or daemon exists INFO Checking if crc daemon plist file is present and loaded INFO Loading bundle: crc_podman_vfkit_4.2.0_arm64... INFO Creating CRC VM for Podman 4.2.0... INFO Generating new SSH key pair... INFO CRC instance is running with IP 127.0.0.1 INFO CRC VM is running INFO Updating authorized keys... INFO Adding new bearer token for cockpit webconsole podman runtime is now running.

Use the 'podman' command line interface: $ eval $(crc podman-env) $ podman COMMAND

Expected

Actual

Logs

Before gather the logs try following if that fix your issue


crc delete -f
Deleted the instance
➜  Downloads crc cleanup
INFO Unloading and removing the daemon plist file
INFO Removing /etc/resolver/testing file
INFO Stopping CRC vfkit process
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
Cleanup finished
➜  Downloads crc setup
INFO Using bundle path /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64.crcbundle
INFO Checking if running as non-root
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Creating symlink for crc executable
INFO Checking if running emulated on a M1 CPU
INFO Checking if vfkit is installed
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64.crcbundle exists
INFO Checking if old launchd config for tray and/or daemon exists
INFO Checking if crc daemon plist file is present and loaded
INFO Adding crc daemon plist file and loading it
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
➜  Downloads  crc start --log-level debug
DEBU CRC version: 2.10.2+065f0741
DEBU OpenShift version: 4.2.0
DEBU Podman version: 4.2.0
DEBU Running 'crc start'
DEBU Total memory of system is 34359738368 bytes
DEBU No new version available. The latest version is 2.10.2
DEBU Checking file: /Users/vahalla3/.crc/machines/crc/.crc-exist
INFO Checking if running as non-root
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
DEBU Checking if an older admin-helper executable is installed
DEBU No older admin-helper executable found
INFO Checking if running on a supported CPU architecture
DEBU GOARCH is arm64 GOOS is darwin
INFO Checking minimum RAM requirements
DEBU Total memory of system is 34359738368 bytes
INFO Checking if crc executable symlink exists
INFO Checking if running emulated on a M1 CPU
INFO Checking if vfkit is installed
INFO Checking if old launchd config for tray and/or daemon exists
INFO Checking if crc daemon plist file is present and loaded
DEBU Running 'bash -c launchctl list | grep com.redhat.crc.daemon | awk '{print $1}''
DEBU Checking file: /Users/vahalla3/.crc/machines/crc/.crc-exist
INFO Loading bundle: crc_podman_vfkit_4.2.0_arm64...
INFO Creating CRC VM for Podman 4.2.0...
DEBU Running pre-create checks...
DEBU Creating machine...
DEBU /Users/vahalla3/.crc/machines/crc/crc.img is already 33285996544 bytes
INFO Generating new SSH key pair...
DEBU Created /Users/vahalla3/.crc/machines/crc/.crc-exist
DEBU Machine successfully created
DEBU Checking file: /Users/vahalla3/.crc/machines/crc/.crc-exist
DEBU Updating CRC VM configuration
DEBU Running 'sw_vers -productVersion'
DEBU Running /Applications/Red Hat OpenShift Local.app/Contents/Resources/vfkit --cpus 2 --memory 15258 --kernel /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/vmlinuz-5.18.18-200.fc36.aarch64 --initrd /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/initramfs-5.18.18-200.fc36.aarch64.img --kernel-cmdline console=hvc0 BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-8edf9753152209bad4dde8faf34b8484000abe010476136b4dba35f7ad83e9e5/vmlinuz-5.18.18-200.fc36.aarch64 mitigations=auto,nosmt ignition.platform.id=qemu ostree=/ostree/boot.1/fedora-coreos/8edf9753152209bad4dde8faf34b8484000abe010476136b4dba35f7ad83e9e5/0 root=UUID=7a8bcc2a-92ab-48ab-b6ec-9ac0cc57b00f rw rootflags=prjquota boot=UUID=bc8b9431-dee2-4950-b029-b9acb3911efd --device virtio-serial,logFilePath=/Users/vahalla3/.crc/machines/crc/vfkit.log --device virtio-rng --device virtio-blk,path=/Users/vahalla3/.crc/machines/crc/crc.img --device virtio-vsock,port=1024,socketURL=/Users/vahalla3/.crc/tap.sock
DEBU time="2022-11-18T12:22:53Z" level=info msg="&{2 15258 /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/vmlinuz-5.18.18-200.fc36.aarch64 console=hvc0 BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-8edf9753152209bad4dde8faf34b8484000abe010476136b4dba35f7ad83e9e5/vmlinuz-5.18.18-200.fc36.aarch64 mitigations=auto,nosmt ignition.platform.id=qemu ostree=/ostree/boot.1/fedora-coreos/8edf9753152209bad4dde8faf34b8484000abe010476136b4dba35f7ad83e9e5/0 root=UUID=7a8bcc2a-92ab-48ab-b6ec-9ac0cc57b00f rw rootflags=prjquota boot=UUID=bc8b9431-dee2-4950-b029-b9acb3911efd /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/initramfs-5.18.18-200.fc36.aarch64.img [virtio-serial,logFilePath=/Users/vahalla3/.crc/machines/crc/vfkit.log virtio-rng virtio-blk,path=/Users/vahalla3/.crc/machines/crc/crc.img virtio-vsock,port=1024,socketURL=/Users/vahalla3/.crc/tap.sock]}"
DEBU time="2022-11-18T12:22:53Z" level=info msg="boot parameters:"
DEBU time="2022-11-18T12:22:53Z" level=info msg="\tkernel: /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/vmlinuz-5.18.18-200.fc36.aarch64"
DEBU time="2022-11-18T12:22:53Z" level=info msg="\tkernel command line:console=hvc0 BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-8edf9753152209bad4dde8faf34b8484000abe010476136b4dba35f7ad83e9e5/vmlinuz-5.18.18-200.fc36.aarch64 mitigations=auto,nosmt ignition.platform.id=qemu ostree=/ostree/boot.1/fedora-coreos/8edf9753152209bad4dde8faf34b8484000abe010476136b4dba35f7ad83e9e5/0 root=UUID=7a8bcc2a-92ab-48ab-b6ec-9ac0cc57b00f rw rootflags=prjquota boot=UUID=bc8b9431-dee2-4950-b029-b9acb3911efd"
DEBU time="2022-11-18T12:22:53Z" level=info msg="\tinitrd: /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/initramfs-5.18.18-200.fc36.aarch64.img"
DEBU time="2022-11-18T12:22:53Z" level=info
DEBU time="2022-11-18T12:22:53Z" level=info msg="virtual machine parameters:"
DEBU time="2022-11-18T12:22:53Z" level=info msg="\tvCPUs: 2"
DEBU time="2022-11-18T12:22:53Z" level=info msg="\tmemory: 15258 MiB"
DEBU time="2022-11-18T12:22:53Z" level=info
DEBU time="2022-11-18T12:22:53Z" level=info msg="Adding virtio-serial device (logFile: /Users/vahalla3/.crc/machines/crc/vfkit.log)"
DEBU time="2022-11-18T12:22:53Z" level=info msg="Adding virtio-rng device"
DEBU time="2022-11-18T12:22:53Z" level=info msg="Adding virtio-blk device (imagePath: /Users/vahalla3/.crc/machines/crc/crc.img)"
DEBU time="2022-11-18T12:22:53Z" level=info msg="Adding virtio-vsock device"
DEBU time="2022-11-18T12:22:53Z" level=info msg="virtual machine is running"
DEBU time="2022-11-18T12:22:53Z" level=info msg="Exposing vsock port 1024 on /Users/vahalla3/.crc/tap.sock"
DEBU time="2022-11-18T12:22:53Z" level=info msg="waiting for VM to stop"
DEBU Waiting for machine to be running, this may take a few minutes...
DEBU retry loop: attempt 0
DEBU Machine is up and running!
INFO CRC instance is running with IP 127.0.0.1
DEBU Waiting until ssh is available
DEBU retry loop: attempt 0
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/vahalla3/.crc/machines/crc/id_ecdsa /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:53450->127.0.0.1:2222: read: connection reset by peer, output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:53450->127.0.0.1:2222: read: connection reset by peer
 - sleeping 1s
DEBU retry loop: attempt 1
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/vahalla3/.crc/machines/crc/id_ecdsa /Users/vahalla3/.crc/cache/crc_podman_vfkit_4.2.0_arm64/id_ecdsa_crc]
DEBU DialContext: tcp unix:/Users/vahalla3/.crc/tap.sock
DEBU SSH command results: err: <nil>, output:
INFO CRC VM is running
DEBU Running SSH command: cat /home/core/.ssh/authorized_keys
DEBU SSH command results: err: Process exited with status 1, output:
INFO Updating authorized keys...
DEBU Creating /home/core/.ssh/authorized_keys with permissions 0644 in the CRC VM
DEBU Running SSH command: <hidden>
DEBU SSH command succeeded
DEBU Running SSH command: rm /home/core/.ssh/authorized_keys.d/ignition
DEBU SSH command results: err: <nil>, output:
DEBU Running SSH command: realpath /dev/disk/by-label/root
DEBU SSH command results: err: <nil>, output: /dev/vda4
DEBU Using root access: Growing /dev/vda4 partition
DEBU Running SSH command: sudo /usr/bin/growpart /dev/vda 4
DEBU SSH command results: err: Process exited with status 1, output: NOCHANGE: partition 4 is size 63961055. it cannot be grown
DEBU No free space after /dev/vda4, nothing to do
DEBU Configuring shared directories
DEBU Running 'sw_vers -productVersion'
DEBU Using root access: make root Podman socket accessible
DEBU Running SSH command: sudo chmod 777 /run/podman/ /run/podman/podman.sock
DEBU SSH command results: err: <nil>, output:
DEBU Running '/Applications/Red Hat OpenShift Local.app/Contents/Resources/crc-admin-helper-darwin rm podman.crc.testing'
DEBU Running '/Applications/Red Hat OpenShift Local.app/Contents/Resources/crc-admin-helper-darwin add 127.0.0.1 podman.crc.testing'
INFO Adding new bearer token for cockpit webconsole
DEBU Creating /home/core/cockpit-bearer-token with permissions 0600 in the CRC VM
DEBU Running SSH command: <hidden>
DEBU SSH command succeeded
DEBU Checking file: /Users/vahalla3/.crc/machines/crc/.crc-exist
DEBU Running '/Users/vahalla3/.crc/bin/oc/podman system connection add --identity /Users/vahalla3/.crc/machines/crc/id_ecdsa crc ssh://core@127.0.0.1:2222/run/user/1000/podman/podman.sock'
DEBU Running '/Users/vahalla3/.crc/bin/oc/podman system connection default crc'
DEBU Running '/Users/vahalla3/.crc/bin/oc/podman system connection add --identity /Users/vahalla3/.crc/machines/crc/id_ecdsa crc-root ssh://core@127.0.0.1:2222/run/podman/podman.sock'
podman runtime is now running.

Use the 'podman' command line interface:
  $ eval $(crc podman-env)
  $ podman COMMAND

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

cfergeau commented 1 year ago

From the log, the podman runtime seems to have started properly. What are you trying to do which is failing?

gbraad commented 1 year ago

As indicated in the startup log:

podman runtime is now running.

Use the 'podman' command line interface:
  $ eval $(crc podman-env)
  $ podman COMMAND

Have you been able to do:

$ eval $(crc podman-env)
$ podman info

Please post the results here

Nanyte25 commented 1 year ago

@gbraad here you go podman info

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@127.0.0.1:2222/run/user/1000/podman/podman.sock) failed: ssh: handshake failed: read tcp 127.0.0.1:54852->127.0.0.1:2222: read: connection reset by peer
~ podman machine list
NAME                    VM TYPE     CREATED      LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default  qemu        2 hours ago  Currently running  1           2.147GB     107.4GB
➜  ~
~ podman info
host:
  arch: arm64
  buildahVersion: 1.27.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 84.56
    systemPercent: 11.23
    userPercent: 4.21
  cpus: 2
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: podman.crc.testing
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.18.18-200.fc36.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 15231533056
  memTotal: 15574286336
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.aarch64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 0m 12.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 32737570816
  graphRootUsed: 1796513792
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1660228991
  BuiltTime: Thu Aug 11 14:43:11 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/arm64
  Version: 4.2.0
 ~ crc console
Only supported with OpenShift bundles
gbraad commented 1 year ago

I am very confused now. ~Are you reporting for CRC or podman machine ?~ [Due to bad messaging from the client]

podman machine list
NAME                    VM TYPE     CREATED      LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default

Of course we will never show up in the podman machine list, as we use a slightly different stack (utilizing native virtualization instead of Qemu).

This might be causes by the message that podman gives

Cannot connect to Podman. Please verify your connection to the Linux system using podman system connection list, or try podman machine init and podman machine start to manage a new Linux VM 

Please ignore these steps. /cc: @vrothberg @ashley-cui

However, from podman info I can see a connection was made as it reports:

  hostname: podman.crc.testing

so this seems to work. According to the output, podman is operational.

What happens when you do:

$ podman run hello-world

As this should give a banner output, similar to:

Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob a538fa3b889c done
Copying config 0411f5c56d done
Writing manifest to image destination
Storing signatures
!... Hello Podman World ...!

         .--"--.
       / -     - \
      / (O)   (O) \
   ~~~| -=(,Y,)=- |
    .---. /`  \   |~~
 ~/  o  o \~~~~.----. ~~
  | =(X)= |~  / (O (O) \
   ~~~~~~~  ~| =(Y_)=-  |
  ~~~~    ~~~|   U      |~~

Project:   https://github.com/containers/podman
Website:   https://podman.io
Documents: https://docs.podman.io
Twitter:   @Podman_io
vrothberg commented 1 year ago

Can you file an issue against podman?

cgruver commented 1 year ago

@Nanyte25 I noticed that you are giving the CRC VM 15GB of RAM. Make sure you are not hitting: https://github.com/crc-org/crc/issues/3349

There is a Podman specific runtime for MacOS that does not use vfkit. It uses Qemu and is very stable on apple silicon.

brew install podman