Error: error inspecting object: no such container "minikube" / minikube with podman and cri-o on MacBook Pro with Apple Silicon M1 Pro

noracenofun commented 2 years ago

What happened?

How can I fix this issue?

I can't successfully start minikube with podman:

Error: error inspecting object: no such container "minikube"
❌  Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f  minikube: exit status 125

minikube with podman and cri-o on MacBook Pro with Apple Silicon M1 Pro

$ arch
arm64

$ sysctl -n machdep.cpu.brand_string
Apple M1 Pro

$ sw_vers
ProductName:    macOS
ProductVersion: 12.1
BuildVersion:   21C52

$ ls -1 ~/.local/share/containers/podman/machine/qemu/
fedora-coreos-35.20220116.2.0-qemu.aarch64.qcow2.xz
podman-machine-default_fedora-coreos-35.20220116.2.0-qemu.aarch64.qcow2
podman-machine-default_ovmf_vars.fd

$ podman machine ls
NAME                    VM TYPE     CREATED         LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default  qemu        13 minutes ago  Currently running  2           2.147GB     21.47GB

$ podman system connection ls
Name                          Identity                                     URI
podman-machine-default        /Users/darwin/.ssh/podman-machine-default  ssh://core@localhost:64866/run/user/1000/podman/podman.sock
podman-machine-default-root*  /Users/darwin/.ssh/podman-machine-default  ssh://root@localhost:64866/run/podman/podman.sock

$ podman version
Client:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Wed Dec  8 19:41:11 2021
OS/Arch:      darwin/arm64

Server:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8
Built:        Wed Dec  8 22:48:10 2021
OS/Arch:      linux/arm64

$ minikube start --driver=podman --container-runtime=cri-o --alsologtostderr -v=7
I0122 17:56:18.718486   63869 out.go:297] Setting OutFile to fd 1 ...
I0122 17:56:18.718820   63869 out.go:349] isatty.IsTerminal(1) = true
I0122 17:56:18.718825   63869 out.go:310] Setting ErrFile to fd 2...
I0122 17:56:18.718829   63869 out.go:349] isatty.IsTerminal(2) = true
I0122 17:56:18.718911   63869 root.go:315] Updating PATH: /Users/darwin/.minikube/bin
W0122 17:56:18.719015   63869 root.go:293] Error reading config file at /Users/darwin/.minikube/config/config.json: open /Users/darwin/.minikube/config/config.json: no such file or directory
I0122 17:56:18.719183   63869 out.go:304] Setting JSON to false
I0122 17:56:18.761270   63869 start.go:112] hostinfo: {"hostname":"6092b957-ea3c-491e-842c-adf1b4811540.fritz.box","uptime":508628,"bootTime":1642361950,"procs":669,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"12.1","kernelVersion":"21.2.0","kernelArch":"arm64","virtualizationSystem":"","virtualizationRole":"","hostId":"838682be-e3ad-5bb1-af99-8c92df6c3bb4"}
W0122 17:56:18.761463   63869 start.go:120] gopshost.Virtualization returned error: not implemented yet
I0122 17:56:18.781025   63869 out.go:176] 😄  minikube v1.25.1 on Darwin 12.1 (arm64)
😄  minikube v1.25.1 on Darwin 12.1 (arm64)
I0122 17:56:18.781349   63869 notify.go:174] Checking for updates...
I0122 17:56:18.782325   63869 driver.go:344] Setting default libvirt URI to qemu:///system
I0122 17:56:18.904051   63869 podman.go:121] podman version: 3.4.4
I0122 17:56:18.921685   63869 out.go:176] ✨  Using the podman (experimental) driver based on user configuration
✨  Using the podman (experimental) driver based on user configuration
I0122 17:56:18.921714   63869 start.go:280] selected driver: podman
I0122 17:56:18.921719   63869 start.go:795] validating driver "podman" against <nil>
I0122 17:56:18.921736   63869 start.go:806] status for podman: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc:}
I0122 17:56:18.921769   63869 start.go:1498] auto setting extra-config to "kubelet.housekeeping-interval=5m".
I0122 17:56:18.922322   63869 cli_runner.go:133] Run: podman system info --format json
I0122 17:56:19.015223   63869 info.go:285] podman info: {Host:{BuildahVersion:1.23.1 CgroupVersion:v2 Conmon:{Package:conmon-2.0.30-2.fc35.aarch64 Path:/usr/bin/conmon Version:conmon version 2.0.30, commit: } Distribution:{Distribution:fedora Version:35} MemFree:1203466240 MemTotal:2048053248 OCIRuntime:{Name:crun Package:crun-1.4-1.fc35.aarch64 Path:/usr/bin/crun Version:crun version 1.4
commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:arm64 Cpus:2 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.15.12-200.fc35.aarch64 Os:linux Rootless:false Uptime:2m 24.18s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I0122 17:56:19.015376   63869 start_flags.go:286] no existing cluster config was found, will generate one from the flags
I0122 17:56:19.015489   63869 start_flags.go:367] Using suggested 1953MB memory alloc based on sys=16384MB, container=1953MB
I0122 17:56:19.015575   63869 start_flags.go:796] Wait components to verify : map[apiserver:true system_pods:true]
I0122 17:56:19.015601   63869 cni.go:93] Creating CNI manager for ""
I0122 17:56:19.015615   63869 cni.go:160] "podman" driver + crio runtime found, recommending kindnet
I0122 17:56:19.015622   63869 start_flags.go:295] Found "CNI" CNI - setting NetworkPlugin=cni
I0122 17:56:19.015628   63869 start_flags.go:300] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b Memory:1953 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.23.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[{Component:kubelet Key:housekeeping-interval Value:5m}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror:}
I0122 17:56:19.017606   63869 out.go:176] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I0122 17:56:19.017649   63869 cache.go:120] Beginning downloading kic base image for podman with crio
I0122 17:56:19.070639   63869 out.go:176] 🚜  Pulling base image ...
🚜  Pulling base image ...
I0122 17:56:19.070749   63869 preload.go:132] Checking if preload exists for k8s version v1.23.1 and runtime crio
I0122 17:56:19.070859   63869 preload.go:148] Found local preload: /Users/darwin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v16-v1.23.1-cri-o-overlay-arm64.tar.lz4
I0122 17:56:19.070865   63869 cache.go:57] Caching tarball of preloaded images
I0122 17:56:19.071024   63869 preload.go:174] Found /Users/darwin/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v16-v1.23.1-cri-o-overlay-arm64.tar.lz4 in cache, skipping download
I0122 17:56:19.071043   63869 cache.go:60] Finished verifying existence of preloaded tar for  v1.23.1 on crio
I0122 17:56:19.071100   63869 cache.go:148] Downloading gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b to local cache
I0122 17:56:19.071203   63869 image.go:59] Checking for gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b in local cache directory
I0122 17:56:19.071215   63869 image.go:62] Found gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b in local cache directory, skipping pull
I0122 17:56:19.071219   63869 image.go:103] gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b exists in cache, skipping pull
I0122 17:56:19.071223   63869 cache.go:151] successfully saved gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b as a tarball
I0122 17:56:19.071386   63869 profile.go:147] Saving config to /Users/darwin/.minikube/profiles/minikube/config.json ...
I0122 17:56:19.071441   63869 lock.go:35] WriteFile acquiring /Users/darwin/.minikube/profiles/minikube/config.json: {Name:mk36a382500a1742a8292ec1ee5e5111ce2c9cde Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
E0122 17:56:19.071794   63869 cache.go:203] Error downloading kic artifacts:  not yet implemented, see issue #8426
I0122 17:56:19.071802   63869 cache.go:208] Successfully downloaded all kic artifacts
I0122 17:56:19.072079   63869 start.go:313] acquiring machines lock for minikube: {Name:mk231f76dc1f823d3efb85f3f6fd74551135e722 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0122 17:56:19.072139   63869 start.go:317] acquired machines lock for "minikube" in 46.125µs
I0122 17:56:19.072163   63869 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b Memory:1953 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.23.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[{Component:kubelet Key:housekeeping-interval Value:5m}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.23.1 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror:} &{Name: IP: Port:8443 KubernetesVersion:v1.23.1 ControlPlane:true Worker:true}
I0122 17:56:19.072233   63869 start.go:126] createHost starting for "" (driver="podman")
I0122 17:56:19.108521   63869 out.go:203] 🔥  Creating podman container (CPUs=2, Memory=1953MB) ...
🔥  Creating podman container (CPUs=2, Memory=1953MB) ...| I0122 17:56:19.108783   63869 start.go:160] libmachine.API.Create for "minikube" (driver="podman")
I0122 17:56:19.108824   63869 client.go:168] LocalClient.Create starting
I0122 17:56:19.108963   63869 main.go:130] libmachine: Creating CA: /Users/darwin/.minikube/certs/ca.pem
- I0122 17:56:19.310065   63869 main.go:130] libmachine: Creating client certificate: /Users/darwin/.minikube/certs/cert.pem
I0122 17:56:19.410489   63869 cli_runner.go:133] Run: podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
| W0122 17:56:19.517710   63869 cli_runner.go:180] podman network inspect minikube --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}" returned with exit code 125
I0122 17:56:19.517945   63869 network_create.go:254] running [podman network inspect minikube] to gather additional debugging logs...
I0122 17:56:19.517977   63869 cli_runner.go:133] Run: podman network inspect minikube
W0122 17:56:19.598611   63869 cli_runner.go:180] podman network inspect minikube returned with exit code 125
I0122 17:56:19.598746   63869 network_create.go:257] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]

stderr:
Error: error inspecting object: no such network "minikube"
I0122 17:56:19.598765   63869 network_create.go:259] output of [podman network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: error inspecting object: no such network "minikube"

** /stderr **
I0122 17:56:19.598904   63869 cli_runner.go:133] Run: podman network inspect podman --format "{{range .plugins}}{{if eq .type "bridge"}}{{(index (index .ipam.ranges 0) 0).subnet}},{{(index (index .ipam.ranges 0) 0).gateway}}{{end}}{{end}}"
/ I0122 17:56:19.678054   63869 network.go:288] reserving subnet 192.168.49.0 for 1m0s: &{mu:{state:0 sema:0} read:{v:{m:map[] amended:true}} dirty:map[192.168.49.0:0x1400061c3e8] misses:0}
I0122 17:56:19.678113   63869 network.go:235] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
I0122 17:56:19.678135   63869 network_create.go:106] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
I0122 17:56:19.678246   63869 cli_runner.go:133] Run: podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 --label=created_by.minikube.sigs.k8s.io=true minikube
- I0122 17:56:19.759667   63869 network_create.go:90] podman network minikube 192.168.49.0/24 created
I0122 17:56:19.759734   63869 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
I0122 17:56:19.759895   63869 cli_runner.go:133] Run: podman ps -a --format {{.Names}}
\ I0122 17:56:19.840052   63869 cli_runner.go:133] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
| I0122 17:56:19.925718   63869 oci.go:102] Successfully created a podman volume minikube
I0122 17:56:19.925849   63869 cli_runner.go:133] Run: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.29 -d /var/lib
| I0122 18:02:19.111870   63869 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0122 18:02:19.112116   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
- I0122 18:02:19.263388   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0122 18:02:19.340326   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I0122 18:02:19.340433   63869 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
/ I0122 18:02:19.618108   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
\ I0122 18:02:19.774387   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
| W0122 18:02:19.855408   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I0122 18:02:19.855511   63869 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
/ I0122 18:02:20.397049   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
- I0122 18:02:20.532655   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
\ W0122 18:02:20.613473   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
W0122 18:02:20.613651   63869 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W0122 18:02:20.613679   63869 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:20.613702   63869 start.go:129] duration metric: createHost completed in 6m1.541549917s
I0122 18:02:20.613708   63869 start.go:80] releasing machines lock for "minikube", held for 6m1.541661458s
W0122 18:02:20.613722   63869 start.go:566] error starting host: creating host: create host timed out in 360.000000 seconds
I0122 18:02:20.615065   63869 cli_runner.go:133] Run: podman container inspect minikube --format={{.State.Status}}
| W0122 18:02:20.702880   63869 cli_runner.go:180] podman container inspect minikube --format={{.State.Status}} returned with exit code 125
I0122 18:02:20.702995   63869 delete.go:82] Unable to get host status for minikube, assuming it has already been deleted: state: unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
W0122 18:02:20.703185   63869 out.go:241] 🤦  StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds

🤦  StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
I0122 18:02:20.703200   63869 start.go:581] Will try again in 5 seconds ...
I0122 18:02:25.704378   63869 start.go:313] acquiring machines lock for minikube: {Name:mk231f76dc1f823d3efb85f3f6fd74551135e722 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0122 18:02:25.704629   63869 start.go:317] acquired machines lock for "minikube" in 204.625µs
I0122 18:02:25.704731   63869 start.go:93] Skipping create...Using existing machine configuration
I0122 18:02:25.704746   63869 fix.go:55] fixHost starting:
I0122 18:02:25.706490   63869 cli_runner.go:133] Run: podman container inspect minikube --format={{.State.Status}}
W0122 18:02:25.789554   63869 cli_runner.go:180] podman container inspect minikube --format={{.State.Status}} returned with exit code 125
I0122 18:02:25.789675   63869 fix.go:108] recreateIfNeeded on minikube: state= err=unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:25.789704   63869 fix.go:113] machineExists: true. err=unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
W0122 18:02:25.789720   63869 fix.go:134] unexpected machine state, will restart: unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:25.808226   63869 out.go:176] 🔄  Restarting existing podman container for "minikube" ...
🔄  Restarting existing podman container for "minikube" ...
I0122 18:02:25.808367   63869 cli_runner.go:133] Run: podman start minikube
W0122 18:02:25.891248   63869 cli_runner.go:180] podman start minikube returned with exit code 125
I0122 18:02:25.891461   63869 cli_runner.go:133] Run: podman inspect minikube
I0122 18:02:25.971070   63869 errors.go:84] Postmortem inspect ("podman inspect minikube"): -- stdout --
[
    {
        "Name": "minikube",
        "Driver": "local",
        "Mountpoint": "/var/lib/containers/storage/volumes/minikube/_data",
        "CreatedAt": "2022-01-22T16:56:19.905243873Z",
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true",
            "name.minikube.sigs.k8s.io": "minikube"
        },
        "Scope": "local",
        "Options": {}
    }
]

-- /stdout --
I0122 18:02:25.972008   63869 cli_runner.go:133] Run: podman logs --timestamps minikube
W0122 18:02:26.048269   63869 cli_runner.go:180] podman logs --timestamps minikube returned with exit code 125
W0122 18:02:26.048366   63869 errors.go:89] Failed to get postmortem logs. podman logs --timestamps minikube :podman logs --timestamps minikube: exit status 125
stdout:

stderr:
Error: channel "123" found, 0-3 supported: lost synchronization with multiplexed stream
I0122 18:02:26.048492   63869 cli_runner.go:133] Run: podman system info --format json
I0122 18:02:26.145833   63869 info.go:285] podman info: {Host:{BuildahVersion:1.23.1 CgroupVersion:v2 Conmon:{Package:conmon-2.0.30-2.fc35.aarch64 Path:/usr/bin/conmon Version:conmon version 2.0.30, commit: } Distribution:{Distribution:fedora Version:35} MemFree:951472128 MemTotal:2048053248 OCIRuntime:{Name:crun Package:crun-1.4-1.fc35.aarch64 Path:/usr/bin/crun Version:crun version 1.4
commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:arm64 Cpus:2 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.15.12-200.fc35.aarch64 Os:linux Rootless:false Uptime:8m 31.39s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I0122 18:02:26.145941   63869 errors.go:106] postmortem podman info: {Host:{BuildahVersion:1.23.1 CgroupVersion:v2 Conmon:{Package:conmon-2.0.30-2.fc35.aarch64 Path:/usr/bin/conmon Version:conmon version 2.0.30, commit: } Distribution:{Distribution:fedora Version:35} MemFree:951472128 MemTotal:2048053248 OCIRuntime:{Name:crun Package:crun-1.4-1.fc35.aarch64 Path:/usr/bin/crun Version:crun version 1.4
commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:arm64 Cpus:2 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.15.12-200.fc35.aarch64 Os:linux Rootless:false Uptime:8m 31.39s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I0122 18:02:26.146042   63869 network_create.go:254] running [podman network inspect minikube] to gather additional debugging logs...
I0122 18:02:26.146059   63869 cli_runner.go:133] Run: podman network inspect minikube
I0122 18:02:26.224488   63869 network_create.go:259] output of [podman network inspect minikube]: -- stdout --
[
    {
        "args": {
            "podman_labels": {
                "created_by.minikube.sigs.k8s.io": "true"
            }
        },
        "cniVersion": "0.4.0",
        "name": "minikube",
        "plugins": [
            {
                "bridge": "cni-podman1",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "192.168.49.1",
                                "subnet": "192.168.49.0/24"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "backend": "",
                "type": "firewall"
            },
            {
                "type": "tuning"
            },
            {
                "capabilities": {
                    "aliases": true
                },
                "domainName": "dns.podman",
                "type": "dnsname"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "podman-machine"
            }
        ]
    }
]

-- /stdout --
I0122 18:02:26.224667   63869 cli_runner.go:133] Run: podman system info --format json
I0122 18:02:26.319902   63869 info.go:285] podman info: {Host:{BuildahVersion:1.23.1 CgroupVersion:v2 Conmon:{Package:conmon-2.0.30-2.fc35.aarch64 Path:/usr/bin/conmon Version:conmon version 2.0.30, commit: } Distribution:{Distribution:fedora Version:35} MemFree:951361536 MemTotal:2048053248 OCIRuntime:{Name:crun Package:crun-1.4-1.fc35.aarch64 Path:/usr/bin/crun Version:crun version 1.4
commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:0 SwapTotal:0 Arch:arm64 Cpus:2 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.15.12-200.fc35.aarch64 Os:linux Rootless:false Uptime:8m 31.56s} Registries:{Search:[docker.io]} Store:{ConfigFile:/etc/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/var/lib/containers/storage GraphStatus:{BackingFilesystem:xfs NativeOverlayDiff:false SupportsDType:true UsingMetacopy:true} ImageStore:{Number:0} RunRoot:/run/containers/storage VolumePath:/var/lib/containers/storage/volumes}}
I0122 18:02:26.320548   63869 cli_runner.go:133] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube
W0122 18:02:26.396184   63869 cli_runner.go:180] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125
I0122 18:02:26.396380   63869 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0122 18:02:26.396421   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
I0122 18:02:26.501461   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0122 18:02:26.579156   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I0122 18:02:26.579266   63869 retry.go:31] will retry after 234.428547ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:26.814549   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
I0122 18:02:26.957766   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0122 18:02:27.035377   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I0122 18:02:27.035584   63869 retry.go:31] will retry after 346.739061ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:27.384474   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
I0122 18:02:27.549168   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0122 18:02:27.626046   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
I0122 18:02:27.626179   63869 retry.go:31] will retry after 667.587979ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:28.295238   63869 cli_runner.go:133] Run: podman version --format {{.Version}}
I0122 18:02:28.405835   63869 cli_runner.go:133] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W0122 18:02:28.486199   63869 cli_runner.go:180] podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125
W0122 18:02:28.486290   63869 start.go:257] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W0122 18:02:28.486305   63869 start.go:239] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for "minikube": podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
I0122 18:02:28.486319   63869 fix.go:57] fixHost completed within 2.781578167s
I0122 18:02:28.486326   63869 start.go:80] releasing machines lock for "minikube", held for 2.781676042s
W0122 18:02:28.486494   63869 out.go:241] 😿  Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

😿  Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

I0122 18:02:28.522062   63869 out.go:176]

W0122 18:02:28.522209   63869 out.go:241] ❌  Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f  minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

❌  Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f  minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"

W0122 18:02:28.522219   63869 out.go:241]

W0122 18:02:28.522983   63869 out.go:241] 
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I0122 18:02:28.577864   63869 out.go:176]

$ tree .minikube/cache
.minikube/cache
├── kic
│   └── kicbase_v0.0.29@sha256_be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b.tar
└── preloaded-tarball
    ├── preloaded-images-k8s-v16-v1.23.1-cri-o-overlay-arm64.tar.lz4
    └── preloaded-images-k8s-v16-v1.23.1-cri-o-overlay-arm64.tar.lz4.checksum

2 directories, 3 files

$ podman network ls
NETWORK ID    NAME        VERSION     PLUGINS
2f259bab93aa  podman      0.4.0       bridge,podman-machine,portmap,firewall,tuning
5086431107ca  minikube    0.4.0       bridge,portmap,firewall,tuning,dnsname,podman-machine

$ podman volume ls
DRIVER      VOLUME NAME
local       minikube

$ minikube status
E0122 18:08:56.011640   65779 status.go:258] status error: host: state: unknown state "minikube": podman container inspect minikube --format={{.State.Status}}: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container "minikube"
E0122 18:08:56.011661   65779 status.go:261] The "minikube" host does not exist!
minikube
type: Control Plane
host: Nonexistent
kubelet: Nonexistent
apiserver: Nonexistent
kubeconfig: Nonexistent

$ minikube profile list
|----------|-----------|---------|----|------|---------|---------|-------|
| Profile  | VM Driver | Runtime | IP | Port | Version | Status  | Nodes |
|----------|-----------|---------|----|------|---------|---------|-------|
| minikube | podman    | crio    |    | 8443 | v1.23.1 | Unknown |     1 |
|----------|-----------|---------|----|------|---------|---------|-------|

Attach the log file

logs.txt minikube_logs_c4108912796292a7d53f0ef436f4d4f3d9325070_0.log

Operating System

macOS (Default)

Driver

Podman

acelot commented 2 years ago

I'm experiencing same problem

MouMouHsu commented 2 years ago

I'm experiencing same problem

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

acelot commented 2 years ago

/remove-lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

RA489 commented 2 years ago

/remove-lifecycle stale

MRIservice commented 2 years ago

I did not have this issue until I started building docker images and pushing them to docker hub. It appears that docker takes out VirtualBox.

reboot minikube delete minikube start

After losing docker everything works

noracenofun commented 1 year ago

I no longer have any issues with minikube v1.28.0 and podman v4.3.1 on macOS 13.1 (arm64). It works.

kubernetes / minikube