provider base image does not correspond to the minikube default image

kaiehrhardt commented 1 year ago

Describe the bug

To Reproduce

minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396

terraform {
  required_version = ">= 1.0"

  required_providers {
    minikube = {
      source  = "scott-the-programmer/minikube"
      version = ">= 0.3.4"
    }
  }
}

provider "minikube" {
  kubernetes_version = "v1.27.3"
}

resource "minikube_cluster" "docker" {
  driver       = "docker"
  cluster_name = "docker"
  cpus         = 4
  memory       = "8192mb"
  cni          = "bridge"
  addons = [
    "ingress",
    "default-storageclass",
    "storage-provisioner"
  ]
}

tf apply ->

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # minikube_cluster.docker will be created
  + resource "minikube_cluster" "docker" {
      + addons                     = [
          + "default-storageclass",
          + "ingress",
          + "storage-provisioner",
...
      + base_image                 = "gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631"
...

tf apply again ->

minikube_cluster.docker: Refreshing state... [id=docker]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # minikube_cluster.docker must be replaced
-/+ resource "minikube_cluster" "docker" {
      ~ apiserver_ips              = [] -> (known after apply)
      ~ apiserver_names            = [
          - "minikubeCA",
        ] -> (known after apply)
      ~ base_image                 = "docker.io/kicbase/stable:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631" -> "gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631" # forces replacement
      ~ client_certificate         = (sensitive value)
      ~ client_key                 = (sensitive value)
      ~ cluster_ca_certificate     = (sensitive value)
      ~ host                       = "https://192.168.49.2:8443" -> (known after apply)
      ~ hyperkit_vsock_ports       = [] -> (known after apply)
      ~ id                         = "docker" -> (known after apply)
      ~ insecure_registry          = [] -> (known after apply)
      ~ iso_url                    = [
          - "https://github.com/kubernetes/minikube/releases/download/v1.31.0/minikube-v1.31.0-amd64.iso",
        ] -> (known after apply)
      ~ nfs_share                  = [] -> (known after apply)
      ~ ports                      = [] -> (known after apply)
      ~ registry_mirror            = [] -> (known after apply)
        # (58 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Expected behavior

No changes compared to the first tf run.

scott-the-programmer commented 1 year ago

This.. is strange👀 - the default value is actually pulled from the cli defaults, so I'm a bit surprised that this is getting switched during runtime.

I guess the minikube team have migrated their images over to the github registery.

I'll work on a fix now 👍🏼

kaiehrhardt commented 1 year ago

Nvm, seems like a local problem.

Some minikube log:

I1005 23:51:00.499712  252097 cache.go:150] Downloading gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 to local cache
I1005 23:51:00.499797  252097 image.go:63] Checking for gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 in local cache directory
I1005 23:51:00.499809  252097 image.go:66] Found gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 in local cache directory, skipping pull
I1005 23:51:00.499814  252097 image.go:105] gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 exists in cache, skipping pull
I1005 23:51:00.499819  252097 cache.go:153] successfully saved gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 as a tarball
I1005 23:51:00.499822  252097 cache.go:163] Loading gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 from local cache
I1005 23:51:00.499908  252097 cache.go:169] failed to download gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631, will try fallback image if available: tarball: **unexpected EOF**
I1005 23:51:00.499915  252097 image.go:79] Checking for docker.io/kicbase/stable:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 in local docker daemon
I1005 23:51:00.534181  252097 image.go:83] Found docker.io/kicbase/stable:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 in local docker daemon, skipping pull
I1005 23:51:00.534193  252097 cache.go:145] docker.io/kicbase/stable:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 exists in daemon, skipping load
W1005 23:51:00.534226  252097 out.go:239] ❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.40, but successfully downloaded docker.io/kicbase/stable:v0.0.40 as a fallback image

After minikube delete --all --purge and docker rmi docker.io/kicbase/stable:v0.0.40 <gcr.io image image id> everything works as expected.

Thx to https://github.com/kubernetes/minikube/issues/8997#issuecomment-1223530041

scott-the-programmer commented 1 year ago

It looks like minikube first looks at the local docker daemon before deciding whether or not to download the image. The mismatch here is that check disregards the host (docker.io vs ghr.io), causing the diff.

I'm somewhat tempted to remove the base_image option entirely, as this doesn't seem useful to be set via terraform i.e. playing around with base_image entails debugging of minikube itself

scott-the-programmer / terraform-provider-minikube