Closed matthewmichihara closed 3 years ago
tstromberg
commented
3 years ago This error is probably safe to ignore - but something we'll need to address nonetheless.
Some Kubernetes objects are immutable, and the addon doesn't yet understand being enabled twice on a running cluster.
sharifelgamal
commented
3 years ago there are some race conditions around enabling and disabling the addon consecutively, the errors are spurious and the addon should still work.
medyagh
commented
3 years ago this is fixed in 1.14.0
matthewmichihara
commented
3 years ago @medyagh are you sure? I ask because in the minikube start output it says it was using version 1.14.0.
sharifelgamal
commented
3 years ago @matthewmichihara Can you try with 1.14.2 and see if it still happens? Just reopen if you can repro it.
matthewmichihara
commented
3 years ago I saw this pop up again with minikube 1.18.1 via Cloud Code. @medyagh @sharifelgamal can we re-open this issue?
/Users/michihara/Library/Application Support/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/bin/minikube start --wait true --interactive false --delete-on-failure
* minikube v1.18.1 on Darwin 11.2.3
- MINIKUBE_WANTUPDATENOTIFICATION=false
* Kubernetes 1.20.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.20.2
* Using the docker driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Restarting existing docker container for "minikube" ...
* Preparing Kubernetes v1.19.2 on Docker 19.03.8 ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v4
- Using image jettech/kube-webhook-certgen:v1.3.0
- Using image gcr.io/k8s-minikube/gcp-auth-webhook:v0.0.4
* Verifying gcp-auth addon...
* Your GCP credentials will now be mounted into every pod created in the minikube cluster.
* If you don't want your credentials mounted into a specific pod, add a label with the `gcp-auth-skip-secret` key to your pod configuration.
! Enabling 'gcp-auth' returned an error: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.19.2/kubectl apply -f /etc/kubernetes/addons/gcp-auth-ns.yaml -f /etc/kubernetes/addons/gcp-auth-service.yaml -f /etc/kubernetes/addons/gcp-auth-webhook.yaml: Process exited with status 1
stdout:
namespace/gcp-auth unchanged
service/gcp-auth unchanged
serviceaccount/minikube-gcp-auth-certs unchanged
clusterrole.rbac.authorization.k8s.io/minikube-gcp-auth-certs unchanged
clusterrolebinding.rbac.authorization.k8s.io/minikube-gcp-auth-certs unchanged
deployment.apps/gcp-auth unchanged
mutatingwebhookconfiguration.admissionregistration.k8s.io/gcp-auth-webhook-cfg unchanged
stderr:
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"name\":\"gcp-auth-certs-create\",\"namespace\":\"gcp-auth\"},\"spec\":{\"template\":{\"metadata\":{\"name\":\"gcp-auth-certs-create\"},\"spec\":{\"containers\":[{\"args\":[\"create\",\"--host=gcp-auth,gcp-auth.gcp-auth,gcp-auth.gcp-auth.svc\",\"--namespace=gcp-auth\",\"--secret-name=gcp-auth-certs\"],\"image\":\"jettech/kube-webhook-certgen:v1.3.0@sha256:ff01fba91131ed260df3f3793009efbf9686f5a5ce78a85f81c386a4403f7689\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"create\"}],\"restartPolicy\":\"OnFailure\",\"serviceAccountName\":\"minikube-gcp-auth-certs\"}}}}
"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"create"}],"containers":[{"image":"jettech/kube-webhook-certgen:v1.3.0@sha256:ff01fba91131ed260df3f3793009efbf9686f5a5ce78a85f81c386a4403f7689","name":"create"}]}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "gcp-auth-certs-create", Namespace: "gcp-auth"
for: "/etc/kubernetes/addons/gcp-auth-webhook.yaml": Job.batch "gcp-auth-certs-create" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"gcp-auth-certs-create", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"987a4bdc-e645-4aa5-b956-71ae9cbaeb27", "job-name":"gcp-auth-certs-create"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"create", Image:"jettech/kube-webhook-certgen:v1.3.0@sha256:ff01fba91131ed260df3f3793009efbf9686f5a5ce78a85f81c386a4403f7689", Command:[]string(nil), Args:[]string{"create", "--host=gcp-auth,gcp-auth.gcp-auth,gcp-auth.gcp-auth.svc", "--namespace=gcp-auth", "--secret-name=gcp-auth-certs"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc0090bc3a0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"minikube-gcp-auth-certs", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00ef9eb00), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"name\":\"gcp-auth-certs-patch\",\"namespace\":\"gcp-auth\"},\"spec\":{\"template\":{\"metadata\":{\"name\":\"gcp-auth-certs-patch\"},\"spec\":{\"containers\":[{\"args\":[\"patch\",\"--secret-name=gcp-auth-certs\",\"--namespace=gcp-auth\",\"--patch-validating=false\",\"--webhook-name=gcp-auth-webhook-cfg\"],\"image\":\"jettech/kube-webhook-certgen:v1.3.0@sha256:ff01fba91131ed260df3f3793009efbf9686f5a5ce78a85f81c386a4403f7689\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"patch\"}],\"restartPolicy\":\"OnFailure\",\"serviceAccountName\":\"minikube-gcp-auth-certs\"}}}}
"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"patch"}],"containers":[{"image":"jettech/kube-webhook-certgen:v1.3.0@sha256:ff01fba91131ed260df3f3793009efbf9686f5a5ce78a85f81c386a4403f7689","name":"patch"}]}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "gcp-auth-certs-patch", Namespace: "gcp-auth"
for: "/etc/kubernetes/addons/gcp-auth-webhook.yaml": Job.batch "gcp-auth-certs-patch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"gcp-auth-certs-patch", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"981acbae-26ce-485f-9bdf-bf7de398b76d", "job-name":"gcp-auth-certs-patch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"patch", Image:"jettech/kube-webhook-certgen:v1.3.0@sha256:ff01fba91131ed260df3f3793009efbf9686f5a5ce78a85f81c386a4403f7689", Command:[]string(nil), Args:[]string{"patch", "--secret-name=gcp-auth-certs", "--namespace=gcp-auth", "--patch-validating=false", "--webhook-name=gcp-auth-webhook-cfg"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc002b15720), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"minikube-gcp-auth-certs", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc009c0dd80), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable
]
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "" namespace by default
minikube started successfully.
Enabling GCP auth addon...
Failed to enable GCP auth addon. Deployment will continue but GCP credentials will not be added to minikube. Please ensure you have up to date application default credentials (ADC) by running `gcloud auth login --update-adc`I'm working on a repro case for this so we can fix this once and for all.
I don't exactly know how I triggered this but I ran into this error when starting minikube today.
I do notice that my
gcloudapplication default credentials are expired, so that may be related:Full output of
minikube logscommand:**