search

kudobuilder / kudo

Kubernetes Universal Declarative Operator (KUDO)
https://kudo.dev
Apache License 2.0
1.18k stars 103 forks source link

Patch failure on jobs #1425

Closed jieyu closed 4 years ago

jieyu commented 4 years ago

Using 0.11.0-rc1.

Observed this issue in kudo controller log:

2020/03/13 18:45:09 PlanExecution: A transient error when executing task deploy.install-base-dependencies.base-dependencies.base-dependencies. Will retry. failed to patch kubeflow/kubeflow-istio-gateway-certs: failed to execute patch: Job.batch "kubeflow-istio-gateway-certs" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"76f7cd5f-adbf-4338-b9e8-7fa97d5af71c", "heritage":"kudo", "job-name":"kubeflow-istio-gateway-certs", "kudo.dev/instance":"kubeflow-instance", "kudo.dev/operator":"kubeflow"}, Annotations:map[string]string{"kudo.dev/last-plan-execution-uid":"c6f7830a-db2d-4f11-b08e-fe7bd10d3692", "kudo.dev/operator-version":"0.0.1", "kudo.dev/phase":"install-base-dependencies", "kudo.dev/plan":"deploy", "kudo.dev/step":"base-dependencies", "sidecar.istio.io/inject":"false"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:"hostname", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(0xc02539c7c0), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"apply-cert-obj", Image:"bitnami/kubectl:1.16", Command:[]string{"/bin/sh", "-c"}, Args:[]string{"#!/bin/sh\nexport ISTIO_GATEWAY_HOSTNAME=$(cat /tmp/hostname/hostname)\ncat <<EOF > /tmp/certs.yaml\napiVersion: certmanager.k8s.io/v1alpha1\nkind: Certificate\nmetadata:\n  name: kubeflow-gateway-certs\n  namespace: istio-system\nspec:\n  secretName: kubeflow-gateway-certs\n  dnsNames:\n  - \"${ISTIO_GATEWAY_HOSTNAME}\"\n  commonName: kubeflow-gateway\n  duration: 19200h0m0s\n  organization:\n  - D2iQ\n  usages:\n  - digital signature\n  - key encipherment\n  - server auth\n  issuerRef:\n    name: kubernetes-ca\n    kind: ClusterIssuer\nEOF\nkubectl apply -f /tmp/certs.yaml\n"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:"hostname", ReadOnly:false, MountPath:"/tmp/hostname", SubPath:"", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc02605dbb0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kubeflow-deployment-job-service-account", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc03367df10), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable

The job spec is here:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .OperatorName }}-istio-gateway-certs
  namespace: {{ .Namespace }}
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: {{ .OperatorName }}-deployment-job-service-account
      containers:
        - name: apply-cert-obj
          image: bitnami/kubectl:1.16
          command: ["/bin/sh", "-c"]
          args:
            - |
              #!/bin/sh
              export ISTIO_GATEWAY_HOSTNAME=$(cat /tmp/hostname/hostname)
              cat <<EOF > /tmp/certs.yaml
              apiVersion: certmanager.k8s.io/v1alpha1
              kind: Certificate
              metadata:
                name: kubeflow-gateway-certs
                namespace: istio-system
              spec:
                secretName: kubeflow-gateway-certs
                dnsNames:
                - "${ISTIO_GATEWAY_HOSTNAME}"
                commonName: kubeflow-gateway
                duration: 19200h0m0s
                organization:
                - D2iQ
                usages:
                - digital signature
                - key encipherment
                - server auth
                issuerRef:
                  name: kubernetes-ca
                  kind: ClusterIssuer
              EOF
              kubectl apply -f /tmp/certs.yaml
          volumeMounts:
            - name: hostname
              mountPath: /tmp/hostname
      restartPolicy: Never
      volumes:
        - name: hostname
          configMap:
            name: {{ .Pipes.istioGatewayHostname }}

And the pipe task pod:

apiVersion: v1
kind: Pod
spec:
  serviceAccountName: {{ .OperatorName }}-deployment-job-service-account
  volumes:
  - name: shared-data
    emptyDir: {}
  initContainers:
    - name: init
      image: bitnami/kubectl:1.16
      command: [ "/bin/sh", "-c" ]
      args:
        - |
          #!/bin/sh
          kubectl get svc istio-ingressgateway \
            --namespace istio-system \
            -o jsonpath="{.status.loadBalancer.ingress[*].hostname}" \
            > /tmp/hostname
      volumeMounts:
        - name: shared-data
          mountPath: /tmp

operator.yaml

apiVersion: kudo.dev/v1beta1
name: "kubeflow"
operatorVersion: "0.0.1"
kudoVersion: "0.10.1"
kubernetesVersion: 1.15.0
appVersion: 0.7
maintainers:
- ...
url: https://kubeflow.org/
tasks:
  - name: cluster-scoped-resources
    kind: Apply
    spec:
      resources:
        - install-job-rbac.yaml
  - name: get-gateway-hostname
    kind: Pipe
    spec:
      pod: get-gateway-hostname.yaml
      pipe:
        - file: /tmp/hostname
          kind: ConfigMap
          key: istioGatewayHostname
  - name: base-dependencies
    kind: Apply
    spec:
      resources:
        - istio-gateway-certs-job.yaml
        - istio-gateway.yaml
...
plans:
  deploy:
    strategy: serial
    phases:
      - name: install-cluster-scoped-resources
        strategy: serial
        steps:
          - name: cluster-scoped-resources
            tasks:
              - cluster-scoped-resources
      - name: install-base-dependencies
        strategy: serial
        steps:
          - name: get-gateway-hostname
            tasks:
              - get-gateway-hostname
          - name: base-dependencies
            tasks:
              - base-dependencies

if I put get-gateway-hostname to be in a separate phase, the problem disappears. so i suspect some kind of race condition. looks like the job is created for some reason, and the subsequent reconcile tries to patch it. and job cannot be patched if it's already done (looks like)

jieyu commented 4 years ago

Even with the workaround (put get-gateway-hostname to be in a separate phase), i still occasionally experience the issue.

zen-dog commented 4 years ago

@jieyu do you have a KUDO manager log for me, from the moment deploy plan starts until it ends in the above error?

zen-dog commented 4 years ago

Ok, I believe I got to the bottom of it. The problem has nothing to do with the pipe task as such but rather with the usage of Jobs although putting a job into the same step as a pipe task certainly triggers the problem more often (due to the fact that the next reconciliation is scheduled much faster).

The exact same problem is easier to replicate with the newly introduced manual plan trigger feature, so I'll use it to demonstrate.

  1. Take any long-running job:

    apiVersion: batch/v1
kind: Job
metadata:
name: busy
spec:
template:
spec:
  containers:
  - name: busy
    image: busybox
    command: ["/bin/sh", "-c"] 
    args: ["sleep infinity"]
  restartPolicy: Never
backoffLimit: 3

  2. Make it part of a deploy plan and install a dummy operator:

    
apiVersion: kudo.dev/v1beta1
name: "dummy"
operatorVersion: "0.1.0"
kubernetesVersion: 1.15.0
maintainers:
- name: zen-dog
url: https://kudo.dev
tasks:
tasks:
- name: job
kind: Apply
spec:
  resources:
    - job.yaml

plans: deploy: strategy: serial phases:

  1. Now, the deploy plan is still running, and we trigger it again with k kudo plan trigger --name deploy --instance dummy-instance. Note, that you need to run the KUDO manager with ENABLE_WEBHOOKS=true for this command to work. Observe the problem: 
    PlanExecution: A transient error when executing task deploy.deploy.dummy.job. 
Will retry. failed to patch default/busy: 
failed to execute patch: 
Job.batch "busy" is invalid: spec.template: Invalid value: core.PodTemplateSpec{...}: field is immutable

TL;DR: the issue boils down to the fact, that the plan is reconciled faster than the previous Status is saved. This raciness in the operator pattern is well described in #1116 This second reconciliation leads to us trying to patch an existing Job pod template which is immutable:

Screenshot 2020-03-17 at 11 28 50

This problem exists for any plan that has jobs.

zen-dog commented 4 years ago

Fixed by this commit