search

ansible-collections / kubernetes.core

The collection includes a variety of Ansible content to help automate the management of applications in Kubernetes and OpenShift clusters, as well as the provisioning and maintenance of clusters themselves.
Other
215 stars 135 forks source link

Applying definition does not appear to run 'jobs' #562

Open mbwhite opened 1 year ago

mbwhite commented 1 year ago
SUMMARY

Applying (via lustomize) a multi-definition file, that contains jobs - all is well apart from the JOBS are not run The same file run with kubectl apply -k does run the jobs

ISSUE TYPE
COMPONENT NAME
ANSIBLE VERSION
ansible [core 2.14.1]
  config file = /home/matthew/github.com/hyperledger/fabric-ansible-collection/examples/opensource-stack/ansible.cfg
  configured module search path = ['/home/matthew/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/matthew/.cache/pypoetry/virtualenvs/ansible-collection-marbrFo2-py3.9/lib/python3.9/site-packages/ansible
  ansible collection location = /home/matthew/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/matthew/.cache/pypoetry/virtualenvs/ansible-collection-marbrFo2-py3.9/bin/ansible
  python version = 3.9.16 (main, Dec  7 2022, 01:11:51) [GCC 9.4.0] (/home/matthew/.cache/pypoetry/virtualenvs/ansible-collection-marbrFo2-py3.9/bin/python)
  jinja version = 3.1.2
  libyaml = True
COLLECTION VERSION
kubernetes.core         2.3.2
STEPS TO REPRODUCE
EXPECTED RESULTS
kubectl apply -k https://github.com/hyperledger-labs/fabric-operator.git/config/ingress/kind

Will create a Ingress and then run 2 jobs to patch this... 

    - name: Create kubernetes resources for the ingress
      k8s:
        definition: "{{ lookup('kubernetes.core.kustomize', dir='https://github.com/hyperledger-labs/fabric-operator.git/config/ingress/kind') }}"

Creates the Ingress but does not run the jobs.

ACTUAL RESULTS
abikouo commented 1 year ago

@mbwhite I was not able to reproduce, I tried to reproduce using the following playbook 

- hosts: localhost
  connection: local

  gather_facts: false

  collections:
    - kubernetes.core

  tasks:
    - name: create resources
      k8s:
        definition: "{{ lookup('kubernetes.core.kustomize', dir='https://github.com/hyperledger-labs/fabric-operator.git/config/ingress/kind') }}"

    - name: get jobs from namespace
      k8s_info:
        kind: "Job"
        version: "batch/v1"
        namespace: "ingress-nginx"
      register: jobs

    - name: assert jobs was created
      assert:
        that:
          - jobs.resources | length > 0

But no error found

ansible-playbook ./play.yaml 
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *********************************************************************************************************************************

TASK [create resources] **************************************************************************************************************************
changed: [localhost]

TASK [get jobs from namespace] *******************************************************************************************************************
ok: [localhost]

TASK [assert jobs was created] *******************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP ***************************************************************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

could you please run again and provide logs?

areguera commented 1 year ago

I am facing a similar issue when try to deploy:

It simply doesn't execute the Jobs manifests. In this deploy there are two, named ingress-nginx-admission-create and ingress-nginx-admission-patch. They create the secrets the deployment.apps/ingress-nginx-controller pods mount. As consequence, since pods cannot mount the secrets, the deployment pods remain in ContainerCreating state, making the kubernetes.core.k8s process to fail because of the timeout of the wait configured in the task.

The playbook tasks I am using look like the following:

- name: Disable controlplane taints
  # controlplane is the only node available at this moment of cluster
  # deployment.
  kubernetes.core.k8s_taint:
    state: absent
    name: controlplane
    taints:
    - effect: NoSchedule
      key: "node-role.kubernetes.io/master"
    - effect: NoSchedule
      key: "node-role.kubernetes.io/control-plane"
  become: false

- name: Install manifests for nginx ingress controller, metallb load balancer, and flannel CNI
  kubernetes.core.k8s:
    template: "{{ item }}"
    wait: true
  loop:
    - kube-flannel.yml.j2
    - metallb-0.13.7.yml.j2
    - metallb-0.13.7-config.yml.j2
    - ingress-nginx-1.5.1.yml.j2
  become: false

- name: Enable controlplane taints
  kubernetes.core.k8s_taint:
    state: present
    name: controlplane
    taints:
    - effect: NoSchedule
      key: "node-role.kubernetes.io/master"
    - effect: NoSchedule
      key: "node-role.kubernetes.io/control-plane"
  become: false

It may be worth mentioning that deploying the same url through kubectl apply works as intended.

areguera commented 1 year ago

Tried to move the manifest from a single file to individual definition tasks and didn't work either. Here is the task:

- name: Job ingress-nginx-admission-create
  kubernetes.core.k8s:
    definition:
      apiVersion: batch/v1
      kind: Job
      metadata:
        labels:
          app.kubernetes.io/component: admission-webhook
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
          app.kubernetes.io/part-of: ingress-nginx
          app.kubernetes.io/version: 1.5.1
        name: ingress-nginx-admission-create
        namespace: ingress-nginx
      spec:
        template:
          metadata:
            labels:
              app.kubernetes.io/component: admission-webhook
              app.kubernetes.io/instance: ingress-nginx
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
              app.kubernetes.io/version: 1.5.1
            name: ingress-nginx-admission-create
          spec:
            containers:
            - args:
              - create
              - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
              - --namespace=$(POD_NAMESPACE)
              - --secret-name=ingress-nginx-admission
              env:
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f
              imagePullPolicy: IfNotPresent
              name: create
              securityContext:
                allowPrivilegeEscalation: false
            nodeSelector:
              kubernetes.io/os: linux
            restartPolicy: OnFailure
            securityContext:
              fsGroup: 2000
              runAsNonRoot: true
              runAsUser: 2000
            serviceAccountName: ingress-nginx-admission
  become: false

and the related output:

TASK [system-k8s-controlplane : Job ingress-nginx-admission-create] ************
fatal: [controlplane]: FAILED! => {"changed": false, "error": 422, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Job.batch \\\\\"ingress-nginx-admission-create\\\\\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\\\\\"ingress-nginx-admission-create\\
\\\", GenerateName:\\\\\"\\\\\", Namespace:\\\\\"\\\\\", SelfLink:\\\\\"\\\\\", UID:\\\\\"\\\\\", ResourceVersion:\\\\\"\\\\\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:\\\\u003cnil\\\\u003e, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\\\\\"app.kubernetes.io/component\\\\\":\\\\\"admission-webh
ook\\\\\", \\\\\"app.kubernetes.io/instance\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/name\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/part-of\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/version\\\\\":\\\\\"1.5.1\\\\\", \\\\\"controller-uid\\\\\":\\\\\"356cc6e9-e926-40ba-8fb8-a34e874eb34d\\\\\", \\\\\"job-name\\\\\":\\\\\"ingress-nginx-ad
mission-create\\\\\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\\\\\"create\\\\\", Image:\\\\\"registry.k8s.io/ingress-nginx/kube-webhook-ce
rtgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f\\\\\", Command:[]string(nil), Args:[]string{\\\\\"create\\\\\", \\\\\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\\\\\", \\\\\"--namespace=$(POD_NAMESPACE)\\\\\", \\\\\"--secret-name=ingress-nginx-admission\\\\\"}, WorkingDir:\\
\\\"\\\\\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\\\\\"POD_NAMESPACE\\\\\", Value:\\\\\"\\\\\", ValueFrom:(*core.EnvVarSource)(0xc010bcde40)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\\\\\"/dev/termination-log\\\\\", TerminationMessagePolicy:\\\\\"File\\\\\", ImagePullPolicy:\\\\\"IfNotPresent\\\\\", SecurityContext:(*core.SecurityContext)(0xc00b597020), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\\\\\"OnFailure\\\\\", TerminationGracePeriodSeconds:(*int64)(0xc00ca10140), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\\\\\"ClusterFirst\\\\\", NodeSelector:map[string]string{\\\\\"kubernetes.io/os\\\\\":\\\\\"linux\\\\\"}, ServiceAccountName:\\\\\"ingress-nginx-admission\\\\\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\\\\\"\\\\\", SecurityContext:(*core.PodSecurityContext)(0xc00b76b8c0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\\\\\"\\\\\", Subdomain:\\\\\"\\\\\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\\\\\"default-scheduler\\\\\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\\\\\"\\\\\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable\",\"reason\":\"Invalid\",\"details\":{\"name\":\"ingress-nginx-admission-create\",\"group\":\"batch\",\"kind\":\"Job\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\\\\\"ingress-nginx-admission-create\\\\\", GenerateName:\\\\\"\\\\\", Namespace:\\\\\"\\\\\", SelfLink:\\\\\"\\\\\", UID:\\\\\"\\\\\", ResourceVersion:\\\\\"\\\\\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:\\\\u003cnil\\\\u003e, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\\\\\"app.kubernetes.io/component\\\\\":\\\\\"admission-webhook\\\\\", \\\\\"app.kubernetes.io/instance\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/name\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/part-of\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/version\\\\\":\\\\\"1.5.1\\\\\", \\\\\"controller-uid\\\\\":\\\\\"356cc6e9-e926-40ba-8fb8-a34e874eb34d\\\\\", \\\\\"job-name\\\\\":\\\\\"ingress-nginx-admission-create\\\\\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\\\\\"create\\\\\", Image:\\\\\"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f\\\\\", Command:[]string(nil), Args:[]string{\\\\\"create\\\\\", \\\\\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\\\\\", \\\\\"--namespace=$(POD_NAMESPACE)\\\\\", \\\\\"--secret-name=ingress-nginx-admission\\\\\"}, WorkingDir:\\\\\"\\\\\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\\\\\"POD_NAMESPACE\\\\\", Value:\\\\\"\\\\\", ValueFrom:(*core.EnvVarSource)(0xc010bcde40)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\\\\\"/dev/termination-log\\\\\", TerminationMessagePolicy:\\\\\"File\\\\\", ImagePullPolicy:\\\\\"IfNotPresent\\\\\", SecurityContext:(*core.SecurityContext)(0xc00b597020), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\\\\\"OnFailure\\\\\", TerminationGracePeriodSeconds:(*int64)(0xc00ca10140), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\\\\\"ClusterFirst\\\\\", NodeSelector:map[string]string{\\\\\"kubernetes.io/os\\\\\":\\\\\"linux\\\\\"}, ServiceAccountName:\\\\\"ingress-nginx-admission\\\\\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\\\\\"\\\\\", SecurityContext:(*core.PodSecurityContext)(0xc00b76b8c0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\\\\\"\\\\\", Subdomain:\\\\\"\\\\\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\\\\\"default-scheduler\\\\\", Tolerat
ions:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\\\\\"\\\\\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core
.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable\",\"field\":\"spec.template\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
areguera commented 1 year ago

Recreated the lab from scratch, ran the playbook again, using individual kubernetes.core.k8s tasks, and the previous error is no longer in the output. 

TASK [system-k8s-controlplane : Deployment ingress-nginx-controller] ***********
changed: [controlplane]

TASK [system-k8s-controlplane : Job ingress-nginx-admission-create] ************
changed: [controlplane]

TASK [system-k8s-controlplane : Job ingress-nginx-admission-patch] *************
changed: [controlplane]

TASK [system-k8s-controlplane : IngressClass nginx] ****************************
changed: [controlplane]

The Jobs tasks were completed successfully and the deployment pods are running:

NAMESPACE          NAME                                            READY   STATUS      RESTARTS   AGE
ingress-nginx      pod/ingress-nginx-admission-create-gg7ls        0/1     Completed   0          68s
ingress-nginx      pod/ingress-nginx-admission-patch-trndw         0/1     Completed   0          66s
ingress-nginx      pod/ingress-nginx-controller-65dc77f88f-mjjgm   1/1     Running     0          69s

The problem seems to happen only when we read the manifest from file or standard input. Otherwise, kubernetes.core.k8s work just fine.

areguera commented 1 year ago

A second run of the same individual tasks over the same lab, works as expected too:

TASK [system-k8s-controlplane : Deployment ingress-nginx-controller] ***********
ok: [controlplane]

TASK [system-k8s-controlplane : Job ingress-nginx-admission-create] ************
ok: [controlplane]

TASK [system-k8s-controlplane : Job ingress-nginx-admission-patch] *************
ok: [controlplane]

TASK [system-k8s-controlplane : IngressClass nginx] ****************************
ok: [controlplane]