Open mbwhite opened 1 year ago
@mbwhite I was not able to reproduce, I tried to reproduce using the following playbook
- hosts: localhost
connection: local
gather_facts: false
collections:
- kubernetes.core
tasks:
- name: create resources
k8s:
definition: "{{ lookup('kubernetes.core.kustomize', dir='https://github.com/hyperledger-labs/fabric-operator.git/config/ingress/kind') }}"
- name: get jobs from namespace
k8s_info:
kind: "Job"
version: "batch/v1"
namespace: "ingress-nginx"
register: jobs
- name: assert jobs was created
assert:
that:
- jobs.resources | length > 0
But no error found
ansible-playbook ./play.yaml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [localhost] *********************************************************************************************************************************
TASK [create resources] **************************************************************************************************************************
changed: [localhost]
TASK [get jobs from namespace] *******************************************************************************************************************
ok: [localhost]
TASK [assert jobs was created] *******************************************************************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
PLAY RECAP ***************************************************************************************************************************************
localhost : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
could you please run again and provide logs?
I am facing a similar issue when try to deploy:
It simply doesn't execute the Jobs
manifests. In this deploy there are two, named ingress-nginx-admission-create
and ingress-nginx-admission-patch
. They create the secrets the deployment.apps/ingress-nginx-controller
pods mount. As consequence, since pods cannot mount the secrets, the deployment pods remain in ContainerCreating
state, making the kubernetes.core.k8s
process to fail because of the timeout of the wait
configured in the task.
The playbook tasks I am using look like the following:
- name: Disable controlplane taints
# controlplane is the only node available at this moment of cluster
# deployment.
kubernetes.core.k8s_taint:
state: absent
name: controlplane
taints:
- effect: NoSchedule
key: "node-role.kubernetes.io/master"
- effect: NoSchedule
key: "node-role.kubernetes.io/control-plane"
become: false
- name: Install manifests for nginx ingress controller, metallb load balancer, and flannel CNI
kubernetes.core.k8s:
template: "{{ item }}"
wait: true
loop:
- kube-flannel.yml.j2
- metallb-0.13.7.yml.j2
- metallb-0.13.7-config.yml.j2
- ingress-nginx-1.5.1.yml.j2
become: false
- name: Enable controlplane taints
kubernetes.core.k8s_taint:
state: present
name: controlplane
taints:
- effect: NoSchedule
key: "node-role.kubernetes.io/master"
- effect: NoSchedule
key: "node-role.kubernetes.io/control-plane"
become: false
It may be worth mentioning that deploying the same url through kubectl apply
works as intended.
Tried to move the manifest from a single file to individual definition tasks and didn't work either. Here is the task:
- name: Job ingress-nginx-admission-create
kubernetes.core.k8s:
definition:
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.5.1
name: ingress-nginx-admission-create
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.5.1
name: ingress-nginx-admission-create
spec:
containers:
- args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f
imagePullPolicy: IfNotPresent
name: create
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
become: false
and the related output:
TASK [system-k8s-controlplane : Job ingress-nginx-admission-create] ************
fatal: [controlplane]: FAILED! => {"changed": false, "error": 422, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Job.batch \\\\\"ingress-nginx-admission-create\\\\\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\\\\\"ingress-nginx-admission-create\\
\\\", GenerateName:\\\\\"\\\\\", Namespace:\\\\\"\\\\\", SelfLink:\\\\\"\\\\\", UID:\\\\\"\\\\\", ResourceVersion:\\\\\"\\\\\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:\\\\u003cnil\\\\u003e, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\\\\\"app.kubernetes.io/component\\\\\":\\\\\"admission-webh
ook\\\\\", \\\\\"app.kubernetes.io/instance\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/name\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/part-of\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/version\\\\\":\\\\\"1.5.1\\\\\", \\\\\"controller-uid\\\\\":\\\\\"356cc6e9-e926-40ba-8fb8-a34e874eb34d\\\\\", \\\\\"job-name\\\\\":\\\\\"ingress-nginx-ad
mission-create\\\\\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\\\\\"create\\\\\", Image:\\\\\"registry.k8s.io/ingress-nginx/kube-webhook-ce
rtgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f\\\\\", Command:[]string(nil), Args:[]string{\\\\\"create\\\\\", \\\\\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\\\\\", \\\\\"--namespace=$(POD_NAMESPACE)\\\\\", \\\\\"--secret-name=ingress-nginx-admission\\\\\"}, WorkingDir:\\
\\\"\\\\\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\\\\\"POD_NAMESPACE\\\\\", Value:\\\\\"\\\\\", ValueFrom:(*core.EnvVarSource)(0xc010bcde40)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\\\\\"/dev/termination-log\\\\\", TerminationMessagePolicy:\\\\\"File\\\\\", ImagePullPolicy:\\\\\"IfNotPresent\\\\\", SecurityContext:(*core.SecurityContext)(0xc00b597020), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\\\\\"OnFailure\\\\\", TerminationGracePeriodSeconds:(*int64)(0xc00ca10140), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\\\\\"ClusterFirst\\\\\", NodeSelector:map[string]string{\\\\\"kubernetes.io/os\\\\\":\\\\\"linux\\\\\"}, ServiceAccountName:\\\\\"ingress-nginx-admission\\\\\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\\\\\"\\\\\", SecurityContext:(*core.PodSecurityContext)(0xc00b76b8c0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\\\\\"\\\\\", Subdomain:\\\\\"\\\\\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\\\\\"default-scheduler\\\\\", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\\\\\"\\\\\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable\",\"reason\":\"Invalid\",\"details\":{\"name\":\"ingress-nginx-admission-create\",\"group\":\"batch\",\"kind\":\"Job\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\\\\\"ingress-nginx-admission-create\\\\\", GenerateName:\\\\\"\\\\\", Namespace:\\\\\"\\\\\", SelfLink:\\\\\"\\\\\", UID:\\\\\"\\\\\", ResourceVersion:\\\\\"\\\\\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:\\\\u003cnil\\\\u003e, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\\\\\"app.kubernetes.io/component\\\\\":\\\\\"admission-webhook\\\\\", \\\\\"app.kubernetes.io/instance\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/name\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/part-of\\\\\":\\\\\"ingress-nginx\\\\\", \\\\\"app.kubernetes.io/version\\\\\":\\\\\"1.5.1\\\\\", \\\\\"controller-uid\\\\\":\\\\\"356cc6e9-e926-40ba-8fb8-a34e874eb34d\\\\\", \\\\\"job-name\\\\\":\\\\\"ingress-nginx-admission-create\\\\\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\\\\\"create\\\\\", Image:\\\\\"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f\\\\\", Command:[]string(nil), Args:[]string{\\\\\"create\\\\\", \\\\\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\\\\\", \\\\\"--namespace=$(POD_NAMESPACE)\\\\\", \\\\\"--secret-name=ingress-nginx-admission\\\\\"}, WorkingDir:\\\\\"\\\\\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\\\\\"POD_NAMESPACE\\\\\", Value:\\\\\"\\\\\", ValueFrom:(*core.EnvVarSource)(0xc010bcde40)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\\\\\"/dev/termination-log\\\\\", TerminationMessagePolicy:\\\\\"File\\\\\", ImagePullPolicy:\\\\\"IfNotPresent\\\\\", SecurityContext:(*core.SecurityContext)(0xc00b597020), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\\\\\"OnFailure\\\\\", TerminationGracePeriodSeconds:(*int64)(0xc00ca10140), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\\\\\"ClusterFirst\\\\\", NodeSelector:map[string]string{\\\\\"kubernetes.io/os\\\\\":\\\\\"linux\\\\\"}, ServiceAccountName:\\\\\"ingress-nginx-admission\\\\\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\\\\\"\\\\\", SecurityContext:(*core.PodSecurityContext)(0xc00b76b8c0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\\\\\"\\\\\", Subdomain:\\\\\"\\\\\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\\\\\"default-scheduler\\\\\", Tolerat
ions:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:\\\\\"\\\\\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core
.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable\",\"field\":\"spec.template\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
Recreated the lab from scratch, ran the playbook again, using individual kubernetes.core.k8s
tasks, and the previous error is no longer in the output.
TASK [system-k8s-controlplane : Deployment ingress-nginx-controller] ***********
changed: [controlplane]
TASK [system-k8s-controlplane : Job ingress-nginx-admission-create] ************
changed: [controlplane]
TASK [system-k8s-controlplane : Job ingress-nginx-admission-patch] *************
changed: [controlplane]
TASK [system-k8s-controlplane : IngressClass nginx] ****************************
changed: [controlplane]
The Jobs
tasks were completed successfully and the deployment pods are running:
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx pod/ingress-nginx-admission-create-gg7ls 0/1 Completed 0 68s
ingress-nginx pod/ingress-nginx-admission-patch-trndw 0/1 Completed 0 66s
ingress-nginx pod/ingress-nginx-controller-65dc77f88f-mjjgm 1/1 Running 0 69s
The problem seems to happen only when we read the manifest from file or standard input. Otherwise, kubernetes.core.k8s
work just fine.
A second run of the same individual tasks over the same lab, works as expected too:
TASK [system-k8s-controlplane : Deployment ingress-nginx-controller] ***********
ok: [controlplane]
TASK [system-k8s-controlplane : Job ingress-nginx-admission-create] ************
ok: [controlplane]
TASK [system-k8s-controlplane : Job ingress-nginx-admission-patch] *************
ok: [controlplane]
TASK [system-k8s-controlplane : IngressClass nginx] ****************************
ok: [controlplane]
SUMMARY
Applying (via lustomize) a multi-definition file, that contains jobs - all is well apart from the JOBS are not run The same file run with
kubectl apply -k
does run the jobsISSUE TYPE
COMPONENT NAME
ANSIBLE VERSION
COLLECTION VERSION
STEPS TO REPRODUCE
EXPECTED RESULTS
Will create a Ingress and then run 2 jobs to patch this...
Creates the Ingress but does not run the jobs.
ACTUAL RESULTS