Can not schedule the newly created pod when Job failed

[NickrenREN]

Deployed as suggested here

When i create the job, the pod created by the job controller is scheduled as expected, but if the pod failed, the job controller will create a new pod, the new pod can not be scheduled.

I0523 07:36:27.118480       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:36:35.696850       1 podwatcher.go:884] {default-token-m4cct {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-m4cct,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} is the volume name for pod ubuntu-92bhn
I0523 07:36:35.696950       1 podwatcher.go:335] enqueuePodAddition: Added pod {ubuntu-92bhn default}
I0523 07:36:35.696969       1 podwatcher.go:462] PodPending {ubuntu-92bhn default}
I0523 07:36:37.410006       1 poseidon.go:47] Scheduler returned 1 deltas
I0523 07:36:44.178136       1 podwatcher.go:397] enqueuePodUpdate: Updated pod state change {ubuntu-92bhn default} Failed
I0523 07:36:44.178172       1 podwatcher.go:538] PodFailed {ubuntu-92bhn default}
I0523 07:36:44.181434       1 podwatcher.go:884] {default-token-m4cct {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-m4cct,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} is the volume name for pod ubuntu-27b7w
I0523 07:36:44.181510       1 podwatcher.go:335] enqueuePodAddition: Added pod {ubuntu-27b7w default}
I0523 07:36:44.181526       1 podwatcher.go:462] PodPending {ubuntu-27b7w default}
I0523 07:36:47.411748       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:36:57.412951       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:37:07.413789       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:37:17.414907       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:37:27.415898       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:37:37.417550       1 poseidon.go:47] Scheduler returned 0 deltas
I0523 07:37:47.418313       1 poseidon.go:47] Scheduler returned 0 deltas

[NickrenREN]

The newly created pod will keep pending, and does not have any events.

[NickrenREN]

@deepak-vij @shivramsrivastava @m1093782566

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[NickrenREN]

/remove-lifecycle stale

[NickrenREN]

This is caused by Firmament

/close

[k8s-ci-robot]

@NickrenREN: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/poseidon/issues/171#issuecomment-523922593): >This is caused by Firmament > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.