search

volcano-sh / volcano

A Cloud Native Batch System (Project under CNCF)
https://volcano.sh
Apache License 2.0
4.25k stars 971 forks source link

scheduler update job failed #3104

Open WulixuanS opened 1 year ago

WulixuanS commented 1 year ago

What happened: A large number of error logs

E0907 03:39:28.604985       1 job_updater.go:105] Failed to update job <video/pgvx20230907113107.701322940-098ad-45e77d6a-f3d6-42a2-9002-c3b9f7c4c506>: Operation cannot be fulfilled on podgroups.scheduling.volcano.sh "pgvx20230907113107.701322940-098ad-45e77d6a-f3d6-42a2-9002-c3b9f7c4c506": StorageError: invalid object, Code: 4, Key: /registry/scheduling.volcano.sh/podgroups/video/pgvx20230907113107.701322940-098ad-45e77d6a-f3d6-42a2-9002-c3b9f7c4c506, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 265232bc-3fe2-47cb-8335-0808e5a569a4, UID in object meta:
E0907 03:39:28.636600       1 job_updater.go:105] Failed to update job <video/gvx20230907113235.385478059-2deeb-c3fdcbe3-6f70-44ed-ae32-3833abbfc01f>: Operation cannot be fulfilled on podgroups.scheduling.volcano.sh "gvx20230907113235.385478059-2deeb-c3fdcbe3-6f70-44ed-ae32-3833abbfc01f": StorageError: invalid object, Code: 4, Key: /registry/scheduling.volcano.sh/podgroups/video/gvx20230907113235.385478059-2deeb-c3fdcbe3-6f70-44ed-ae32-3833abbfc01f, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 673133c8-ba6a-468c-820b-b2c540769575, UID in object meta:
E0907 03:39:28.698327       1 job_updater.go:105] Failed to update job <video/gvx20230907113230.237306638-fcab6-af772ecb-df01-4977-ac5c-d30d7275b365>: Operation cannot be fulfilled on podgroups.scheduling.volcano.sh "gvx20230907113230.237306638-fcab6-af772ecb-df01-4977-ac5c-d30d7275b365": StorageError: invalid object, Code: 4, Key: /registry/scheduling.volcano.sh/podgroups/video/gvx20230907113230.237306638-fcab6-af772ecb-df01-4977-ac5c-d30d7275b365, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 020110aa-deca-4503-a635-87eb75e09ad2, UID in object meta:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

william-wang commented 1 year ago

/assign @Mufengzhe

volcano-sh-bot commented 1 year ago

@william-wang: GitHub didn't allow me to assign the following users: Mufengzhe.

Note that only volcano-sh members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/volcano-sh/volcano/issues/3104#issuecomment-1711165421): >/assign @Mufengzhe Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
Mufengzhe commented 1 year ago

I did not find the problem you described. Can you provide more information to help me reproduce this problem?

calvin0327 commented 7 months ago

I took a look at the code, and there's only one place that could be causing this error.