While creating jobs with multiple tasks such as to reboot a host, it is common to see intermittent job failures because the rufio controller is unable to get an updated list of submitted tasks from the informer cache
for example following is a sample job to reboot a host.
This seems to be caused by https://github.com/tinkerbell/rufio/blob/main/controller/job.go#L109-L138 the controller attempting to list tasks owned by the job which fails because the list returns no objects. As a result the controller attempts to create task-0 again, which fails as this task already exists.
Expected Behaviour
Current Behaviour
Possible Solution
One possible fix, that I have available in my local fork is disable caching for Job objects while setting up the manager with a minor change to controller options as shown below. This ensures that the objects are read from the apiserver and avoids this issue.
While creating jobs with multiple tasks such as to reboot a host, it is common to see intermittent job failures because the rufio controller is unable to get an updated list of submitted tasks from the informer cache
for example following is a sample job to reboot a host.
This seems to be caused by https://github.com/tinkerbell/rufio/blob/main/controller/job.go#L109-L138 the controller attempting to list tasks owned by the job which fails because the list returns no objects. As a result the controller attempts to create task-0 again, which fails as this task already exists.
Expected Behaviour
Current Behaviour
Possible Solution
One possible fix, that I have available in my local fork is disable caching for Job objects while setting up the manager with a minor change to controller options as shown below. This ensures that the objects are read from the apiserver and avoids this issue.
Steps to Reproduce (for bugs)
Context
Your Environment
Operating System and version (e.g. Linux, Windows, MacOS): Opensuse 15.4
How are you running Tinkerbell? Using Vagrant & VirtualBox, Vagrant & Libvirt, on Packet using Terraform, or give details: k8s sandbox
Link to your project or a code example to reproduce issue: