Distinguish between Preemption due to reclamation and fair sharing

[alculquicondor]

What would you like to be added:

An additional reason in the Preemption condition to indicate that the preemption was due to fair sharing and not regular reclamation.

The current reason for reclamation or fair sharing is InCohort

We could have: InCohortReclamation and InCohortFairSharing.

Why is this needed:

For better visibility

Completion requirements:

This enhancement requires the following artifacts:

• [ ] Design doc
• [ ] API change
• [ ] Docs update

The artifacts should be linked in subsequent comments.

[vladikkuzn]

/assign

[vladikkuzn]

@alculquicondor https://github.com/kubernetes-sigs/kueue/blob/6ecaa877ee65791e6333f147fafcbb7888039a03/pkg/scheduler/scheduler.go#L247-L251

If Assigment is borrowing then the reason should be InCohortFairSharing? Or should the decision be based on enableFairSharing of Preemptor? https://github.com/kubernetes-sigs/kueue/blob/45eee46b1c1264a8a025d0a811de78e99fbc7abb/pkg/scheduler/preemption/preemption.go#L49-L59

[alculquicondor]

I think the decision is based entirely on the preemptor. If the preemptor fits in the nominal quota of its ClusterQueue, then it's preempting others that are borrowing its quota. Otherwise, it's past the nominal quota, meaning that it's using resources from the fair share.

Note that whether fair sharing is enabled is global.

/assign @vladikkuzn

[alculquicondor]

cc @gabesaba

[trasc]

I think the decision is based entirely on the preemptor. If the preemptor fits in the nominal quota of its ClusterQueue, then it's preempting others that are borrowing its quota. Otherwise, it's past the nominal quota, meaning that it's using resources from the fair share.

Also, if the preemptor and preemptee are in the same cluster queue we can say it's a simple priority based eviction (no quota is reclaimed from a borrower)